Senior Quality Assurance Automation Engineer

About NoteSpark
NoteSpark is an AI-powered call analytics product built as a connector on the LeadSquared
marketplace. It converts recorded sales calls into structured intelligence: transcripts,
diarization, sentiment, BANT signals, and action items, and enables conversational querying
over call history using RAG. Because the product’s core output is AI-generated, quality here
means more than “does the button work”; it means validating that the intelligence itself is
accurate, consistent, and trustworthy.
What you’ll do
Own test strategy and execution for NoteSpark across its modules: configuration playground, ingestion pipeline, lead-level insights, reports, and dashboards
Build and maintain automation suites for API and UI testing, integrated into our CI/CD pipeline
Write BDD-style test scenarios (Gherkin/Cucumber or similar) in collaboration with product and engineering, so requirements and tests stay in sync
Design test approaches for AI-generated outputs: evaluating transcription accuracy, diarization quality, summary correctness, extraction precision (action items, BANT), and regression across model/prompt changes
Test event-driven flows: webhook triggers, queue processing, and end-to-end call analysis pipelines
Log clear, reproducible defects and work closely with engineers through fix and verification
Use AI tools day to day to accelerate your own testing work: generating test data, drafting scenarios, and analyzing results
Must have
3–4 years of QA experience with a strong foundation in test automation (Selenium, Playwright, Cypress, or similar; API automation with Postman/RestAssured or similar)
Hands-on experience with BDD frameworks: writing and maintaining Gherkin scenarios with Cucumber, SpecFlow, or equivalent
Good working knowledge of AI tools and comfort using them as part of the testing workflow
Solid understanding of QA fundamentals: test design, regression planning, defect lifecycle, and CI integration
Strong analytical skills and attention to detail, especially for outputs where “correct” isn’t binary
Good to have
Experience testing AI/LLM-based applications: evaluating non-deterministic outputs, building golden datasets, prompt regression testing, or model output scoring
Exposure to audio/speech products or data-heavy SaaS platforms
Basic scripting ability (Python, JavaScript, or C#) beyond test frameworks
Familiarity with AWS or cloud-based test environments
Why join
You’ll be the quality owner on a small, senior team shipping an AI product to enterprise
customers. This is a chance to define how AI-based applications should be tested (a skill set the industry is only beginning to figure out) rather than following an established playbook.

Work arrangement
No