QA Engineer

About Chargebee
Chargebee is a leading provider of billing and monetization solutions, empowering businesses with recurring revenue models to streamline revenue and finance operations, capture actionable insights, and drive growth.
Chargebee is trusted by businesses of all sizes, including Zapier, LegalZoom, Lambda, Freshworks, DeepL, Condé Nast, and Pret a Manger, and is proud to have been consistently recognized by customers as a Leader in Subscription Management on G2.
With headquarters in North Bethesda, Maryland, our team members are based primarily in India, the U.S., and Europe.
About the Team
Our AI Products team builds Chargebee Copilot and other product experiences that bring Chargebee’s AI Agents platform to real users. We turn the platform’s agents, tools, and APIs into products people use every day, from the backend services and APIs that power them to the interfaces on top. We care about good conversational experiences and about giving people the tools to configure and improve how an AI agent behaves.
About the role (Job Summary)
We’re looking for an AI Quality Engineer to own the quality of our AI-powered products and agents throughout their lifecycle. Unlike traditional QA, AI systems aren’t deterministic, so they need continuous evaluation before release and ongoing improvement based on real-world usage. You’ll make sure our AI delivers accurate, reliable, high-quality experiences by designing effective evaluations, validating new capabilities, analyzing production behavior, and driving improvements across the AI stack. This role is ideal for someone who enjoys understanding why AI systems behave the way they do and is passionate about improving AI products through evidence, experimentation, and engineering collaboration.
What you’ll do
Design, review, and improve AI evaluation datasets, benchmarks, and regression suites, ensuring coverage reflects real customer scenarios, edge cases, and expected user behavior
Validate new AI capabilities before they are released to customers, and continuously improve evaluations and release criteria
Monitor AI quality in production using customer feedback, sampled conversations, telemetry, and quality metrics
Perform root cause analysis to understand why AI interactions succeed or fail, classifying issues across prompts, retrieval, tool usage, model behavior, workflows, product knowledge, and application logic
Identify recurring quality patterns and recommend engineering improvements
Convert production learnings into evaluation cases to prevent future regressions
Partner with Engineering, Product, and Support to improve prompts, agent workflows, retrieval systems, tools, and evaluation frameworks
Define and track AI quality KPIs, and measure the impact of improvements using benchmarks and production metrics
What you’ll bring (required skills and experience)
Strong analytical and investigative mindset with excellent problem-solving skills
2+ years of experience
Ability to identify root causes rather than symptoms
Excellent written communication and documentation skills
Comfort working with APIs, logs, structured data, and technical systems
Ability to collaborate effectively with engineering teams
Passion for building high-quality AI products
Nice to have (preferred)
Experience in QA, SDET, Technical Support Engineering, Solutions Engineering, Product Analysis, or AI/ML applications
Familiarity with LLMs, prompt engineering, RAG, MCP, AI agents, or AI evaluation frameworks
Experience designing evaluation datasets, benchmarks, or testing strategies
Basic programming or scripting skills (Python, SQL, JavaScript, etc.)
Benefits we offer
At Chargebee, we believe that feeling supported is the foundation for doing great work. We offer a comprehensive benefits package designed to support your personal, financial, and professional growth and ensure you thrive in every aspect of your life. Our commitment to supporting our employees is universal, while our benefits are thoughtfully designed to meet the unique needs of each region. We offer comprehensive packages that generally include:
Health & Wellness: Robust coverage and access to programs to support your physical and mental well-being.
Financial Security: Initiatives to help you build long-term financial stability.
Work-Life Balance: Flexible options and resources to help you thrive both professionally and personally.
Chargebee is an equal opportunity employer. At Chargebee, we believe that diversity of backgrounds, perspectives, and experiences is not a “good-to-have” but a “must-have” as it strengthens us as a company and as individuals. We are committed to fostering an inclusive workplace where everyone feels valued, respected, and empowered to thrive.
We do not discriminate on the basis of race, color, religion, sex (including pregnancy, childbirth, and related medical conditions), sexual orientation, gender identity/expression, age, national origin, ancestry, caste, citizenship, physical or mental disability, genetic information, marital status, military or veteran status, or any other characteristic protected by applicable law.

Work arrangement
Hybrid