Senior Quality Engineer - AI Customer Engagement

Responsibilities:
Design, develop, execute, and automate comprehensive test plans, cases, and scripts for AI-driven and cloud-native products.
Collaborate with Product Managers and Developers to validate system functionality, usability, and performance.
Define and track quality metrics to ensure continuous improvement and timely delivery.
Own the end-to-end quality lifecycle from test strategy and automation to defect triage and release sign-off.
Leverage GenAI tools (e. g., ChatGPT, Claude, Copilot) to accelerate test design, automate testing tasks, and analyze defects efficiently.
Use Langfuse for LLM tracing, output validation, and LLM-as-a-Judge scoring to assess accuracy, relevance, and consistency.
Conduct API, integration, and system testing for distributed, cloud-based environments.
Build and maintain automation frameworks using Python, Playwright, Selenium, and BDD methodologies.
Identify quality risks, define mitigation strategies, and maintain measurable quality checkpoints.
Champion a culture of accountability, precision, and a quality-first mindset within the team.
AI and Cloud Expertise:
Own the end-to-end quality lifecycle from test strategy and automation to defect triage and release sign-off.
Leverage GenAI tools (e. g., ChatGPT, Claude, Copilot) to accelerate test design, automate testing tasks, and analyze defects efficiently.
Conduct API, integration, and system testing for distributed, cloud-based environments.
Build and maintain automation frameworks using Python, Selenium, and BDD methodologies.
Identify quality risks, define mitigation strategies, and maintain measurable quality checkpoints.
Champion a culture of accountability, precision, and a quality-first mindset within the team.
Requirements:
Must have exposure to LLM evaluation techniques, including output scoring, benchmarking, and validation frameworks.
Understanding of prompt engineering, Retrieval-Augmented Generation (RAG), model orchestration, and hallucination detection.
Experience testing accuracy, relevance, and consistency of AI model outputs and generated responses.
Ability to define and validate performance metrics for AI-driven services.
Awareness of AI safety, bias detection, and explainability methods to ensure fair and interpretable outcomes.
Familiarity with AI governance frameworks such as NIST AI RMF and EU AI Act, ensuring responsible and compliant testing practices.
Strong belief in ethical AI, transparency, and maintaining end-user trust.
Bachelor's degree in Computer Science, Engineering, or related discipline.
5-8 years of QA automation and testing experience across complex applications.
Proficiency in Python, Selenium, and BDD frameworks.
Strong experience in API testing, cloud-based distributed systems, and test management systems.
Familiarity with GenAI tools to enhance testing productivity and efficiency.
Understanding of LLM evaluation techniques, prompt testing, and hallucination detection.
Analytical mindset with strong debugging and problem-solving skills.
Excellent communication and collaboration abilities.
Self-starter who thrives in a fast-paced, innovative environment.

Experience
5-9 yrs