Live opening · Posted 1 day ago
The key details from the original listing.
Live timing from JobBeeper.
Description supplied by the original job listing.
Job title: AI Test Engineer
Location: Pan India
Work Mode: Remote
Candidate Specification: 3+ years overall, Relevant – 3+ years
Candidate Location: Pan India
Skills: MCP, RAG, agentic AI, LangSmith
Candidate Industry: IT
Education and Certification: Any Graduation
Job Description
Must have 3+ years in software QA/testing, with hands-on senior-level experience evaluating LLM-based or agentic AI systems (not just traditional QA)
Direct experience with eval frameworks/tools — LangSmith strongly preferred; also acceptable: Ragas, DeepEval, PromptFoo, Braintrust, Langfuse
Strong understanding of RAG architecture and how to independently evaluate retrieval quality vs. generation quality
Practical familiarity with MCP (Model Context Protocol) or comparable agent tool calling frameworks
Strong Python skills for building test harnesses, scoring pipelines, and automation
Experience designing evaluation approaches for non-deterministic systems (probabilistic scoring, not binary pass/fail)
Comfortable working directly with client engineering teams — strong communication, able to explain quality risk to both technical and business stakeholders
Work arrangement
Yes
Recently tracked roles with full details and direct application links.