Software Engineer

We are looking an experienced AI Test Engineer to drive quality assurance for AI-powered applications, including LLM models, conversational bots, autonomous agents, and MCP-enabled systems. The ideal candidate will have strong expertise in Playwright with .NET, API/UI automation, AI/LLM validation, and CI/CD automation. The role involves building scalable test frameworks, validating AI responses, testing agent workflows, and ensuring the reliability, security, and performance of GenAI solutions.
Role :AI Test Engineer
Location: PAN India
Experience Range: 5 to 12years
Key Responsibilities:
Design and execute test strategies for LLM-powered applications, AI chatbots, conversational AI, and autonomous agents.
Develop and maintain Playwright (.NET) automation frameworks for UI and API testing.
Validate AI model outputs for:
Accuracy
Relevance
Hallucinations
Consistency
Safety and Guardrails
Test MCP (Model Context Protocol) endpoints and MCP-enabled agent workflows.
Automate API testing and integrate test suites into CI/CD pipelines.
Create test datasets, evaluation metrics, and AI validation frameworks.
Perform regression, functional, integration, and end-to-end testing for AI systems.
Collaborate with developers, data scientists, and product teams to ensure quality delivery.
Monitor and troubleshoot AI system failures, agent interactions, and model behavior.
Implement quality gates within Azure DevOps/GitHub Actions/Jenkins pipelines.
Required Skills:
AI / GenAI Testing
LLM Testing
Prompt Validation
AI Response Validation
Conversational Bot Testing
Autonomous Agent Testing
MCP (Model Context Protocol)
Hallucination Detection
AI Safety & Guardrail Testing
RAG Validation
Automation Testing
Playwright with .NET (C#)
UI Automation
API Automation
Postman / REST APIs
Selenium (Good to Have)
Programming
C#
.NET Core / ASP.NET
JavaScript / TypeScript (Good to Have)
DevOps & CI/CD
Azure DevOps
GitHub Actions
Jenkins
Docker (Preferred)
Other Skills
Agile/Scrum
SQL Databases
Test Framework Design
Version Control (Git)

Work arrangement
No