Forward Deployed Engineer - AI Assurance

ABOUT:
Owns quality for AI-native applications — functional testing plus the AI-specific evaluation (accuracy, drift, hallucination).
KEY RESPONSIBILITIES
Build test plans and automation for AI-native application features (functional + AI-specific)
Design evaluation harnesses for model/agent outputs — accuracy, consistency, hallucination rate
Run regression testing across model/prompt/config changes to catch silent quality drift
Red-team AI features for edge cases and adversarial inputs where relevant
Build automated eval pipelines integrated into CI/CD
Partner with AI Architects to define testability requirements before build starts
Own the quality gate before any AI feature ships to production
Communicate quality risk to delivery leadership in terms they can act on
Train delivery teams on AI-specific testing practices
Own the evals framework for the practice — golden datasets, scoring rubrics, LLM-as-judge calibration, and versioned benchmarks per use case
Define eval acceptance thresholds per engagement and gate releases on them
Build eval engineering tooling — dataset curation, trace capture, offline/online eval runs, and dashboards delivery teams can read
Instrument production evals and drift monitoring, feeding failures back into the golden datasets
REQUIREMENTS & SKILLS
5–9 yrs QA/test engineering, with 2+ yrs testing AI/ML-powered features specifically
Strong test automation skills (Python-based frameworks, CI/CD integration)
Understands AI-specific failure modes — hallucination, bias, drift, non-determinism — and designs tests for them
Statistically literate enough to interpret model evaluation metrics, not just pass/fail results
Familiarity with red-teaming methodologies for AI systems
Clear, assertive communicator — willing to block a release over a quality concern
Detail-oriented and methodical under delivery-timeline pressure
Collaborative but independent — doesn't rubber-stamp under delivery pressure
Explains quality risk in business-impact terms, not just technical jargon
Hands-on evals engineering — builds and maintains eval suites with frameworks such as OpenAI Evals, Ragas, DeepEval, LangSmith, Azure AI Foundry evaluations
Designs golden datasets and rubrics, and calibrates LLM-as-judge scoring against human review
Understands RAG and agent eval metrics — groundedness, retrieval precision/recall, task completion, tool-call correctness, cost/latency
Experience wiring evals and drift monitoring into CI/CD and production observability

Work arrangement
No