AI Engineer

Responsibilities:
Design and build evaluation systems for conversational AI across chat and voice.
Define and track core metrics: task success, correctness, latency, conversation quality, etc.
Analyse conversation logs to identify failure patterns and edge cases.
Build failure taxonomies (intent errors, hallucinations, tool misuse, conversation breakdowns).
Create automated eval pipelines (LLM-based + rule-based + dataset-driven).
Set up regression test suites for prompts, workflows, and agent behaviour.
Run experiments and A/B tests across prompts, models, and agent configurations.
Work closely with Applied AI and product teams to translate insights into system improvements.
Build internal tooling and dashboards to monitor AI performance at scale.
Requirements:
3-5 years experience in backend engineering, data systems, or ML-adjacent roles.
Experience working with data pipelines, logs, or analytics systems.
Strong understanding of metrics, experimentation, and system behaviour.
Ability to debug complex systems and trace issues across multiple components.
Comfort working with ambiguity and defining structure where none exists.
Additional experience (not mandatory):
Experience with LLMs, NLP systems, or conversational AI.
Familiarity with prompt design or agent workflows.
Experience with evaluation frameworks, test systems, or QA automation.
Exposure to speech systems (ASR/TTS) or voice products.
Experience in high-scale or fast-moving startup environments.
Good to Have:
You think in terms of systems and feedback loops, not just outputs.
You care about measuring quality, not guessing it.
You naturally break down failures into patterns and root causes.
You are comfortable working with messy real-world data (especially conversations).
You've built systems that helped teams improve continuously, not just once.

Experience
3-6 yrs