Sr. Product Software Engineer

The candidate will have responsibilities across the following functions:
Evaluation Pipeline Development:
Implement automated benchmarking workflows for model versions and providers.
Build regression detection systems comparing model outputs across updates.
Develop evaluation harnesses for structured test datasets.
Integrate evaluation results into CI/CD release workflows.
Model Quality and Testing Systems:
Support A/B testing systems for model comparison.
Contribute to response consistency and hallucination analysis tooling.
Assist in implementing prompt and workflow drift detection automation.
Ensure reproducibility and traceability of evaluation results.
Collaboration and Execution:
Partner with AI Core Engineering to validate new model integrations.
Work with Runtime and Reliability teams to correlate evaluation results with production performance.
Participate in model readiness reviews and certification processes.
Requirements:
5+ years of backend or systems engineering experience.
Strong proficiency in Python.
Experience building automated testing or evaluation pipelines.
Experience integrating external APIs or model services.
Experience working in cloud-based environments.
Preferred Qualifications:
Exposure to AI-powered systems or LLM integrations.
Experience with benchmarking or structured dataset validation systems.
Familiarity with CI/CD-based release gating workflows.
Experience contributing to experimentation or validation platforms.
Experience with:
Backend services and distributed systems.
Automated testing frameworks and regression detection systems.
Evaluation dataset processing and validation tooling.
CI/CD pipelines and automated release gating.
Observability or analytics systems for performance measurement.
LLM APIs or model-based services.
Cloud-native deployment environments.
Required Education:
Bachelor's degree in engineering or a related technical field (Master's preferred), or equivalent experience.

Experience
4-7 yrs