Staff Product Software Engineer

Requirements:
8+ years of backend or systems engineering experience.
Strong proficiency in Python.
Experience designing automated testing or evaluation frameworks at scale.
Experience operating AI-powered or model-dependent systems in production.
Strong understanding of CI/CD integration and release gating processes.
Experience defining measurable quality standards across distributed systems.
Proven ability to influence architectural and engineering standards across multiple teams.
Preferred Qualifications:
Experience designing benchmarking systems for LLMs or ML models.
Familiarity with hallucination detection, safety scoring, or response consistency analysis.
Experience building A/B testing infrastructure for model comparison.
Exposure to multi-provider model environments (Azure, OpenAI, Bedrock).
Background in experimentation systems or engineering-led release governance.
Experience with designing automated regression testing frameworks for AI systems.
Building structured evaluation datasets and benchmarking pipelines.
Model comparison systems and A/B testing infrastructure.
Prompt drift detection and workflow validation. CI/CD integration for automated evaluation gating.
AI provider APIs and model performance characteristics.
Observability and analytics systems for quality measurement.
Cross-functional collaboration with AI Core, Architecture, and Governance leadership.

Experience
8-12 yrs