Associate Vice Presidentsenior - Lead MLOps Engineer

Responsibilities:
Own day-to-day operations for multiple AI services (LLM apps, ML models, RAG services, agents), meeting SLO/SLA targets for availability, latency, cost, and quality.
Maintain operational documentation (runbooks, escalation matrices) and participate in the on-call schedule.
Coordinate incident response including triage, mitigation, and root cause analysis.
Manage releases and deployments (including canary deployments and blue/green rollouts) and make sure test and production environments remain aligned.
Follow the company's formal change management process and maintain required documentation for audits.
Design and maintain full observability across AI systems covering both service health metrics and model quality indicators.
Track and act on KPIs such as latency, throughput, error rates, safety filter triggers, factual accuracy issues, user acceptance rates, and operational cost per 1K tokens.
Monitor data and model drift and take action when behaviour changes.
Build dashboards and alerts; adjust thresholds based on trends.
Run A/B tests and use controlled side-by-side evaluations to validate improvements.
Operate and improve continuous integration and continuous delivery workflows for machine learning and large language model components, including pipelines, model registries, and version control.
Trigger retraining or prompt updates when quality drops, new data becomes available, or policies change.
Coordinate any required data labelling tasks.
Manage rollbacks and controlled promotions between models to ensure reliability and auditability.
Maintain full traceability for decisions, model versions, and deployments.
Ensure compliance with data protection rules, regional data storage requirements, and secure secret handling.
Integrate AI-safety measures such as content filtering and misuse detection.
Maintain documentation describing how each model works, how it was trained, and any risks or limitations.
Collaborate with Security, Risk, and Governance teams to provide required evidence for audits and reviews.
Work closely with testing teams on release preparation, environment planning, and resilience testing.
Contribute AI-specific checks such as prompt quality regression tests, bias/safety evaluations, and retrieval quality assessments.
Collaborate with data engineering, application engineering, and AI-engineering teams to optimise operations and control costs.
Help build shared components, templates, and best practices that other AI teams can reuse.
Ensure all operational and risk-related documentation is kept up to date.
Requirements:
Experience in Site Reliability Engineering, DevOps, Test Operations, Platform Engineering, or MLOps ideally with responsibility for production systems.
Hands-on experience with container orchestration platforms such as OpenShift or Kubernetes; CI/CD tools; and observability solutions (metrics, logs, traces, dashboards).
Practical experience operating machine learning or large language model workloads: serving models, running pipelines, maintaining model registries, and evaluating performance.
Strong understanding of evaluating ML/LLM quality and safety, detecting drift, running A/B tests, and managing controlled deployments.
Proficiency in Python and familiarity with one ML platform (MLflow, Vertex AI, Kubeflow, etc. ).
Ability to communicate clearly and produce high-quality documentation (runbooks, change requests, incident reports).
Experience with retrieval-based AI systems (vector databases, retrieval pipelines), prompt engineering, or multi-step AI agents.
Background in AI security or risk assessments (safety testing, privacy, auditability).
Knowledge of enterprise change management processes and business continuity testing.

Experience
7-11 yrs