AI Operations Analyst

At Johnson & Johnson, we believe health is everything. Our strength in healthcare innovation empowers us to build a world where complex diseases are prevented, treated, and cured, where treatments are smarter and less invasive, and solutions are personal. Through our expertise in Innovative Medicine and MedTech, we are uniquely positioned to innovate across the full spectrum of healthcare solutions today to deliver the breakthroughs of tomorrow, and profoundly impact health for humanity. Learn more at jnj.com.
As guided by Our Credo, Johnson & Johnson is responsible to our employees who work with us throughout the world. We provide an inclusive work environment where each person is considered as an individual. At Johnson & Johnson, we respect the diversity and dignity of our employees and recognize their merit.
Job Function:
Technology Product & Platform Management
Job Sub Function:
Technical Product Management
Job Category:
Scientific/Technology
All Job Posting Locations:
Lisbon, Portugal, Madrid - Lafinca, Milano, Italy
Job Description:
Role Overview
The AI Operations Analyst manages day-to-day operational analysis and monitoring for AI applications and AI agents, helping maintain reliable, secure, transparent, and cost-effective services. The role combines hands-on investigation, dashboard and reporting ownership, incident coordination, and continuous improvement of established operational processes.
The analyst takes ownership of assigned operational activities across observability, consumption reporting, incident triage, audit evidence, and service readiness. The role works with AI Operations engineers and partner teams to investigate issues, recommend improvements, and implement approved changes within defined access and change-management controls. The analyst also supports solution design by contributing operational insights and practical requirements for monitoring, supportability, and cost visibility.
The role does not own the end-to-end lifecycle management of agents or AI products, platform architecture, or final security and compliance approvals. Agent design, development, functional evaluation, release content, product evolution, and retirement decisions remain with the Agent Factory and the relevant AI product teams. The analyst independently delivers routine operational work and coordinates more complex investigations with engineers and accountable control owners.
Key ResponsibilitiesAI Observability & Production Monitoring
· Own routine monitoring of AI applications and agents, analysing logs, traces, model and tool calls, retrieval steps, latency, failures, and token consumption to identify service issues.
· Validate telemetry completeness across agent workflows and supporting services, investigate gaps, and coordinate corrective actions with engineers and platform owners.
· Build and maintain operational dashboards and reports covering availability, latency, errors, quality signals, policy alerts, consumption, and service health.
· Analyse performance against agreed service-level indicators and objectives, tune operational alert thresholds within approved standards, and recommend improvements to monitoring coverage.
· Investigate incidents using logs and traces, develop evidence-based hypotheses, and coordinate next steps while following approved access and sensitive-data handling procedures.
· Investigate agent loops, retrieval issues, degraded tool performance, and unexpected runtime behaviour; prioritise findings and escalate complex defects to engineering teams.
· Own incident analysis and follow-up for assigned operational issues, contribute to root-cause reviews, and implement agreed preventive actions within the role’s technical authority.
LLMOps & AgentOps Support
· Maintain configuration, prompt, model, and agent-component version records, and improve the consistency of operational checks across environments.
· Execute and improve deployment-readiness checks, investigate failed integration, regression, security, or observability checks, and coordinate resolution with responsible teams.
· Coordinate operational checks and monitoring during rollouts, validate fallback and rollback readiness, and execute authorised operational actions under approved change procedures; release approval remains with accountable owners.
· Maintain consistent use of shared operational tools and identify opportunities to standardise support processes across models, frameworks, and agent patterns. Support engineers and product teams during design by helping define telemetry, alerting, troubleshooting, and recovery requirements; design decisions remain with accountable engineering and product owners.
· Analyse functional evaluation results supplied by the Agent Factory or AI product teams alongside operational signals, flag regressions, and support readiness reviews; functional quality ownership remains with those teams.
· Own the accuracy and maintenance of assigned runbooks, checklists, knowledge articles, and reusable operational examples.
AI Consumption & Cost Reporting
· Own recurring consumption and cost reporting by product, agent, workflow, model, environment, or business unit, validating allocation data and explaining material variances.
· Analyse token consumption, model usage, repeated loops, retrieval overhead, infrastructure usage, and cost-per-transaction metrics to identify inefficiencies and emerging trends.
· Investigate consumption anomalies and budget alerts, recommend threshold adjustments, and coordinate approved containment actions with engineers and budget owners.
· Recommend optimisation opportunities in model selection, context size, caching, retries, and tool usage, partnering with engineers and product teams to assess quality and compliance impacts.
· Track the outcomes of approved optimisation actions and provide cost and usage analysis for capacity planning; budget and investment decisions remain with accountable leads.
Security & Compliance Operations Support
· Apply approved security, privacy, Responsible AI, and compliance requirements to operational activities, identifying control gaps and coordinating remediation with the relevant owners.
· Coordinate routine access reviews, investigate discrepancies, and track corrective actions; permission changes and secrets administration remain with authorised owners.
· Triage and investigate runtime security alerts, including suspected prompt injection, unauthorised tool use, data leakage, and anomalous access; coordinate response with security and engineering teams.
· Validate that operational logging, retention, redaction, and access practices follow approved requirements, and resolve or escalate identified deviations.
· Maintain complete and accurate operational evidence linking versions, configurations, identities, actions, approvals, and outcomes.
· Coordinate recurring policy checks and evidence collection, analyse gaps, and track remediation to closure with accountable control owners.
· Contribute to security testing and incident response through operational investigation, evidence analysis, and delivery of assigned remediation actions.
Operational Service Support & Documentation
· Manage assigned monitoring, support, incident, problem, and change activities in accordance with agreed service processes and escalation paths.
· Coordinate operational-readiness assessments, validate service acceptance evidence, and maintain runbooks, escalation paths, and recovery documentation.
· Coordinate operational handoffs across AI Operations, the Agent Factory, product teams, and platform owners, ensuring open issues have clear owners and follow-up actions.
· Develop and maintain scripts or workflow improvements that automate repetitive reporting, validation, and support tasks, using approved testing and change controls.
· Maintain operational knowledge articles and share practical guidance to improve consistency and effectiveness across support teams.
Stakeholder Collaboration
· Act as the operational point of contact for assigned services, coordinating updates, investigations, and information requests across AI Operations and partner teams.
· Translate operational findings into clear requirements, prioritised issues, and improvement recommendations for engineering and control owners; contribute to design reviews by highlighting operational lessons, support needs, and potential service risks.
· Promote adoption of approved operational practices and resolve routine process questions, escalating architectural or policy decisions to accountable owners.
Governance, Risk & Compliance
· Coordinate assigned governance and compliance activities, validate evidence completeness, and raise unresolved risks or exceptions to accountable o

Skills
Product Management, Documentation