Lead Principal Site Reliability Engineer

Serves as a consultant and leads the design and architecture of infrastructure and service, ensuring alignment with reliability and functionality standards. Takes full ownership of forecasting of demands and responding to capacity needs. Owns collaborations with software development teams to develop reliable and scalable infrastructures. Recommends methods for performing data collection to maintain and optimize operations and reliability. Oversees incident response and/or maintenance tasks. Provides strategic, future-oriented health and performance reporting. Contributes to strategies for automation and reviews the development and implementation of automation. Provides expert-level communication about services and anticipates, analyzes, and explains the impact of changes, considering strategic goals. Serves as a role model in providing support for technology and reviews documentation for accuracy. Leads the implementation of innovative tools and provides expertise in site reliability trends.
Mandatory
8+ years of hands-on experience across software development, site reliability engineering, monitoring, and automation, with demonstrated ownership of production platforms.
Strong programming skills in Python and experience developing backend services, REST APIs, integrations, and event-driven workflows.
Proven experience building monitoring and observability solutions that collect and analyze alerts, metrics, logs, and service context.
Hands-on experience integrating AI/LLM capabilities into applications for incident analysis, contextual recommendations, and operational decision support.
Ability to develop reusable agents that combine live monitoring data, historical incidents, and previous remediation actions to recommend next steps.
Experience implementing alert correlation, deduplication, enrichment, and prioritization to reduce noise and improve incident detection.
Strong experience developing automated remediation workflows with approval controls, execution validation, audit trails, and recovery safeguards.
Experience integrating monitoring platforms with incident management systems, knowledge repositories, and notification or escalation services.
Strong understanding of distributed systems, Linux, networking, databases, and application performance, with the ability to translate failure patterns into platform capabilities.
Experience applying software engineering practices, including automated testing, code reviews, CI/CD, version control, and secure development.
Ability to evaluate platform reliability and AI output quality, measure operational outcomes, and improve capabilities using engineer feedback.
Excellent communication and collaboration skills, with the ability to work with monitoring teams, service owners, and engineers to onboard services and define requirements.
Good-to-Have
Experience with agentic AI frameworks, tool calling, retrieval-augmented generation, and vector databases.
Knowledge of anomaly detection, time-series analysis, and predictive techniques for identifying recurring alarm patterns.
Experience building centralized notification services, automated escalation calls, and incident communication workflows.
Familiarity with OCI, Kubernetes, and Infrastructure-as-Code tools such as Terraform or Ansible.
Experience with observability platforms such as OCI Observability, Prometheus, Grafana or equivalent tools.
Exposure to Oracle Database and Exadata monitoring, performance diagnostics, and remediation automation.
Experience using AI coding assistants to improve operational scripts and validate proposed changes.
Knowledge of SLOs, error budgets, and incident metrics such as detection time, recovery time, and recurrence rate.
Key Responsibilities
Capacity Ingestion and Management:
- Serves as a consultant across teams and leads the design and architecture of infrastructure and/or service, ensuring alignment with reliability and functionality standards.
- Takes full ownership of forecasting infrastructure demands and strategically responds to capacity needs, ensuring systems have sufficient resources to handle current and future workloads, anticipating resource gaps.
- Owns collaborations and serves as a decision maker with the software development team in designing and developing infrastructures that are highly reliable and scalable, removing barriers to meet deployment requirements.
- Seeks opportunities for prototyping and encourages the implementation of prototyping initiatives (e.g., testing new applications or infrastructures, assisting in onboarding), driving new approaches.
- Identifies and projects resource gaps using cost calculators, identifying alternative solutions to lower costs, when necessary.
Incident and Service Lifecycle Management:
- Recommends methods for data collection, triage, technical analysis, and redirection to maintain and optimize operations and infrastructure reliability.
- Provides guidance to others when monitoring services, maintains up-to-date knowledge of their performance, and meticulously documents their condition.
- Oversees root cause analyses for incidents and/or maintenance on assigned services (e.g., software installs, version upgrades, security updates, backup and recovery), ensuring efficient execution and preventing incident reoccurrence.
- Provides strategic, future-oriented health and performance reporting, and anticipates actions needed based on trends in data.
- May guide provisioning to support infrastructure, applications, and services.
-