Principal Site Reliability Engineer

Designs and architects infrastructure and service to ensure reliability and functionality. Forecasts demands and responds to capacity needs. Collaborates with software development teams to develop reliable and scalable infrastructures. Exercises judgment when performing data collection to maintain and optimize operations and reliability. Leverages advanced knowledge to perform incident response and/or maintenance tasks. Provides comprehensive health and performance reporting. Identifies and recommends opportunities for automation. Communicates comprehensive information about services and proactively anticipates and articulates the potential impact of changes. Provides comprehensive support for technology and documents incidents. Conducts advanced experiments with new tools and develops and maintains advanced knowledge of site reliability trends.
Mandatory
5+ years of hands-on experience across software development, site reliability engineering, monitoring, and automation, with demonstrated ownership of production platforms.
Strong programming skills in Python and experience developing backend services, REST APIs, integrations, and event-driven workflows.
Proven experience building monitoring and observability solutions that collect and analyze alerts, metrics, logs, and service context.
Hands-on experience integrating AI/LLM capabilities into applications for incident analysis, contextual recommendations, and operational decision support.
Ability to develop reusable agents that combine live monitoring data, historical incidents, and previous remediation actions to recommend next steps.
Experience implementing alert correlation, deduplication, enrichment, and prioritization to reduce noise and improve incident detection.
Strong experience developing automated remediation workflows with approval controls, execution validation, audit trails, and recovery safeguards.
Experience integrating monitoring platforms with incident management systems, knowledge repositories, and notification or escalation services.
Strong understanding of distributed systems, Linux, networking, databases, and application performance, with the ability to translate failure patterns into platform capabilities.
Experience applying software engineering practices, including automated testing, code reviews, CI/CD, version control, and secure development.
Ability to evaluate platform reliability and AI output quality, measure operational outcomes, and improve capabilities using engineer feedback.
Excellent communication and collaboration skills, with the ability to work with monitoring teams, service owners, and engineers to onboard services and define requirements.
Good-to-Have
Experience with agentic AI frameworks, tool calling, retrieval-augmented generation, and vector databases.
Knowledge of anomaly detection, time-series analysis, and predictive techniques for identifying recurring alarm patterns.
Experience building centralized notification services, automated escalation calls, and incident communication workflows.
Familiarity with OCI, Kubernetes, and Infrastructure-as-Code tools such as Terraform or Ansible.
Experience with observability platforms such as OCI Observability, Prometheus, Grafana or equivalent tools.
Exposure to Oracle Database and Exadata monitoring, performance diagnostics, and remediation automation.
Experience using AI coding assistants to improve operational scripts and validate proposed changes.
Knowledge of SLOs, error budgets, and incident metrics such as detection time, recovery time, and recurrence rate.
Key Responsibilities
Capacity Ingestion and Management:
- Designs and architects infrastructure and/or service according to terms for reliability and functionality.
- Forecasts demands for infrastructure and responds to capacity needs, ensuring systems have sufficient resources to handle current and future workloads and identifying resource gaps.
- Collaborates with the software development team to develop infrastructures, ensuring features are reliable and scalable according to deployment requirements.
- Proactively identifies opportunities for prototyping and drives prototyping initiatives (e.g., testing new applications or infrastructures, assisting in onboarding) to explore novel approaches.
Incident and Service Lifecycle Management:
- Exercises judgment when performing data collection, triage, technical analysis, and redirection to maintain and optimize operations and infrastructure reliability.
- Takes proactive steps to monitor services, maintain up-to-date knowledge of their performance, and document their condition.
- Leverages advanced knowledge to perform incident response, root cause analyses, and/or maintenance on assigned services (e.g., software installs, version upgrades, security updates, backup and recovery).
- Provides comprehensive health and performance reporting and takes appropriate actions based on trends in data.
- May perform provisioning to support infrastructure, applications, and services.
- May experiment with new approaches for and performs decommissioning (e.g., shutting down servers, removing data from databases) to remove objects that are no longer needed.
Automation:
-