Databricks Infrastructure & DevOps Engineer

About the Role
We are looking for an experienced Databricks Platform Engineer to help design, build, and operate a modern enterprise data platform. This role focuses on developing core platform automation, deployment frameworks, governance capabilities, and observability solutions that enable scalable, secure, and reliable data operations across the organization.
You will play a critical role in building and maintaining data contract-driven deployment systems, automation frameworks, monitoring infrastructure, and governance tooling. The ideal candidate combines deep Databricks expertise with strong Python development skills and software engineering best practices.
This position requires a hands-on engineer who can proactively identify complex design challenges, propose practical solutions, and contribute throughout the entire platform lifecycle, from architecture and implementation to operations and continuous improvement.
Responsibilities
Data Contract System & Deployment Pipeline
Design and implement a contract-driven automation bundle generator that converts Open Data Contract Standard (ODCS) YAML files into deployable Databricks assets.
Build and maintain deployment pipelines for Declarative Automation Bundles, including:
Environment parameterization
Deployment atomicity
Rollback mechanisms
Develop and operate Azure DevOps/Azure Pipelines CI/CD workflows, including:
Multi-stage pipelines
Environment approvals
Variable groups
Service connections
Automate governance processes within Unity Catalog, including:
Schemas
Tables
Tags
Grants
Develop integrations utilizing Databricks SDKs and REST APIs.
Generate and execute SQL DDL statements for row-level filters and column-level masking based on data contract definitions.
Design and maintain a contract index, including recovery and atomic maintenance mechanisms.
Implement dead-letter asset deployment patterns and ensure deployment consistency.
Establish and enforce Git workflows, including branching models, merge controls, and promotion gates.
Develop schema validation frameworks, smoke testing processes, and idempotent deployment verification.
Platform Observability & Monitoring
Build platform health dashboards and operational alerting systems using Databricks system tables.
Design freshness monitoring using Delta transaction log analysis.
Implement SLA/SLO monitoring and compliance validation against contractual data commitments.
Develop dead-letter accumulation monitoring and other critical platform health indicators.
Design monitoring and alerting for:
SCIM synchronization latency
Deployment ordering failures
Emergency access ("break-glass") activations
Develop cost attribution and reporting capabilities.
Build data quality monitoring solutions using Unity Catalog quality constraints and expectation-based frameworks.
Platform Infrastructure
Contribute to Databricks workspace administration and deployment activities.
Collaborate with cloud platform and infrastructure teams to improve overall platform operations and scalability.
Required Qualifications
5+ years of experience in Data Engineering, Data Platform Engineering, or DataOps.
3+ years of hands-on experience with Databricks, including:
Unity Catalog
Workspace administration
Databricks SDK
REST APIs
Strong Python programming skills, including:
Automation development
YAML processing
Code generation
API integrations
Solid software engineering fundamentals, including:
Modular architecture
Code maintainability
Testing methodologies
Error handling
Documentation
Experience with Declarative Automation Bundles or similar infrastructure-as-code deployment frameworks in Databricks environments.
Strong understanding of governance and access control models, including:
RBAC
Schemas
Grants
Column masking
Row filtering
Experience building CI/CD pipelines, preferably using Azure DevOps and Azure Pipelines.
Experience designing and maintaining:
Monitoring platforms
Operational dashboards
Alerting solutions
Reporting systems
Preferred Qualifications
Strong SQL skills.
Familiarity with the Open Data Contract Standard (ODCS) or equivalent data contract frameworks.
Experience working with Delta Lake internals, including:
Transaction log analysis
Audit trails
Schema evolution
Experience implementing data quality frameworks and SLA/SLO monitoring.
Knowledge of:
Azure Entra ID
SCIM provisioning
Workload identity federation
Experience designing enterprise Git workflows and release management processes.
Experience supporting IT operational domains such as:
Asset management
License management

Work arrangement
Yes