Databricks data engineer

Senior Databricks Data Engineer
Item Detail
Role level Senior, technical lead for engineering standards
ToR qualification Fulfils the mandatory "Databricks-certified Data Engineer with production Lakehouse, medallion and SDP / Lakeflow experience"
Role summary. Owns the engineering standards in the Way of Working: medallion layers, Lakeflow pipelines, Workflows orchestration and CI/CD. In Phase 2, leads automated read-only harvesting of the Oracle estate and builds the dependency matrix.
Responsibilities – Phase 1 (Way of Working)
• Define medallion standards: Bronze/Silver/Gold naming conventions, schema design, data contracts, layer-progression rules and transformation boundaries.
• Define pipeline development standards for Lakeflow Declarative Pipelines (SDP): coding, modularization, reusability patterns, testing expectations and minimum documentation.
• Produce worked examples of pipeline patterns, naming and catalog structure on representative UNICEF domains (Finance & Operations, Supply Chain, HR & People, Programme Data) – not generic templates.
• Design the Databricks Workflows model that replaces crontab: dependency management, retry and failure handling, alerting, observability and data-freshness SLAs by layer.
• Lead the CI/CD framework: branching strategy, Databricks Asset Bundles configuration, Azure DevOps deployment pipelines, environment promotion, approval gates and traceability.
• Contribute to environment and workspace design (dev / pre-prod / prod) with the Solution Architect.
Responsibilities – Phase 2 (Migration Assessment)
• Run read-only harvesting of Oracle DW metadata, DDL and code; secure UNICEF security approval for any tooling.
• Build the rules-based first-pass classification (Deprecate / Lift & Shift / Refactor) and its confidence scoring.
• Parse PL/SQL, views and pipelines to build the dependency matrix across systems, datasets and processes.
• Support complexity banding and effort estimation for retained assets, including the 128 complex procedures.
Deliverables owned
• Medallion & Pipeline Development Standards (with worked examples)
• Orchestration & Monitoring Strategy
• CI/CD Framework Specification
Required qualifications
• 8+ years in data engineering, including 3+ years on Databricks in production Lakehouse implementations.
• Databricks Certified Data Engineer Associate (minimum); Professional preferred.
• Hands-on medallion architecture design and Delta Lake (schema evolution, CDC, MERGE, optimization).
• Lakeflow Declarative Pipelines / Delta Live Tables, including expectations-based data quality.
• Databricks Workflows orchestration: task dependencies, retries, alerting, job clusters.
• CI/CD with Azure DevOps and Databricks Asset Bundles; Git branching and environment promotion.
• Strong SQL and PySpark; ability to read and analyse Oracle PL/SQL packages and procedures.
• Experience writing engineering standards that other teams adopt.
Preferred
• Oracle-to-Databricks or SAP HANA-to-Databricks migration or assessment experience.
• Exposure to SAP Datasphere, Azure Data Factory and Unity Catalog.
• Prior UN, NGO or public-sector engagement; experience presenting to client architecture boards.
Behaviours
• Teaches as they build: documents every design decision with its rationale.
• Writes clearly for technical teams; comfortable defending standards in workshop settings.

Skills
Databricks

Experience
8+ yrs

Work arrangement
No