Databricks Engineer

Hi,
We have a 12-18 month requirement for Databricks Engineer – Azure & AI for the location - REMOTE.
ONLY GC/USC
Job Title: Databricks Engineer – Azure & AI
Location: REMOTE
Job Type: Contract 12-18 months (possibilities of further extension)
Hourly Rate: $DOE
Position Overview
We are seeking an experienced Senior Databricks Engineer with deep expertise in Azure cloud data engineering, large-scale data processing, and enterprise analytics. The ideal candidate will design, develop, optimize, and maintain scalable data platforms handling approximately 50 TB of data, 10,000+ tables, and 300 dashboards.
The candidate should have strong experience with Azure Databricks, Delta Lake, Apache Spark, data pipeline architecture, performance tuning, and enterprise-scale data governance. Experience integrating AI capabilities using Claude (Anthropic), Google Vertex AI, or similar large language model (LLM) platforms is highly desirable.
Key Responsibilities
Design, build, and maintain scalable data engineering solutions using Azure Databricks and Microsoft Azure services.
Develop and optimize ETL/ELT pipelines to process and manage approximately 50 TB of enterprise data.
Manage and optimize data architecture involving 10,000+ tables across multiple source systems.
Support, enhance, and optimize approximately 300 business intelligence dashboards and their underlying data pipelines.
Develop robust data ingestion, transformation, aggregation, and data quality frameworks using PySpark, Spark SQL, and Python.
Implement lakehouse architecture using Delta Lake, Delta tables, and scalable data storage patterns.
Optimize Spark jobs, cluster configurations, partitioning, caching, and query execution to improve performance and control costs.
Integrate Azure Data Factory, Azure Data Lake Storage Gen2, Azure Key Vault, and other relevant Azure services with Databricks.
Architect integration patterns with cloud services (e.g., Azure ADLS Gen2 / AWS S3, Azure Synapse / Snowflake, Kafka, Event Hubs, Power BI / Tableau).
Build reliable data models and curated datasets to support Power BI and other enterprise reporting platforms.
Implement data governance, access controls, lineage, monitoring, and security using Unity Catalog and applicable Azure security services.
Troubleshoot complex production issues, data pipeline failures, performance bottlenecks, and data inconsistencies.
Design and implement AI/LLM-powered data solutions using Claude, Vertex AI, or comparable AI platforms, where applicable.
Develop AI-assisted data discovery, metadata search, natural-language-to-SQL, and analytics automation capabilities.
Collaborate with data architects, BI developers, analytics teams, AI engineers, and business stakeholders across distributed teams.
Establish best practices for CI/CD, version control, automated testing, deployment, and production support.
Required Technical Skills
Azure & Databricks
Azure Databricks and Databricks Lakehouse Platform
Azure Data Factory (ADF)
Azure Data Lake Storage Gen2 (ADLS)
Delta Lake and Delta Live Tables / Lakeflow Declarative Pipelines
Unity Catalog and data governance
Data Engineering
Apache Spark, PySpark, Spark SQL, and Python
Advanced SQL, query optimization, and data modeling
ETL/ELT pipeline development and orchestration
Large-scale data processing and distributed computing
Data quality, reconciliation, and pipeline monitoring
Enterprise Data Scale
Experience handling multi-terabyte datasets, ideally around 50 TB or more
Experience managing complex data estates with thousands of tables
Experience supporting large BI environments with hundreds of dashboards
Performance optimization, workload management, scalability, and cost optimization
AI & LLM Technologies
Experience integrating LLMs into data engineering or analytics workflows
Familiarity with Anthropic Claude APIs and/or Google Vertex AI
Prompt engineering, API integration, and secure handling of AI requests
Experience with RAG, vector search, metadata-driven data discovery, or natural-language analytics is a plus
DevOps & Security
Git, CI/CD pipelines, and automated deployments
Azure DevOps or GitHub Actions
Identity and access management, secrets management, and secure data access
Production monitoring, logging, and incident resolution
Preferred Qualifications
Experience migrating legacy data warehouses or on-premises platforms to Azure Databricks.
Experience with Power BI semantic models, datasets, refresh pipelines, and dashboard performance.
Experience with Microsoft Purview or comparable data catalog and governance platforms.
Familiarity with enterprise data architecture, dimensional modeling, and medallion architecture.
Experience working with regulated or security-sensitive enterprise data.
Relevant Azure or Databricks certifications.
Success Criteria
Reliable processing and transformation of approximately 50 TB of data.
Effective management and optimization of a data environment containing 10,000+ tables.
Stable, performant data pipelines supporting approximately 300 dashboards.
Improved data quality, pipeline reliability, query performance, and cloud cost efficiency.
Secure and scalable integration of AI/LLM capabilities into enterprise data workflows.

Work arrangement
Yes