Lead Azure Databricks Developer ( 9+ Years)

Azure Databricks Developer
Location: India
Work Mode: 100% Remote
Shift: EST Time Zone
Experience: 9–10+ Years
Engagement: Contract
Job Overview
We are seeking an experienced Azure Databricks Developer with strong hands-on expertise in Azure Data Engineering, Databricks, PySpark, Spark SQL, Delta Lake, Azure Data Factory (ADF), and ADLS Gen2.
The successful candidate will be responsible for designing, developing, optimizing, and supporting scalable enterprise-grade data engineering solutions using modern Lakehouse Architecture. The role requires strong experience in ETL/ELT development, incremental processing, CDC, Spark performance optimization, and Azure cloud services.
Key Responsibilities
Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
Build and orchestrate robust ETL/ELT workflows using Azure Data Factory (ADF).
Develop ingestion and transformation pipelines for structured and semi-structured data using ADLS Gen2, Delta Lake, and Spark SQL.
Implement Medallion Architecture (Bronze, Silver, Gold) and modern Lakehouse design patterns.
Develop incremental data processing, CDC, MERGE operations, and SCD Type 1/Type 2 implementations.
Optimize Spark workloads using partitioning, caching, file-size optimization, and other performance-tuning techniques.
Configure and manage Databricks Jobs/Workflows, clusters, and workspace environments.
Implement data quality checks, validation, reconciliation, error handling, monitoring, and alerting processes.
Work with Unity Catalog for data governance, lineage, RBAC, and secure data access.
Integrate Databricks with ADLS, Azure SQL, APIs, JDBC sources, and external systems.
Develop and maintain CI/CD pipelines using Azure DevOps, Git, and YAML.
Implement Infrastructure-as-Code using Terraform, ARM, or Bicep.
Troubleshoot production data pipelines, perform root-cause analysis, and ensure SLA compliance.
Monitor and optimize workloads for performance, scalability, reliability, and cost efficiency.
Collaborate with architects, BI teams, data analysts, developers, and business stakeholders in an Agile/Scrum environment.
Required Skills & Qualifications
9–10+ years of experience in Data Engineering / Azure Cloud Data Development.
Strong hands-on experience with:
Azure Databricks
PySpark
Spark SQL
Delta Lake
Azure Data Factory (ADF)
ADLS Gen2
SQL
Strong understanding of Lakehouse Architecture and distributed data processing.
Proven experience developing enterprise-scale ETL/ELT pipelines.
Hands-on experience with Medallion Architecture, CDC, MERGE, and SCD Type 1/Type 2.
Strong knowledge of Apache Spark performance tuning and optimization.
Experience with Azure DevOps, Git, and YAML-based CI/CD pipelines.
Knowledge of Terraform, ARM, or Bicep for Infrastructure-as-Code.
Strong SQL development and query performance-tuning capabilities.
Experience working in Agile/Scrum environments.
Strong troubleshooting and production-support experience.
Preferred Skills
Experience with Databricks platform administration, including clusters, jobs, workflows, and workspace configuration.
Hands-on experience with Unity Catalog, RBAC, governance, and data lineage.
Experience integrating Databricks with APIs, JDBC sources, Azure SQL, and external systems.
Knowledge of monitoring and observability tools such as Datadog or Grafana.
Experience with Azure and Databricks cost optimization.
Experience supporting production-grade data platforms and resolving critical data pipeline issues.
Candidate Profile
The ideal candidate should be:
Strongly hands-on with Azure Databricks and PySpark.
Experienced in designing and supporting enterprise-scale data pipelines.
Strong in Spark optimization, SQL, Delta Lake, and Lakehouse architecture.
Comfortable working in an EST time-zone shift.
Capable of independently troubleshooting production issues and meeting defined SLAs.
Comfortable collaborating with cross-functional technical and business teams.
Experienced in working in fast-paced Agile/Scrum environments.
Technology Stack
Azure Databricks | PySpark | Spark SQL | Delta Lake | Azure Data Factory | ADLS Gen2 | Azure SQL | SQL | Unity Catalog | Azure DevOps | Git | YAML | Terraform | ARM | Bicep | Lakehouse Architecture | CDC | SCD Type 1/2
Engagement Details
ParameterDetailsRole :- Azure Databricks Developer
Experience9–10+ YearsLocationIndiaWork Mode100% RemoteShiftESTEngagementContractJoining PreferenceImmediate Joiners Preferred

Work arrangement
Yes