Lead/Senior Databricks Data Engineer (10+ Experience)

Lead/Senior Databricks Data Engineer / Databricks Architect
We are looking for an experienced Lead Databricks Data Engineer / Databricks Architect with 12+ years of overall experience in Data Engineering and strong hands-on expertise in Databricks, Apache Spark, PySpark, SQL, Delta Lake, and cloud-based data platforms.
The candidate will be responsible for designing and implementing scalable Lakehouse architectures, enterprise data pipelines, data integration solutions, governance frameworks, and high-performance analytics platforms using Databricks.
Interested Professionals please share your updated CV to suchi.n@sdhsystems.com
Job Title: Senior Databricks Engineer
Location: 100% Remote
Visa: Any
Interview Process: Video (2 Rounds)
Work Schedule: 100% Remote
Job Description:
Key Responsibilities
Design and develop scalable data engineering solutions using Databricks and Lakehouse architecture.
Build robust ETL/ELT pipelines using PySpark, Spark SQL, Python, and SQL.
Design and implement Bronze, Silver, and Gold/Medallion architecture.
Develop and optimize Delta Lake tables, including MERGE, schema evolution, Change Data Feed, and incremental processing.
Build batch and real-time/streaming pipelines using Structured Streaming, Auto Loader, and Lakeflow.
Develop and manage Databricks Jobs/Workflows for pipeline orchestration, scheduling, dependencies, retries, and monitoring.
Implement enterprise data governance using Unity Catalog, including access control, data lineage, auditing, catalogs, schemas, and external locations. Unity Catalog provides centralized governance, access control, lineage, and auditing across Databricks data and AI assets. (Databricks Docs)
Perform Spark and Databricks performance tuning, including cluster configuration, partitioning, caching, query optimization, Photon, and workload optimization.
Design data models supporting Data Warehousing, BI, Analytics, and AI/ML workloads.
Integrate Databricks with cloud platforms such as AWS, Azure, or GCP.
Work with cloud services such as AWS S3, Azure ADLS Gen2, Azure Data Factory, AWS Glue, Synapse, Event Hubs/Kafka/Kinesis, as applicable.
Implement CI/CD pipelines using Git, Azure DevOps/GitHub/Jenkins and Databricks deployment capabilities.
Work with Terraform/IaC for infrastructure provisioning and automation.
Troubleshoot production pipeline failures, performance issues, data-quality problems, and Spark/cluster issues.
Establish data quality, monitoring, logging, and observability practices.
Provide technical leadership, code reviews, architecture guidance, and mentorship to junior/mid-level engineers.
Collaborate with Data Architects, Data Scientists, Business Analysts, DevOps teams, and application teams.
MLflow is particularly useful if the role touches ML/AI, as Databricks supports model tracking, lifecycle management, and deployment workflows alongside governed data. (Databricks Docs)
Required Technical Skills
Databricks
Databricks Lakehouse Platform
Delta Lake
Unity Catalog
Databricks Workflows/Jobs
Lakeflow / Delta Live Tables
Auto Loader
Databricks SQL
Databricks notebooks
Databricks Asset Bundles
Photon
Cluster/workload optimization
Big Data
Apache Spark
PySpark
Spark SQL
Structured Streaming
Kafka
Batch and real-time data processing
Programming
Python
SQL
PySpark
Scala – good to have
Cloud – Strong experience in at least one
AWS: S3, Glue, EMR, Lambda, Redshift, IAM, Kinesis
Azure: ADLS Gen2, ADF, Synapse, Azure DevOps, Event Hubs, Key Vault
GCP: GCS, BigQuery, Dataflow, Pub/Sub
Data Engineering
ETL/ELT
Data Warehousing
Dimensional Modeling
Data Lake/Lakehouse
Medallion Architecture
CDC
Data Quality
Data Governance
Metadata and Data Lineage
DevOps / CI-CD
Git
Azure DevOps / GitHub
Jenkins
Terraform
CI/CD automation
Infrastructure as Code
Preferred / Nice-to-Have Skills
MLflow
Databricks Machine Learning
Feature Store
Mosaic AI / GenAI
dbt
Apache Airflow
Power BI / Tableau
Delta Sharing
Lakehouse Federation
Liquid Clustering
Data security and PII masking
Qualifications
Bachelor's degree in Computer Science, Engineering, Information Technology, or related field.
12+ years of experience in Data Engineering / Big Data / Analytics.
4+ years of hands-on Databricks experience preferred.
Strong experience designing enterprise-scale data platforms.
Demonstrated experience leading technical projects and mentoring engineers.
Strong communication and stakeholder-management skills.

Work arrangement
Yes