Data Architect

Senior Data Engineer - AWS Databricks
Experience
8+ Years
Primary Skills
AWS Databricks
PySpark
AWS Cloud
Data Engineering
ETL/ELT Pipelines
Secondary Skills
Databricks Workflows
Databricks Notebooks
Delta Lake
Unity Catalog
Auto Loader
CI/CD
DAB (Databricks Asset Bundles)
Job Description
We are looking for an experienced Senior Data Engineer with strong expertise in AWS Databricks, PySpark, and modern Lakehouse architectures. The candidate will be responsible for designing, developing, and optimizing scalable batch and streaming data pipelines while implementing enterprise-grade data solutions on AWS.
Key Responsibilities
Design, develop, and maintain data pipelines using AWS Databricks and PySpark.
Build scalable ETL/ELT solutions for structured and semi-structured data.
Develop and manage Databricks Notebooks, Workflows, and Jobs.
Implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.
Build ingestion frameworks using Auto Loader and streaming technologies.
Optimize Spark jobs for performance, scalability, and cost efficiency.
Work with AWS services such as S3, Glue, Athena, Lambda, Redshift, and IAM.
Implement data quality, validation, monitoring, and governance frameworks.
Utilize Unity Catalog for access control, lineage, and governance.
Build and maintain CI/CD pipelines for Databricks deployments.
Implement and manage Databricks Asset Bundles (DAB) for automated deployments.
Collaborate with architects, analysts, and business stakeholders to deliver scalable data solutions.
Required Skills
8+ years of Data Engineering experience.
Strong hands-on experience with AWS Databricks.
Expertise in PySpark and Spark SQL.
Experience with Databricks Notebooks, Workflows, and Delta Lake.
Strong understanding of Medallion Architecture.
Experience with AWS S3, Glue, Redshift, Athena, Lambda, and IAM.
Hands-on experience with Auto Loader and incremental data processing.
Strong SQL and data modeling skills.
Experience with Git-based version control and CI/CD implementation.
Knowledge of Databricks Asset Bundles (DAB).

Work arrangement
No