AWS Data Engineer

Job Title: Data Engineer - Databricks (AWS) | PySpark | Delta Lake
Location: Bengaluru, Noida, Gandhi Nagar, Pune, Chennai
Experience: 4+ Years
Job Summary
We are looking for a talented Data Engineer with strong expertise in Databricks on AWS, PySpark, Delta Lake, and modern data engineering practices. The ideal candidate will be responsible for building scalable data pipelines, implementing data governance frameworks, optimizing platform performance, and supporting enterprise analytics initiatives within the Healthcare/Life Sciences domain.
Key Responsibilities
Design, develop, and maintain scalable data pipelines using Databricks on AWS.
Build and support batch and streaming data ingestion frameworks.
Implement data quality, governance, lineage, and security controls using Unity Catalog and Databricks best practices.
Develop and optimize ETL/ELT processes using PySpark and SQL.
Design and implement Medallion Architecture (Bronze, Silver, Gold) data solutions.
Build and maintain CDC-based ingestion pipelines using AWS and Databricks services.
Optimize Spark performance, cluster utilization, and platform costs.
Implement and support Delta Lake features to ensure data reliability and governance.
Manage Structured Streaming, Auto Loader, and real-time data processing requirements.
Develop CI/CD processes using Git, Databricks Asset Bundles, and deployment automation tools.
Monitor production environments, troubleshoot issues, and ensure SLA compliance.
Collaborate with business stakeholders, analysts, and engineering teams to deliver high-quality data solutions.
Mentor junior engineers and contribute to data engineering best practices.
Required Skills & Qualifications
Technical Skills
4+ years of experience in Data Engineering.
4+ years of hands-on experience with Databricks on AWS.
Strong expertise in PySpark and Spark optimization techniques.
Experience with Delta Lake and Lakehouse architecture.
Strong SQL programming skills.
Experience with Unity Catalog and Databricks governance frameworks.
Knowledge of dimensional data modeling and Medallion Architecture.
Hands-on experience with:
Structured Streaming
Auto Loader
Change Data Capture (CDC)
Experience with AWS services:
Amazon S3
IAM
AWS DMS
Kinesis and/or MSK
AWS Glue
Experience with Git, CI/CD pipelines, and Databricks Asset Bundles.
Experience supporting production environments, performance tuning, and cost optimization.
Domain Experience
Healthcare or Life Sciences domain experience preferred.
Understanding of data governance, analytics, privacy, and regulatory compliance requirements.
Education
Bachelor's or Master's Degree in Computer Science, Information Technology, Engineering, or related field.
Flexible for candidates with strong relevant experience.
Certifications (Preferred)
Databricks Certified Data Engineer Associate/Professional
AWS Certified Data Engineer
AWS Certified Solutions Architect
Microsoft Azure Certified Developer
Google Professional Cloud Developer

Work arrangement
No