DATA ENGINEER - Databricks

Job Description: Senior Databricks Data Engineer
Role Overview
We are looking for an experienced Senior Databricks Data Engineer to design, develop, and maintain scalable ETL/ELT data pipelines using Databricks, PySpark, Spark SQL, and Delta Lake. The candidate should have strong hands-on experience in data transformation, pipeline orchestration, performance optimization, data quality, and production support within an AWS-based data engineering environment.
The role also requires practical experience using Cursor, GitHub Copilot, Windsurf, or similar agentic AI-enabled development tools to accelerate development, testing, debugging, documentation, and code optimization.
Key Responsibilities
Design, develop, test, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark.
Develop and optimize data transformation logic using PySpark, Spark SQL, and SQL.
Build and maintain Delta Lake tables following Bronze, Silver, and Gold layers of the Medallion Architecture.
Implement Delta Lake capabilities, including:
MERGE and upsert operations
Schema enforcement and schema evolution
Time Travel and version management
OPTIMIZE and VACUUM
Incremental and change-based data processing
Develop and manage Databricks Workflows and Jobs for pipeline orchestration, scheduling, dependency management, retries, and alerting.
Design efficient incremental-load and restart/recovery mechanisms to ensure reliable pipeline execution.
Troubleshoot pipeline failures, performance bottlenecks, data-quality issues, and production incidents.
Optimize Spark workloads through effective use of partitioning, joins, caching, broadcast strategies, shuffle management, file sizing, and query execution plans.
Develop reusable data engineering utilities, libraries, and frameworks using PySpark and Python.
Write complex SQL queries for data transformation, reconciliation, validation, analysis, and reporting.
Integrate Databricks pipelines with external applications and systems using REST APIs.
Develop and support inbound and outbound file integrations using SFTP.
Process structured and semi-structured data in formats such as CSV, JSON, Parquet, and Delta.
Work with AWS services used in the data engineering ecosystem, particularly Amazon S3.
Implement exception handling, logging, auditing, monitoring, alerting, and operational recovery mechanisms.
Implement data-quality validations and reconciliation controls across different pipeline stages.
Participate in code reviews and ensure adherence to coding, security, performance, and data engineering best practices.
Collaborate with architects, business analysts, source-system teams, QA teams, DevOps teams, and other engineering stakeholders.
Support production deployments, release validation, operational monitoring, and troubleshooting of production data pipelines.
Create and maintain technical documentation, pipeline specifications, operational procedures, and support runbooks.
Use Cursor or similar agentic AI-enabled IDEs to improve development productivity, including:
Generating and refactoring PySpark, Python, and SQL code
Creating unit tests and data-validation scripts
Troubleshooting errors and performance issues
Generating technical documentation and code explanations
Reviewing and validating AI-generated code for correctness, security, maintainability, and performance
Must-Have Skills
Strong hands-on experience in Databricks-based data engineering.
Advanced proficiency in PySpark, Spark SQL, and SQL.
Strong understanding of Apache Spark architecture and distributed data processing.
Hands-on experience building ETL/ELT pipelines at enterprise scale.
Strong experience with Delta Lake and Medallion Architecture.
Practical experience with Delta Lake features such as MERGE, schema evolution, Time Travel, OPTIMIZE, and VACUUM.
Hands-on experience developing and managing Databricks Workflows and Jobs.
Strong understanding of Spark performance optimization, including:
Data partitioning
Join strategies
Caching and persistence
Shuffle optimization
Data skew handling
Query execution plans
Small-file management
Strong SQL skills, including complex joins, window functions, aggregations, reconciliation, and data-validation queries.
Working knowledge of Python for developing reusable utilities, frameworks, and automation scripts.
Experience working with CSV, JSON, Parquet, and Delta file formats.
Hands-on experience integrating data pipelines with REST APIs and SFTP systems.
Experience working with Amazon S3 in a data engineering environment.
Experience implementing logging, exception handling, auditing, monitoring, alerting, and restart/recovery mechanisms.
Experience troubleshooting and supporting production data pipelines.
Hands-on knowledge of Cursor, GitHub Copilot, Windsurf, or another agentic AI-enabled development environment.
Ability to write effective prompts, review AI-generated code, identify hallucinated or incorrect logic, and apply organizational security and data-privacy standards while using AI tools.
Strong problem-solving, debugging, communication, and stakeholder-collaboration skills.
Preferred Skills
Knowledge of Databricks Unity Catalog, access controls, lineage, and data governance.
Experience with Databricks Asset Bundles, Databricks CLI, or CI/CD-based deployment approaches.
Basic understanding of AWS IAM, Secrets Manager, CloudWatch, and Lambda.
Experience with cloud security, secrets management, and credential-handling best practices.
Knowledge of automated testing frameworks for PySpark and data pipelines.
Familiarity with Git-based development, branching strategies, pull requests, and CI/CD pipelines.
Experience working in Agile/Scrum delivery environments.
Databricks or AWS certification would be an advantage.
Education And Experience
Bachelor?s degree in Computer Science, Information Technology, Engineering, or a related discipline.
Relevant professional experience in data engineering, including substantial hands-on experience with Databricks, PySpark, SQL, and Delta Lake.
Experience delivering and supporting enterprise-scale data platforms in production environments.
Databricks, SQL

Work arrangement
No