Databricks Engineer

We are looking for a highly skilled Databricks engineer to design, build, optimise, and maintain large-scale data processing pipelines and analytics workloads on the Databricks platform. This role requires strong hands-on experience with Databricks Jobs, Delta Tables, Pipelines, Spark SQL, AWS integrations, Genie workflows, and Databricks features such as Unity Catalogue, Delta Lake optimisations, workflow orchestration, and advanced cluster management. The ideal candidate has both data engineering and cloud engineering expertise.
The candidate will have responsibilities across the following functions:
Databricks Platform Engineering:
Build, configure, and maintain Databricks jobs, workflows, pipelines, and Delta tables.
Develop scalable high-performance ETL pipelines using PySpark, SQL, and Databricks notebooks.
Apply optimisation techniques - SQL queries, Delta Lake tables, partitioning strategies, and liquid clustering.
Ensure workflows scale efficiently and maintain reliability.
Manage cluster performance, job debugging, and resource auto-scaling.
Document workflows and data models.
Build, publish and maintain Databricks dashboards.
Apply the Databricks backup and restore methods based on requirements.
Security, Governance and Compliance:
Implement Unity Catalogue, data lineage, access controls, and governance best practices.
Manage secrets, tokens, and secure integrations.
Ensure compliance with enterprise security and data governance standards.
AWS and Databricks Integration:
Integrate Databricks with AWS services (S3 EC2 IAM).
Manage roles and secure data lake permissions.
Optimise cluster configuration for cost, performance, and scalability on AWS.
Requirements:
Technical Skills: Working with Genie, debugging skills, CI/CD experience and good understanding of networking and security practices.
Soft Skills: Problem-solving, ownership mindset and communication.
Nice-to-Have Skills:
AI and machine learning capabilities in Databricks.

Experience
5-9 yrs