Senior BigData Engineer (Databricks + Azure)

About The Role
In this role, you will build scalable, efficient data solutions on the Databricks Lakehouse platform in Azure environments. You will work on batch and streaming data pipelines, support data platform optimization initiatives, and collaborate with technical and business stakeholders to deliver reliable, production-ready data solutions across different stages of the project lifecycle.
Responsibilities
Design, develop, and maintain scalable batch and streaming ETL pipelines on the Databricks Lakehouse platform
Build and optimize data processing solutions using PySpark, Python, and SQL
Configure and maintain Databricks clusters and SQL Warehouses
Implement and support data governance practices using Unity Catalog, including access control, lineage, and cross-workspace sharing
Develop and deploy data solutions using Databricks Asset Bundles and CI/CD processes
Design and maintain bronze–silver–gold Lakehouse architectures using Delta Lake
Expose and visualize data using Databricks SQL and BI tools such as Power BI
Collaborate with technical and business stakeholders to understand requirements and deliver effective data solutions
Participate in architecture discussions and contribute to continuous improvement initiatives within the team
Support implementation and usage of modern Databricks AI/BI capabilities such as Dashboards, Metric Views, MLflow, Genie, and agents where applicable
Requirements
5+ years of experience in Data Engineering, including designing data models and scalable ETL pipelines
Strong expertise in PySpark, Python, and SQL, including performance tuning and optimization
Hands-on experience with Databricks, including clusters and SQL Warehouses
Practical knowledge of Unity Catalog governance, including fine-grained access control, lineage, and data sharing
Experience delivering end-to-end data solutions using Databricks Asset Bundles and CI/CD practices
Strong understanding of Delta Lake and Lakehouse architecture concepts
Experience working with Databricks SQL and BI tools such as Power BI
Strong communication and collaboration skills with technical and business stakeholders
Upper-intermediate or higher level of English
Familiarity with Databricks AI/BI tools such as MLflow, Dashboards, Metric Views, Genie, or agents (is a plus)
Experience with Azure data services such as ADF, Synapse, ADLS, Azure DevOps, or Microsoft Fabric (nice to have)
SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.

Work arrangement
Yes