Data Engineer - Azure

We are seeking an experienced Data Engineer to build and refine robust big data solutions. You will be responsible for designing high-performance ETL pipelines using Apache Spark across major cloud environments (Azure, AWS, or GCP), with a heavy focus on data engineering excellence and system optimization.
Responsibilities:
Lead the end-to-end design and optimization of Databricks ETL pipelines to ensure high-quality data delivery.
Develop scalable data ingestion and transformation workflows by leveraging the full power of the Apache Spark framework.
Manage complex data lifecycles involving both structured and unstructured datasets from diverse originating sources.
Fine-tune Spark jobs to balance peak performance with cost-efficiency and long-term scalability.
Establish robust CI/CD patterns and version control standards for all Databricks-based workflows.
Collaborate with stakeholders across engineering, data science, and analytics to translate business needs into technical solutions.
Enforce rigorous security protocols and access management policies within the Databricks environment.
Diagnose and resolve complex bottlenecks related to big data processing and system scalability.
Requirements:
Proven track record of developing high-performance solutions using Databricks and Apache Spark (proficient in PySpark, Scala, or Java).
Expert-level command of SQL for complex data modeling, transformation, and analytical querying.
Deep familiarity with cloud ecosystems (Azure, AWS, or GCP), with specific expertise in storage and warehousing solutions like ADLS, S3 or BigQuery.
Significant experience building end-to-end ETL/ELT pipelines and managing scalable Data Warehousing environments.
Strong grip on CI/CD pipelines, Git version control, and DevOps methodologies tailored for data engineering.
In-depth knowledge of Delta Lake and Lakehouse architecture, with a focus on advanced performance tuning and optimization.
Skilled in managing massive-scale datasets within distributed computing environments.
5-7 years of specialized experience in Databricks development and big data architecture.
Mandatory background in the Retail sector, with a deep understanding of industry-specific data challenges.
Preferred Skills:
Skilled in leveraging Databricks MLflow to track, deploy, and scale machine learning models.
Familiarity with modern data stack components such as Apache Kafka for streaming, Airflow for workflow automation, or Snowflake for cloud warehousing.
Deep understanding of data privacy standards, access control policies, and corporate compliance requirements.
Bachelor's or Master's degree in Computer Science, Data Engineering, or a strictly related technical discipline.

Experience
4-6 yrs