Azure Data Engineer

Overall Experience: 6 to 12 Years
Job Location: Bangalore/Hyderabad/Chennai
Job Requirements*
Azure data factory, Databricks, synapse, delta lake development with hands-on coding experience
Implement ETL solution to integrate, transform and load data from various sources into data lake and data warehouse
Hands-on expertise in python, pyspark and sql for large scale data processing
Optimize and tune data pipelines for performance and scalability
Ability to write complex SQL queries
Collaborate with business analysts and business stakeholders to gather requirements and ensure data quality and availability.
Good understanding of Agile Methodologies and DevOps Culture
Strong Problem-solving skills
.
Key Responsibilities*
Design, develop, and deploy scalable data pipelines using Databricks (PySpark, Spark SQL), Azure Synapse, Azure Data Factory, and other Azure data services.
Implement ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
Optimize and tune data pipelines for performance and scalability.
Write and optimize complex SQL queries for data extraction, transformation, and analysis.
Use PySpark for large-scale data processing and analytics.
Implement data partitioning, bucketing, z-ordering, liquid clustering and indexing strategies for efficient data retrieval.
Integrate data from multiple sources, including structured, semi-structured, and unstructured data.
Work with APIs, streaming data, and batch processing to ensure seamless data integration.
Implement data governance practices to ensure data quality, consistency, and security.
Monitor and troubleshoot data pipelines to ensure data accuracy and availability.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
Work closely with DevOps teams to deploy and monitor data pipelines in production environments.
Document data pipelines, workflows, and processes for knowledge sharing and future reference.
Maintain up-to-date documentation on data architecture and data models.

Work arrangement
No