Live opening · Posted 15 hours ago
The key details from the original listing.
Live timing from JobBeeper.
Description supplied by the original job listing.
Hiring for: A technology company delivering enterprise data engineering and analytics solutions, with a strong focus on modern cloud data platforms and Databricks.
Role: Senior Databricks Data Engineer - PySpark
Positions: 10
Experience: 4 to 7 years
Location(s): Pune / Bangalore (Preferred) / Remote
Type: On-site, Remote / Permanent
Salary: Up to INR 25 LPA
Notice Period: Immediate to 30 days
Must Have: Databricks Development, Pyspark, Python, SQL, Unity Catalogs,
Job Summary
We are seeking an experienced PySpark ETL Developer with 4+ years of experience in designing, developing, and optimizing enterprise ETL pipelines using PySpark. The ideal candidate should have strong expertise in Python, Apache Spark, Databricks, and Snowflake, along with hands-on experience in processing large-scale data and building scalable data engineering solutions. You will work closely with business stakeholders and cross-functional teams to develop high-performance data pipelines that support business intelligence and analytics initiatives.
Key Responsibilities
Design, develop, and maintain scalable ETL pipelines using PySpark and Spark SQL.
Collaborate with stakeholders to gather business requirements and translate them into efficient data engineering solutions.
Extract data from multiple sources, including databases, APIs, data lakes, files, and streaming platforms.
Transform, cleanse, and validate data using PySpark to ensure high data quality and consistency.
Develop and optimize Spark jobs for performance, scalability, and efficient resource utilization.
Build and maintain batch and streaming data pipelines to support real-time and near real-time processing.
Load transformed data into data lakes, data warehouses, and analytical platforms.
Implement robust error handling, logging, monitoring, and troubleshooting mechanisms for ETL workflows.
Document ETL processes, data lineage, transformation logic, and technical specifications.
Develop unit, integration, and performance tests to ensure reliable and accurate data processing.
Collaborate with cross-functional teams to deliver scalable, secure, and high-quality data solutions.
Required Skills
4+ years of experience as a Data Engineer or ETL Developer.
Strong hands-on experience with PySpark, Apache Spark, Spark SQL, and Python.
Expertise in Databricks.
Strong understanding of ETL design, data transformation, and data integration.
Experience with Data Lakes, Data Warehouses, and Big Data technologies.
Knowledge of distributed computing, parallel processing, and data partitioning concepts.
Strong SQL skills with experience in performance tuning and query optimization.
Experience working with batch and streaming data processing.
Excellent analytical, debugging, and problem-solving skills.
Strong verbal and written communication skills.
Preferred Skills
Experience with cloud platforms such as AWS or Azure.
Knowledge of Hadoop ecosystem and modern data engineering technologies.
Experience with API integration and data ingestion from multiple data sources.
Familiarity with CI/CD pipelines and Agile development methodologies.
Exposure to enterprise-scale data engineering projects.
Work arrangement
No
Recently tracked roles with full details and direct application links.