Data Engineer (Pyspark)

Job Description: Data Engineer – PySpark
Company: ZAGENO
Role: Data Engineer – PySpark / Data Platform
Experience: 3+ years
Employment Type: Full-time
Primary Skills: Python, SQL, PySpark, Databricks, Delta Lake, Apache Airflow, Change Data Capture (CDC)
Location: Bangalore
Notice Period Preference: Candidates who can join within 30 days are preferred.
About ZAGENO
ZAGENO is a digital platform focused on simplifying the procurement of life-sciences research products. Reliable, accurate, and well-structured product data is essential to helping researchers and life-sciences organizations discover and procure the products they need.
We are looking for a hands-on Data Engineer with strong PySpark expertise and proven production pipeline ownership to strengthen the reliability, scalability, and correctness of our data platform.
In this role, you will work on the data pipelines and systems that power our life-sciences product catalog. You will address existing pipeline failures and technical debt, build efficient data synchronisation mechanisms, improve product data quality, and develop maintainable data products for business and technical stakeholders.
This is an engineering role for someone who enjoys solving real production problems, taking ownership from design through deployment, and building reliable systems that scale.
Role Overview
As a Data Engineer, you will own the design, development, testing, deployment, and operational reliability of production-grade data pipelines.
You will work with large and complex datasets using PySpark, Python, SQL, Databricks, Delta Lake, and Apache Airflow. You will also build and maintain change data capture (CDC) workflows, low-latency APIs, and data synchronisation processes.
A key part of the role involves transforming inconsistent product and supplier data into standardised, accurate, and reusable datasets. You will collaborate with Catalog Operations, Analytics, Data Science, and other business stakeholders to translate evolving requirements into robust engineering solutions.
We are looking for an engineer who can independently investigate failures, identify causes, implement lasting fixes, and continuously improve data quality and system performance.
Key Responsibilities
1. Production Data Pipeline Engineering
Design, develop, maintain, and optimize scalable data pipelines using PySpark, Python, and SQL.
Take primary ownership of production pipelines, from data ingestion and transformation through validation, deployment, and ongoing support.
Troubleshoot unreliable pipelines, resolve recurring failures, and eliminate fragile implementations and manual workarounds.
Improve pipeline performance, execution efficiency, reliability, and maintainability.
Implement incremental processing, efficient transformations, and appropriate data partitioning strategies.
Develop reusable components and engineering patterns that reduce duplication and technical debt.
2. Databricks and Delta Lake
Build and maintain data processing workflows on Data bricks using PySpark.
Work with Delta Lake tables and implement reliable data storage, transformation, and update patterns.
Apply appropriate incremental processing and table optimization techniques.
Maintain data consistency and correctness across pipeline runs

Work arrangement
Hybrid