Data Engineer - Python | PySpark

Job Title: Data Engineer - Python | PySpark
Experience: 7+ years
Job Type: India, Full-Time, Remote
Innovya Technologies is a fast‑growing software consulting firm specializing in business automation and modern digital solutions. We enable organizations to accelerate value realization from their technology and transformation initiatives through expert, context‑driven product development and high‑quality engineering services.
About the Role:
We are looking for an experienced PySpark Developer with deep expertise data streaming development and deployment to join our dynamic team. A Python developer who will play a key role in maintaining, enhancing, and modernizing critical high volume and real time data pipelines. The ideal candidate will have a strong background in building scalable, fault-tolerant, and high-performance ELT applications, with a keen interest in AI enabled development.
You will work closely with cross-functional teams to design, analyse, build and deploy complex systems, providing technical direction and expertise to ensure the delivery of robust, efficient, and scalable solutions.
Key Responsibilities:
Design, develop, and optimize scalable data pipelines using PySpark
Process and analyze large datasets in distributed environments
Collaborate with data engineers, analysts, and stakeholders to deliver data solutions
Write efficient, reusable, and reliable PySpark code
Perform data cleansing, transformation, and validation
Optimize Spark jobs for performance and cost efficiency
Integrate data from various sources (databases, APIs, streaming platforms, etc.)
Troubleshoot and debug data pipeline issues
Maintain documentation for data workflows and processes
Required Skills:
7+ years of total experience in Python development, with at least 5 years in Apache Spark (PySpark).
Hands-on experience with Python and PySpark in production data engineering environments.
Use Spark DataFrames for large-scale data transformation and processing.
Experience with complex aggregations, window functions, conditional aggregations, and performance tuning in Spark.
Good understanding of Spark execution concepts such as partitions, shuffles, join behavior, and optimization techniques.
Experience with API integrations, including pagination, request handling, response transformation, and fault tolerance.
Working knowledge of AWS, preferably with services such as S3, Lambda, Glue, EMR, Athena, Step Functions, or similar.
Experience handling JSON and schema-based parsing in distributed data pipelines.
Familiarity with SQL-based data extraction and transformation.
Ability to write production-quality, maintainable, and testable code.
Good analytical and problem-solving skills, with the ability to understand complex data flows and business rules.
Experience identifying bugs, troubleshooting processing issues, performing root cause analysis, and implementing fixes in distributed systems.
Ability to create and maintain unit tests to improve code quality and reduce regressions.
Preferred - 2+ years of DataBricks and Data Lake development experience

Work arrangement
No