Spark Engineer

Role: Spark Engineer
Experience: 08 to 12yrs
Notice Period: 60-90 Days
Work Location: TCS-Pune, Hyderabad
Mode of Interview: Virtual
Responsibilities :-
Seeking an experienced Spark Engineer with 8–12 years of IT experience in enterprise data engineering and ETL solutions.
Strong hands-on expertise in PySpark / Apache Spark for developing and optimizing large-scale data processing applications.
Extensive experience with IBM DataStage for designing, developing, and maintaining complex ETL workflows and data pipelines.
Proficiency in Python for data processing, automation, transformation, and reusable utilities.
Strong working knowledge of Linux/Unix and Shell Scripting for job execution, automation, monitoring, and troubleshooting.
Experience in designing and implementing high-volume batch and distributed data processing pipelines.
Good understanding of Spark performance tuning, including partitioning, caching, memory optimization, and query optimization.
Strong knowledge of SQL, data transformation, data quality, and ETL/ELT concepts.
Ability to troubleshoot DataStage and Spark jobs, analyze failures, and improve pipeline performance and reliability.
Experience working with Git, CI/CD practices, production deployments, and enterprise data engineering environments is preferred.
Strong problem-solving, debugging, communication, and collaboration skills, with the ability to work with cross-functional teams.
Note : Ex TCSer can't apply

Work arrangement
No