Python Developer

PySpark Developer Job DescriptionAbout The Company
Tata Consultancy Services (TCS) is a global leader in IT services, consulting, and business solutions. With a rich history spanning over five decades, TCS has established itself as a trusted partner for organizations worldwide, delivering innovative technology solutions that drive digital transformation. The company’s extensive portfolio includes services across various industries such as banking and finance, healthcare, manufacturing, and retail. TCS is committed to fostering a diverse and inclusive work environment, emphasizing continuous learning and development to empower its employees. As a pioneer in the IT sector, TCS leverages cutting-edge technologies and industry best practices to help clients achieve their strategic objectives and stay ahead in a competitive marketplace.
About The Role
We are seeking a highly skilled PySpark Developer to join our dynamic data engineering team. The ideal candidate will have a strong background in Python, PySpark, SQL, and data warehousing concepts. This role involves designing, developing, and maintaining large-scale data processing pipelines and ETL solutions utilizing Spark-based technologies. The successful candidate will work closely with cross-functional teams, including data engineers, analysts, and business stakeholders, to deliver scalable, reliable, and efficient data solutions. The role offers an exciting opportunity to contribute to complex data projects, optimize data workflows, and implement best practices in data processing and management.
Qualifications
The ideal candidate should possess a combination of technical expertise and practical experience, including:
Proven hands-on experience in Python and PySpark development.
Expertise in Spark SQL, DataFrames, and RDDs for large-scale data processing.
Strong knowledge of Hadoop ecosystem components such as Hive, HDFS, and YARN.
Excellent SQL skills with a focus on query optimization and performance tuning.
Understanding of data warehousing concepts and architecture.
Proficiency with data formats like Parquet, Avro, and JSON.
Experience with version control systems, particularly Git.
Familiarity with scheduling and workflow management tools such as Airflow or Oozie.
Knowledge of cloud platforms including AWS, Azure, or GCP is an added advantage.
A degree in Computer Science, Information Technology, or related fields is preferred.
Responsibilities
The key responsibilities of the PySpark Developer include:
Designing, developing, and maintaining scalable ETL/ELT pipelines using PySpark to handle large datasets efficiently.
Building optimized Spark jobs focused on performance, reliability, and scalability to meet business needs.
Processing and transforming large datasets utilizing Spark SQL, DataFrames, and RDDs for analytical and operational purposes.
Developing both batch and real-time data processing solutions to support various data-driven initiatives.
Integrating data pipelines with various data platforms such as Hive, HDFS, Snowflake, and Redshift to ensure seamless data flow.
Collaborating with data engineers, analysts, and business stakeholders to understand requirements and deliver effective solutions.
Monitoring data pipelines, troubleshooting issues, and ensuring adherence to service level agreements (SLAs).
Following coding best practices, implementing version control, and participating in CI/CD processes for continuous improvement.
Working with Hadoop ecosystem tools and leveraging cloud platforms like AWS, Azure, or GCP for scalable data solutions.
Benefits
Joining TCS as a PySpark Developer offers numerous benefits, including:
Competitive salary packages aligned with industry standards.
Comprehensive health insurance coverage for employees and their dependents.
Opportunities for continuous learning and professional development through training programs and certifications.
Exposure to cutting-edge technologies and innovative projects across diverse industries.
Flexible work arrangements to promote work-life balance.
Inclusive and collaborative work environment fostering diversity and innovation.
Employee engagement initiatives and recognition programs that reward outstanding performance.
Equal Opportunity
Tata Consultancy Services is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate based on race, religion, gender, sexual orientation, age, disability, or any other characteristic protected by law. We believe that a diverse workforce enhances our ability to innovate and deliver exceptional solutions to our clients. All qualified applicants will receive consideration for employment without regard to any protected status.

Work arrangement
Yes