Live opening · Posted 7 hours ago
The key details from the original listing.
Live timing from JobBeeper.
Description supplied by the original job listing.
Dear Professionals,
Greetings from Tata Consultancy Services (TCS)!!!
Job Title : Pyspark Data Engineer
Experience : 6- 10 Years
Location: Hyderabad/ Pune/ Chennai/ Bangalore
Job Requirements ;
We are seeking an experienced AWS Data Engineer to design, develop,
and maintain scalable data solutions on the Amazon Cloud Platform
(AWS). The ideal candidate will have strong expertise in building
modern data pipelines, data warehousing, big data processing, and
cloud-native analytics solutions. The role requires close collaboration
with business stakeholders, data architects, data scientists, and
application teams to deliver reliable and high-performance data
platforms.
Key Responsibilities
• Design, develop, and optimize scalable data pipelines using
AWS services.
• Build and maintain batch and real-time data ingestion and
processing frameworks.
• Develop enterprise-grade data warehousing solutions using PySpark.
• Convert Scala-based ETL, batch, and streaming pipelines into
PySpark frameworks.
• Optimize PySpark jobs for performance, scalability, and
resource utilization.
• Support cloud modernization initiatives on AWS/Databricks
platforms.
berribot jd version 3
• Implement ETL/ELT processes for structured and unstructured
data.
• Integrate data from multiple sources including databases, APIs,
files, and streaming platforms.
• Ensure data quality, governance, security, and compliance
across data platforms.
• Automate deployment and operational processes using CI/CD
and Infrastructure as Code (IaC).
• Monitor data pipelines and troubleshoot production issues.
• Collaborate with Data Architects, Business Analysts, and Data
Scientists to translate business requirements into technical
solutions.
• Implement data models, metadata management, and data
lineage best practices.
• Support migration of on-premises or multi-cloud data platforms
to AWS.
• Lead the migration, modernization, and optimization of Apache
Spark workloads on AWS EMR, including conversion of Scalabased Spark applications to PySpark, performance tuning,
cluster optimization, dependency management, and ensuring
scalable, cost-effective, and resilient data processing solutions.
Required Technical Skills
• PySpark
• SparkSQL
• AWS Glue
• AWS S3
• Step Functions
• Glue
• Lambda
• Pub/Sub
• Python
• SQL (Advanced)
• Event Bridge
• ECS
Work arrangement
No
Recently tracked roles with full details and direct application links.