PySpark Developer

Greetings from TCS,
We are Hiring for PySpark Developers
About the Role
We are looking for skilled PySpark Developers to join a global banking team. The role involves designing and developing large-scale data pipelines, implementing machine learning models, and building robust data processing solutions on Big Data platforms.
The ideal candidate will have expertise in PySpark, Scala, Hadoop ecosystem technologies, and experience within Credit Risk, Regulatory Risk, Banking, or Financial Services domains.
Key Responsibilities
Design, develop, and optimize scalable ETL/ELT pipelines using PySpark and Scala.
Build ingestion frameworks for structured, semi-structured, and unstructured data sources.
Develop and enhance machine learning model pipelines in partnership with data scientists.
Implement feature engineering, data transformation, model scoring, and retraining processes.
Build reusable Spark-based frameworks for batch and near real-time data processing.
Optimize Big Data workloads running on Spark, Hadoop, EMR, Databricks, or similar platforms.
Ensure data quality, lineage, governance, and documentation standards.
Collaborate with Risk Analytics, Data Science, and Business teams to deliver data solutions.
Support deployment, version control, CI/CD practices, and production support activities.
Mandatory Skills
5+ years of experience in Data Engineering, Big Data Development, or Software Development.
Strong hands-on experience in:
PySpark
Scala
Python
Spark
Hive
Hadoop Ecosystem
MapReduce
Unix Shell Scripting
SQL
Experience building enterprise-grade data pipelines and ingestion frameworks.
Experience with distributed computing environments such as Hadoop, Spark, Databricks, or AWS EMR.
Strong understanding of Data Lake and Data Warehouse concepts.
Experience working with Banking, Financial Services, Credit Risk, or Regulatory Risk projects.
Knowledge of relational databases and platforms such as PostgreSQL, Redshift, or equivalent.
Preferred Skills
Spark Structured Streaming
Kafka
Delta Lake
AWS Glue
Machine Learning frameworks (Spark ML, Scikit-Learn, TensorFlow, PyTorch)
Jenkins, Git, CI/CD pipelines
Data Governance and Data Lineage tools
Domain Preference
Candidates with experience in:
Credit Risk
Regulatory Risk
Risk Modeling
Consumer Banking
Wholesale Banking
Financial Services
Education
Master's or bachelor's Degree in:
Engineering
Statistics
Mathematics
Finance
Computer Science

Work arrangement
No