Live opening · Posted 3 days ago
The key details from the original listing.
Live timing from JobBeeper.
Description supplied by the original job listing.
Immediate joiners or candidates with 30-day notice will be preferred
Senior Data Platform Engineer (Python/Spark Platform Engineering)
We are looking for a highly skilled and self-driven Senior Data Platform Engineer with strong expertise in Python, Spark, Big Data Platforms, and Cloud Data Engineering. The ideal candidate should be capable of independently understanding existing data frameworks, enhancing them with new capabilities, and leading modernization initiatives involving cloud-native technologies.
Key Responsibilities :
Design, develop, and maintain scalable data ingestion and transformation frameworks using Python, Spark, and Shell Scripting.
Independently analyze existing frameworks and implement new features and enhancements.
Drive platform modernization through cloud-native technologies, automation, and engineering best practices.
Develop optimized integrations with object storage platforms, databases, and enterprise data systems.
Build and support high-performance data pipelines on large-scale big data platforms.
Required Skills :
10 + Years of experience of hands on experience
Strong hands-on expertise in Python and Apache Spark development and framework design.
Experience building reusable and scalable data ingestion and transformation frameworks.
Deep understanding of Hive, Impala, HDFS, and the Hadoop ecosystem, preferably on Cloudera-based platforms.
Strong knowledge of HDFS internals, data storage architecture, compaction strategies, partition management, resource utilization, and cluster performance optimization.
Proven ability to troubleshoot complex platform, storage, compute, and cluster-level issues, perform root cause analysis, and drive performance improvements.
Experience working with diverse database technologies and object storage platforms, with a focus on optimized connectivity and data access patterns.
Strong understanding of distributed data processing, Spark optimization, and platform engineering best practices.
Preferred Skills :
Strong hands-on experience with Kubernetes-based application development, containerized Spark workloads, pod management, scaling, troubleshooting, and operational support.
Experience building and deploying Spark workloads using JFrog-managed container images and CI/CD pipelines.
Hands-on experience with Databricks and modern cloud-native data platforms.
Exposure to cloud migration and enterprise data platform modernization initiatives.
Experience leveraging AI Agents, Generative AI, and LLM-powered development tools to accelerate software delivery.
Work arrangement
No
Recently tracked roles with full details and direct application links.