Live opening · Posted 3 days ago
The key details from the original listing.
Live timing from JobBeeper.
Description supplied by the original job listing.
Role- Lead Data Platform Engineer
Experience- 8 to 15 years
Location- Pune, Hyderabad, Bangalore
Required Skill set- Spark, Kafka, Flink, Airflow, Iceberg, Polaris/Nessie, Kubernetes
Required Skills & Experience
Must Have
8-15 years of data engineering or data platform experience, with substantial recent hands-on responsibility for distributed processing platforms.
Deep hands-on experience with Apache Spark, including DataFrame or SQL workloads, performance tuning, packaging, configuration, and production troubleshooting.
Strong Apache Kafka experience covering topics, partitions, consumer groups, offsets, delivery semantics, schemas, security, and operations.
Practical Apache Flink experience covering state, checkpoints, savepoints, event time, recovery, and backpressure.
Strong understanding of Apache Iceberg, Google Cloud Storage, schema and partition evolution, snapshots, compaction, and table maintenance.
Experience with Apache Polaris, Project Nessie, or REST-based Iceberg catalogs.
Hands-on experience with Apache Airflow for distributed data-processing orchestration.
Working knowledge of GKE-based execution, containers, Helm, service accounts, Workload Identity Federation, secrets, networking, storage, and observability.
Proficiency in Python, Java, or Scala, SQL, Git, automated testing, CI/CD, and reusable software engineering practices.
Strong technical leadership, troubleshooting, documentation, mentoring, and stakeholder communication skills.
Good to Have
Experience with Trino, OpenMetadata, JupyterHub, Dataproc, data-quality frameworks, lineage, or schema registries.
Exposure to Debezium, Kafka Connect, Datastream, database logs, or comparable CDC patterns.
Experience with Spark Operator, Flink Kubernetes Operator, Strimzi, KubernetesExecutor, or operator-based GKE integration.
Understanding of data security, encryption, masking, governance, data contracts, and regulated enterprise controls.
Experience with multi-tenant data platforms, regulated environments, and platform cost optimisation on GCP.
Work arrangement
No
Recently tracked roles with full details and direct application links.