Live opening · Posted 1 day ago
At a glance
The key details from the original listing.
Your early-applicant advantage
Live timing from JobBeeper.
About the role
Description supplied by the original job listing.
Responsibilities:
Design and develop scalable batch and near-real-time ETL/ELT pipelines using Snowflake (AWS) and Apache Spark (PySpark, Spark SQL, Structured Streaming).
Build structured streaming pipelines using Kafka and Spark Structured Streaming.
Design dimensional data models (Fact/Dimension, SCD Type 2).
Orchestrate pipelines using Databricks Workflows / Apache Airflow.
Integrate CI/CD pipelines using Jenkins, Git, Bitbucket/GitHub for automated deployment across DEV/UAT/PROD.
Requirements:
Hands-on experience in Delta Lake optimisation and incremental processing strategies.
Good to have: Apache Spark (Core, Structured Streaming), PySpark, Databricks (AWS/Azure), Advanced SQL DevOps & CI/CD Jenkins Git/GitHub/Bitbucket Programming Python SQL Cloud & Infrastructure AWS (S3 EMR, EC2 IAM, CloudWatch) Databricks Runtime & Cluster Management- Preferred Streaming & Integration Apache Kafka Snowflake integration (preferred) Airflow (preferred).
Experience in financial services, payouts, or enterprise data platforms.
Hands-on experience in Delta Lake optimisation and incremental processing strategies.
Experience with Snowflake data warehousing.
Strong understanding of distributed computing principles.
Experience
8-10 yrs
More openings worth a look
Recently tracked roles with full details and direct application links.