Live opening · Posted 4 days ago
The key details from the original listing.
Description supplied by the original job listing.
Role: Lead Data Engineer (Azure Databricks, PySpark & Spark Optimization)
Remote (Canada)
Job Type: Contract
Role Summary
We are looking for a hands-on Lead Data Engineer to support a proof of concept in Azure Databricks.
The role focuses on converting complex SQL stored procedures into optimized Python and PySpark code. The objective is to demonstrate that the platform can process 100,000 simulation years in minutes rather than hours, while maintaining accurate results.
Key Responsibilities
Refactor complex T-SQL logic, including pricing rules, layer applications, and reinstatements, into Python and PySpark.
Replace row-by-row processing with efficient, vectorized DataFrame operations.
Develop Azure Data Factory pipelines to ingest loss data from Touchstone/RMS into ADLS Gen2.
Optimize Spark jobs through effective partitioning, memory management, and skew handling.
Validate the new solution against existing SQL results to ensure accuracy.
Contribute to performance testing and troubleshooting.
Required Skills
Advanced Python skills, including Pandas and PySpark.
Strong T-SQL experience and the ability to analyze complex stored procedures.
Hands-on experience with Azure Databricks, Delta Lake, and Azure Data Factory.
Strong knowledge of distributed computing and Spark optimization.
Experience with partitioning, sharding, broadcast variables, and data-skew management.
Familiarity with CI/CD and version control for data engineering projects.
Desirable Skills
Experience with Monte Carlo simulations or stochastic modelling.
Knowledge of Parquet and Avro file formats.
Experience analyzing SQL Server execution plans.
Background in insurance, risk analytics, or simulation-based modelling.
Work arrangement
Yes
Live timing from JobBeeper.
Recently tracked roles with full details and direct application links.