Data Platform Engineer-Security & SOX

Data Platform Engineer
Location-Remote
Role Summary:
This position supports the CSO team by managing two segregated Azure Databricks platforms — the CSO Security Data Platform and the SOX Application Data Platform — both built on the Medallion architecture. The engineer ensures data quality, source onboarding, pipeline maintenance, and SOX audit support.
Key Responsibilities
Pipeline Operations & Data Quality
Monitor end-to-end Medallion pipelines (L0–L3) for 99.5%+ success rate.
Handle quarantine spikes, schema drift, and dead-letter queues.
Maintain Delta Lake (OPTIMIZE, VACUUM, Z-ORDER), enforce ADLS lifecycle policies, and track DBU usage and storage costs.
Manage dashboards and audit tables for unified visibility of data health and delivery metrics.
Customer Enablement
Support SOC Analysts, Investigators, and Compliance teams with data discovery and ad hoc queries.
Build and maintain Power BI and AI/BI dashboards, curate Genie Space content, and validate SQL queries.
Use MS Purview for data discovery, enrichment gap analysis, and consumer onboarding.
New Source Onboarding
Lead onboarding of new security log sources: schema definition, dedup/partition keys, OCSF mapping, and quarantine strategy.
Register schemas in Unity Catalog, version mappings in Git, and validate through Dev→UAT→Production.
Pipeline Development
Design and maintain Databricks Structured Streaming pipelines for large-scale transitions and data ingestion.

Work arrangement
No