Principal Data Engineer

We are looking for an experienced Principal Data Engineer to lead the design, development, and optimisation of large-scale, cloud-native data platforms that power business-critical analytics, AI/ML, personalisation, supply chain, merchandising, and customer experiences across Walmart's global ecosystem. As a Principal Engineer, you will drive technical strategy, establish engineering best practices, mentor senior engineers, and collaborate with cross-functional teams to build highly scalable, secure, and reliable data solutions capable of processing petabytes of structured and unstructured data.
Responsibilities:
Architect enterprise-scale data platforms and distributed data processing systems.
Design and build scalable batch and real-time data pipelines.
Lead data architecture decisions across multiple engineering teams.
Develop high-performance ETL/ELT frameworks using modern big data technologies.
Build cloud-native data solutions leveraging Azure, GCP, or AWS.
Design robust streaming architectures using Kafka and Spark Streaming.
Optimise data storage, partitioning, indexing, and query performance.
Build reusable data engineering frameworks and platform services.
Ensure data quality, governance, security, lineage, and compliance.
Partner with Product, Data Science, ML Engineers, and Business stakeholders.
Define engineering standards, coding practices, and architecture guidelines.
Mentor senior engineers and provide technical leadership across organisations.
Drive platform modernisation initiatives and cloud migration programs.
Conduct architecture reviews and production readiness assessments.
Improve system observability, monitoring, scalability, and resiliency.
Lead incident resolution and performance optimisation efforts.
Requirements:
Bachelor's or Master's degree in Computer Science, Engineering, or related field.
7-10 years of experience in Data Engineering or Big Data Platform development.
Strong expertise in Python, Java or Scala.
Extensive experience with Apache Spark.
Strong knowledge of distributed computing architectures.
Experience designing large-scale data lakes and data warehouses.
Hands-on expertise in SQL and NoSQL databases.
Experience with Apache Kafka and event-driven architectures.
Expertise in Airflow or enterprise workflow orchestration tools.
Strong understanding of data modelling (Dimensional, Star, Snowflake, Data Vault).
Experience building cloud-native data platforms.
Excellent understanding of data governance and metadata management.
Strong software engineering fundamentals including OOP, design patterns, and system design.
Experience working with CI/CD pipelines and Infrastructure as Code.
Excellent stakeholder management and leadership skills.
Technical Skills:
Programming Languages: Python, Java, Scala, SQL.
Big Data Technologies: Apache Spark, Hadoop, Hive, HDFS, Apache Kafka, Spark Streaming, Delta Lake, Apache Iceberg, Apache Hudi.
Cloud Platforms: Microsoft Azure, Google Cloud Platform (GCP), Amazon Web Services (AWS).
Data Warehousing: Snowflake, BigQuery, Azure Synapse, Amazon Redshift.
Workflow and Orchestration: Apache Airflow, Azure Data Factory, Databricks Workflows.
Databases: PostgreSQL, MySQL, Cassandra, MongoDB, Redis.
DevOps and Infrastructure: Docker, Kubernetes, Terraform, GitHub Actions, Jenkins.
Monitoring: Prometheus, Grafana, Splunk.
Preferred Qualifications:
Experience building enterprise data platforms serving billions of records daily.
Strong expertise in lakehouse architecture.
Experience with the Databricks ecosystem.
Exposure to AI/ML feature engineering pipelines.
Experience implementing Data Mesh or Data Fabric architectures.
Knowledge of privacy, governance, and regulatory compliance.
Experience working in large-scale retail or e-commerce environments.
Strong understanding of distributed systems and high-availability architectures.
Nice to Have: Databricks, Unity Catalogue, dbt, MLflow, Feature Store, Kubernetes Operators, Data Contracts, OpenLineage, Great Expectations, Iceberg Catalogues.
Leadership Expectations:
Provide technical vision across multiple engineering organizations.
Influence architecture decisions and engineering roadmaps.
Mentor Staff and Senior Engineers.
Drive engineering excellence and operational maturity.
Lead design reviews and technical governance.
Champion innovation and adoption of modern data technologies.
Collaborate effectively with Product, Platform, Security, and Infrastructure teams.

Experience
7-10 yrs