Lead Data Engineer

The Lead Developer on this team should be a data engineer, familiar with SQL, Spark, and either Databricks or Athena/Iceberg/Trino. They should understand how to write and optimize ETL/ELT jobs to ingest raw data from a variety of sources into bronze tables, refine that data into scrubbed/silver tables, and optimize/tune those datasets into production/gold tables. They should be comfortable with relational and non-relational databases like SQL and PostgreSQL, big data systems such as Spark, and cloud deployments of software and ETL jobs in AWS. Airflow experience is a bonus. Fluency in Python/Scala is ideal.
Responsibilities:
Architect and develop well-designed, testable, efficient, and secure code.
Analyze complex problems and devise innovative solutions, adapting existing approaches to resolve diverse challenges with limited information.
Exercise evaluation, judgment, and interpretation to independently select appropriate courses of action, with work reviewed at key milestones.
Engage in the entire software development lifecycle, including CI/CD processes. Mentor and support new and junior team members.
Creatively assess and resolve a wide range of issues, suggesting alternative approaches as needed.
Work closely with data consumption stakeholders and display strong analytical knowledge to define data models that are both efficient and cost-effective.
Requirements:
Bachelor's or advanced degree in Computer Science, Software Engineering, or a related field.
A minimum of 10 years of full-time work experience as a software developer.
Proficiency in technologies like Apache Spark, Databricks, Kafka, SQL, and Terraform.
Strong programming experience in Python, Golang, Java, Scala, or another advanced object-oriented language.
Hands-on experience with creating robust ETL pipelines.
In-depth knowledge of data structures and algorithms.
Skills in writing unit tests, data quality checks, and automated data testing frameworks.
Proven backend development experience, including work on scalable, high-availability services.
Excellent communication and interpersonal skills, with a track record of effective collaboration across cross-functional teams.
Additional Skills:
Experience with AWS technologies.
Practical experience with Kubernetes for orchestration and containerization.
Proficiency in designing, implementing, and optimizing data models for data warehouses, databases, and data lakes.
Familiarity with data warehousing concepts and platforms like Databricks, Iceberg, Snowflake, Amazon Redshift, or Google BigQuery.
Experience with Schema Registries and Data Catalogs, e. g. AWS Glue, Unity Catalog.
Experience with real-time processing systems like Apache Kafka Streams and Kinesis.
Experience with monitoring tools like Prometheus, Grafana, or cloud-native monitoring solutions, and skills in optimizing performance for large-scale data processing.

Experience
7-11 yrs