Lead Data Engineer – AWS, Databricks & PySpark

**FULLY REMOTE**
ScolerTec is seeking Lead Data Engineer – AWS, Databricks & PySpark to design, build, and lead enterprise-scale cloud data engineering solutions.
This is a hands-on technical leadership role responsible for scalable ETL/ELT pipelines, batch and near-real-time data integration, lakehouse/data-platform implementation, cloud data services, CI/CD, data quality, governance, performance optimization, and technical direction of data engineers.
The ideal candidate has strong recent experience with AWS, Databricks, Apache Spark/PySpark, Python, SQL, Delta Lake, ETL/ELT, and technical leadership.
Key Responsibilities
Design and implement scalable cloud-based data platforms, including data lake, lakehouse, and data warehouse architectures.
Build enterprise-scale ingestion and ETL/ELT pipelines for batch and near-real-time workloads.
Develop data transformation and integration solutions using Python, SQL, Spark/PySpark, and modern cloud data technologies.
Implement Databricks/Delta Lake-based data engineering solutions and downstream curated/gold data layers.
Integrate pipelines with AWS compute, storage, messaging, and analytics services.
Lead technical design reviews, code reviews, engineering standards, and reusable data-engineering patterns.
Provide day-to-day technical guidance and mentoring to data engineers.
Implement data-quality checks, validation rules, error handling, observability, and pipeline monitoring.
Support streaming and event-driven data processing.
Implement Infrastructure as Code and CI/CD for repeatable data-platform and pipeline deployment.
Optimize pipeline performance, scalability, reliability, and cloud cost.
Support production incident resolution and root-cause analysis.
Partner with architecture, governance, migration, QA, DevSecOps, and security teams.
Maintain technical documentation, runbooks, and audit-ready engineering artifacts.
Qualifications
7+ years of specialized data-engineering experience.
Strong hands-on experience building enterprise-scale data pipelines.
Strong production experience with AWS data and compute services.
Strong hands-on Databricks experience.
Strong Apache Spark/PySpark and Delta Lake experience.
Strong Python and SQL skills.
Strong ETL/ELT and data-transformation experience.
Experience with batch and streaming/near-real-time ingestion.
Experience with data orchestration tools.
Experience with CI/CD and Infrastructure as Code.
Experience with data quality, governance, metadata, and classification.
Proven technical leadership, design review, code review, and mentoring experience.
Federal, public-sector, or regulated-environment experience preferred.
Preferred Certifications
Databricks Certified Data Engineer Professional
AWS Certified Data Engineer
AWS Certified Solutions Architect – Professional
AWS Certified Data Analytics – Specialty
CDMP

Work arrangement
Yes