Lead Data Test Automation Engineer

About The Role
In this role, you will lead data quality and testing efforts across large-scale Big Data platforms, ensuring the reliability of data pipelines, processing workflows, analytics, and ML-related solutions. You will work closely with Data Engineering, Data Science, Analytics, and client teams to drive data validation strategies, investigate data issues, mitigate quality risks, and improve the overall quality of the data ecosystem using modern cloud and data technologies.
Responsibilities
Define test strategies and testing approaches, and coordinate the testing scope across Data Engineering, Data Science, Analytics, and QA
Identify dependencies, prerequisites, and potential blockers for data test automation solutions
Manage data quality incidents, communicate findings and risks to clients, and drive issues toward quicker resolution to help unblock the team
Execute and monitor Airflow DAG runs and validate pipeline results
Analyze critical datasets and data quality rules, and implement or maintain appropriate checks using Alation or similar tools
Validate data processing results and key metrics after pipeline execution
Verify Tableau reports and other Web UI results after data is published
Support the definition and implementation of regression testing and data validation approaches
Contribute to improving data quality standards, testing processes, and automation practices
Requirements
Strong professional experience in Data Testing, with a solid understanding of ETL/ELT and data pipeline testing using Apache Airflow or similar orchestration tools
Experience testing data-intensive applications and Big Data solutions, AWS experience is preferred
Advanced SQL skills and strong ability to investigate and validate data using complex queries, Snowflake experience is preferred
Good Python knowledge and experience creating data validation or automation solutions using Python, Pandas, and/or PySpark
Experience with data quality and observability tools such as Alation, Soda, or similar solutions
Strong understanding of data quality dimensions and common data validation techniques
Experience with regression testing and test automation principles
Experience investigating, documenting, and resolving data quality incidents
Understanding of CI/CD and version control practices, such as GitHub Actions or similar tools
Strong ownership and ability to work independently on complex data quality problems
Strong communication skills and confidence working with engineering, analytics, and customer teams
SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.

Work arrangement
Yes