Data Engineer - III

Responsibilities:
Designing data systems.
Reviewing Code.
Developing/Implementing Part of Systems.
Develop a comprehensive understanding of the team's projects and interdependencies, ensuring alignment across initiatives.
Partially own the team-wide and cross-team projects, driving the definition, scoping, and delivery of features.
Mentoring people in the team.
On-call monitoring and support.
Working across engineering and business teams to clarify requirements.
Main challenges to solve:
Building strong technical ownership (build, ship, and own) and driving technical vision for the team.
Should own the product from requirements to maintenance.
Build strong ownership of the ACR-related data pipeline development.
Contribute to governance initiatives and the migration of on-prem infrastructure to the cloud (Databricks).
Scalable Data Models Pipeline processing TBs of data daily.
Data quality and monitoring challenges at scale.
Requirements:
Understand and build ownership of Big Complex technical components.
Should start deriving vision and development within a month.
People should not limit themselves to the tech and should be able to pick up new stuff based on the team's needs.
Core Skills requirements: Spark (Scala Spark/PySpark), Git, Airflow (Must), Kafka (Nice), SQL, DBMS (Must), S3 (Nice), ElasticSearch/OpenSearch (Nice), and IAC (must), AWS. (Nice), Databricks (Big Plus), Data Modelling.

Experience
6-10 yrs