Lead Data Engineer

Experience Required - 8+ Years.
Work Locations - Bengaluru, Chennai, Hyderabad, Pune.
Notice Period - max 30 Days.
Primary Skills –
Spark Structured Streaming, Apache Spark, Scala, Apache Kafka, Databricks, Any cloud, preferably GCP.
In depth coding knowledge in Spark Structured Streaming.
Kafka producers and Spark structure streaming consumers.
Strong technical knowledge on Big Data Technologies ( Data bricks, Spark Streaming , GCP , Scala )
Strong technical skills in Spark Structure streaming, Scala, python, data bricks for lake flow engineering and orchestration, Cloud SQL, Git.
Spark internals and execution architecture.
Spark execution concepts - including query planning, DAG generation, shuffle operations, partitioning strategies, and memory management.
Understanding of Spark optimization techniques such as broadcast joins, caching/persistence strategies, partition pruning, adaptive query execution, and skew handling
Databricks platform expertise and optimization techniques - cluster sizing strategies, job optimization, Delta Lake internals, workflow orchestration, and performance tuning.
Advanced batch and streaming ingestion patterns.
Data Ingestion and Architecture Patterns - Exposure to ingestion frameworks and modern data integration patterns.
Performance and Optimization Experience - troubleshooting performance bottlenecks, optimizing large-scale data pipelines, or implementing cost and performance improvements.
Performance tuning and troubleshooting at scale.
Data architecture and design decision-making.
Technical leadership, mentoring, and stakeholder communication.
Key Responsibilities
Possess very strong communication skills and stakeholder management.
Hands on project technical expertise and involve 80% in technical work and 20% management activities
Carries very strong Development project experience and manage a team of 10 to 15 technical project team.
Involve in Technical architecture review meetings and understanding the proposed technical architecture for the project requirements
Provide required technical mentoring to delivery team
Efficient coordination with Onshore Tech Lead on the deliverables and ensure to meet quality & schedule adherence
Report the Monthly Metrics (Internal & Client)
Generate all the required reports.
Review and arrange training for the team depending on the need basis.
Part of deployment activities
Good analytical and Problem-solving skills

Work arrangement
No