Coffeebeans.io - Senior Data Engineer - Python

Data Engineer L2
Experience: 3 - 10 years in data engineering.
Location: Bangalore.
Work Mode: Bangalore - Hybrid.
Role Overview
Join CoffeeBeans Consulting as a Data Engineer L2 and immerse yourself in a transformative role where your expertise will directly contribute to the future of AI. Located in Bangalore, this position offers a unique opportunity to work at the forefront of data engineering, shaping the way businesses leverage their data to drive innovation. With 4 - 7 years of experience, you will play a pivotal role in building and optimizing scalable data pipelines that empower analytics and AI/ML solutions. This is not just a job; it's a chance to elevate your career in a company that values engineering excellence and client impact.
Key Responsibilities
Design and implement enterprise-grade Databricks Lakehouse architectures using Delta Lake and Unity Catalog.
Build scalable batch and real-time data ingestion pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.
Design and implement CDC architectures using Debezium, Kafka/Kafka Connect, and relational databases such as PostgreSQL, MySQL, SQL Server, and Oracle.
Implement streaming and event-driven data pipelines using Kafka, Spark Structured Streaming, and related technologies.
Design and manage schema evolution and data contracts using Karapace / Schema Registry.
Implement centralized governance using Unity Catalog, including catalogs, schemas, RBAC, row/column-level security, lineage, and data access policies.
Develop metadata-driven ingestion frameworks, data quality, reconciliation, profiling, and observability solutions.
Design Bronze, Silver, and Gold data layers and appropriate data modeling strategies for analytical workloads.
Establish engineering best practices covering CI/CD, testing, deployment, monitoring, logging, and operational support.
Use Databricks Asset Bundles (DAB) and CI/CD tools such as Jenkins/GitHub Actions for automated deployment.
Work with cloud services such as AWS S3, IAM, networking, monitoring, and security services.
Lead technical discussions with clients, translate business requirements into technical solutions, and drive architecture decisions.
Troubleshoot complex data engineering, CDC, streaming, performance, and production issues.
Mentor engineers and provide technical direction across data engineering initiatives.
Must-Have Skills
Strong hands-on experience with Databricks and Lakehouse architecture.
Advanced Python and SQL skills.
Strong expertise in Apache Spark / PySpark and distributed data processing.
Hands-on experience with Unity Catalog and Delta Lake.
Experience with Lakeflow Connect, SDP / Spark Declarative Pipelines, and Auto Loader.
Strong understanding of CDC architectures using Debezium and Kafka.
Hands-on experience with Kafka / Kafka Connect.
Experience with Karapace or Schema Registry and schema evolution.
Strong understanding of ETL/ELT, data modeling, data warehousing, streaming, and data integration patterns.
Experience with production-grade data pipelines and orchestration.
Strong understanding of cloud-native data services, particularly AWS.
Experience with CI/CD and Databricks Asset Bundles (DAB).
Experience leading technical implementations and working directly with business/client stakeholders.
Good To Have
Experience with Snowflake and dbt.
Experience with Apache Flink or other real-time processing frameworks.
Experience implementing data governance, lineage, security, data quality, and observability.
Experience with AWS S3, IAM, Glue, MSK/Kafka, and cloud networking.
Experience designing metadata-driven data platforms.
Experience with AI/ML data platforms and GenAI workloads.
Databricks certifications, particularly Databricks Certified Data Engineer Professional.
AWS Data Engineering/Data Analytics certifications.
Other Expectations
Strong ownership and problem-solving mindset.
Ability to balance hands-on engineering with architecture and technical leadership.
Strong client-facing and communication skills.
Ability to mentor and guide engineering teams.
Willingness to adapt to new technologies and client environments.
Willingness to travel within India and internationally for short/medium-term client assignments.
(ref:hirist.tech)

Work arrangement
Hybrid