Data Engineer

Experience: 3–5 years in Data Engineering, specifically with distributed systems and cloud-native architectures
Coding: Expert-level Python/PySpark and SQL. Familiarity with Go/Java/Scala is a plus
Infrastructure: Hands-on experience with AWS (S3, EKS, MSK) and Infrastructure-as-Code
Orchestration: Experience with Airflow or Temporal for complex workflow management
AI-Native: Proficiency in using AI tools (Claude, Codex, Copilot) to write, test, and document code efficiently
Systems Thinking: Ability to explain the trade-offs between different storage formats and processing frameworks
Tech Execution : Drive key tech initiatives by preparing TRD and actively involve in design reviews
Domain Modelling - Should be hands on in designing Domain models for OLAP like Fact, Dimension, Cumulative , types of SCD’s and OBT pattern tables
Self Starter - Lead the team technically and bring in new ideas to contribute to the growth of the charter
Stakeholder Interaction - Interact with the Product & Key Stakeholders & help them by adding value to the business workflow with data & analytics
Good to have -
Real-time CDC: Ownership of high-throughput ingestion from RDBMS to Lakehouse using Debezium, PeerDB
Lakehouse Architecture: Designing and optimizing table formats (Iceberg, Delta, Hudi) for both performance and storage efficiency
Unified Compute: Developing robust ETL/ELT frameworks in PySpark and Flink (handling both batch and streaming workloads)
Infrastructure & Ops: Managing data workloads on AWS (EMR, EKS, MSK, S3) and automating everything via Gitlab/Github Actions
Query & BI: Tuning Trino or Clickhouse to power real-time dashboards in Metabase, Superset, and PowerBI
Our Tech Stack-
Ingestion & CDC: OLake and PeerDB for near real-time sync from production systems into Alchemy; Debezium/Kafka for CDC-heavy use cases, with support for Kafka and S3-based sources
Lakehouse / Storage: Alchemy on Apache Iceberg with S3 as the data lake storage layer and AWS Glue Catalog for metadata; exposure to Delta/Hudi is a plus
Processing & Compute: PySpark on EMR/EKS for batch and streaming workloads; Flink and Spark Structured Streaming fundamentals for low-latency pipelines
Streaming Platform: MSK / Kafka for event-driven ingestion, CDC propagation, replay, backfills, and operational monitoring through Kafka UI
Query & Serving Layer: Trino over Alchemy/Iceberg for lakehouse analytics, ClickHouse for high-throughput operational and real-time dashboards, and BigQuery exposure where applicable
Workflow Orchestration: Airflow for scheduled data pipelines, DQ/reconciliation DAGs, backfills, and SLA-driven jobs; Temporal for durable workflow execution in ingestion services
Data Quality & Governance: DQ checks, freshness/SLA monitoring, source-to-lake reconciliation, deduplication, schema evolution handling, and cataloging/lineage through OpenMetadata
Infrastructure & DevOps: AWS (S3, EKS, EMR, MSK), Kubernetes, Terraform/IaC, GitLab/GitHub Actions, observability via Grafana/CloudWatch, and production runbook discipline
BI & Analytics: Metabase, Superset, Tableau, and PowerBI for business dashboards; strong ability to model curated marts, fact/dimension tables, SCDs, and OBT patterns for stakeholder-facing analytics
Skills: analytics,kafka,cdc,s3,infrastructure,data,alchemy,storage,aws,workflow

Work arrangement
No