Data Engineer

Responsibilities:
AI-Driven Automation: Design and implement AI systems that automate pipeline creation, data transformation logic, and workflow orchestration, pushing the boundaries of what's possible in data engineering.
Pipeline Development: Build scalable, high-performance ETL pipelines that handle critical financial market data with reliability and efficiency.
Event-Driven Architecture: Design event-driven ETL systems to support both real time and batch data processing needs Intelligent Infrastructure: Experiment with and deploy AI agents, code generation tools, and LLM-powered solutions to accelerate data platform development.
Analytics and Insights: Run analytics on large datasets to uncover insights, validate models, and propose new platform features.
Data Modelling: Architect data structures for both analytical (OLAP) and transactional systems using appropriate database technologies.
Cross-Team Collaboration: Partner with engineers across Bangalore and New York to maintain data integrity, security, and performance standards.
Requirements:
A bachelor's or master's degree in computer science or a comparable subject.
3+ years of software engineering experience with a focus on data engineering.
Exceptional programming skills in Python and SQL.
Expertise with data manipulation libraries (Pandas, Polars).
Experience building production-grade ETL pipelines using orchestration tools (Airflow, Dagster, Prefect).
Hands-on experience with DBT and DuckDB for data transformation and querying.
Familiarity with event-based ETL architectures for data processing.
Deep understanding of database technologies and when to use them; experience with Snowflake, MSSQL, Postgres, or similar.
Passion for AI: Genuine curiosity about how AI can transform data engineering; experience using AI tools in development workflows (Claude, GitHub Copilot, or similar).
Strong problem-solving abilities and comfort handling ambiguity.
Excellent communication skills for collaborating across distributed, cross-functional teams.
Strongly Preferred:
Experience building resilient data pipelines on Databricks using Delta Lake, Auto Loader, and workflow orchestration.
AI Implementation Experience: Hands-on experience with AI agentic frameworks (Claude Code, LangGraph), LLM APIs, prompt engineering, or building AI-powered automation tools.
Exposure to financial domain analytics or experience working with hedge funds/asset managers.
Understanding of financial data models (positions, securities, market data, risk metrics).
Experience with cloud platforms (Azure, AWS, GCP) and cloud-native data architectures.
Familiarity with code generation, AI-assisted development, or building developer tools.
Familiarity with data quality frameworks and monitoring systems.
Strategic thinking with the ability to contribute to product roadmap discussions.
Computer science, engineering, or related degree.

Experience
4-8 yrs