#EG Senior AI Data Engineer

Role Summary
You are an autonomous AI Engineer, responsible for creating the harness for designing and building the AI harness, agent framework, and reusable engineering components required to operate a modern data platform based on Apache Iceberg, Trino, and Parquet.
Your primary responsibility is not only to build data pipelines, but also to establish the agentic execution framework (AI Harness) that enables multiple specialized agents to collaborate, execute tasks, maintain state, orchestrate workflows, and deliver end-to-end data engineering capabilities.
Depending on your level of experience, your focus in the team will be on either task execution, independent delivery, or technical leadership.
Key Responsibilities
AI Harness Development
Design and implement the foundational agent framework that provides:
Agent orchestration and coordination
Workflow execution and state management
Memory and context handling
Tool and function calling
Agent-to-agent communication
Human approval workflows
Logging and audit trails
Error handling and recovery
Security and access controls
Observability and monitoring
The harness must support autonomous and multi-agent execution patterns.
Specialized Agents Creation
The agents will need to perform the following:
Discover Iceberg catalogs and schemas
Profile datasets
Generate metadata and lineage
Ingest data from databases, APIs, files, and streams
Create Parquet datasets
Register and maintain Iceberg tables
Generate Trino SQL
Build transformation workflows
Create curated and business-ready datasets
Validate schemas and data integrity
Detect anomalies
Generate reconciliation reports
Analyze Trino workloads
Recommend partitioning strategies
Optimize Iceberg tables and Parquet files
Classify sensitive information
Manage metadata and lineage
Apply retention and compliance policies
Monitor pipelines and agent activities
Detect failures and bottlenecks
Generate alerts and operational insights
Expected Deliverables
Iceberg table definitions and schemas
Trino SQL scripts and views
Python data pipeline code
Data quality validation suites
Metadata and lineage documentation
CI/CD deployment artifacts
Monitoring and observability configurations
Technical documentation and runbooks
Skills and Experience
The candidate should demonstrate expertise in:
Experience with Generative AI, Agentic AI, or LLM-based solutions.
Proficiency in Python, including FastAPI, Pydantic and Pandas.
Experience designing and implementing AI Harness or multi-agent frameworks.
Hands-on experience with LangChain & LangGraph.
Experience with Apache Iceberg, Trino, and Parquet-based lakehouse architectures.
Experience building data ingestion, transformation, and orchestration pipelines.
Strong knowledge of metadata management, data lineage, and data governance practices.
Experience developing REST APIs, SDKs, and reusable engineering frameworks.
Experience with observability, monitoring, tracing, and operational support practices.
Strong understanding of distributed systems, event-driven architectures, and microservices patterns.
Proven ability to design, develop, and operationalize enterprise-grade AI agents for data discovery, ingestion, transformation, optimization, governance, and observability.
Familiar with modern data engineering technologies, including DuckDB, dbt, and related engineering tools for data transformation, modelling, and pipeline development.
Senior engineers will be expected to have demonstrated evidence of leading technical designs, guiding junior engineering team members, working independently with clients and delivery teams, and translating AI/data requirements into scalable, secure, and deployable solutions.

Employment type
Full-time

Work arrangement
No