Lead AI Engineer with Spark, AWS Services

We're looking for a Lead AI Engineer skilled in Spark and AWS Services to become part of the RBQM Production Pod within the program. In this position, you'll construct and sustain data pipelines that drive AI/GenAI applications supporting Risk-Based Quality Management for clinical trials. The primary focus of this role includes RAG document ingestion, vector indexing, and developing data APIs for AI applications.
Experience the freedom of remote work from anywhere in Georgia, whether from the comfort of your home, our modern offices in Tbilisi and Batumi or a coworking space in Kutaisi.
Responsibilities
Architect and construct RAG document ingestion pipelines (chunking, embedding, vector indexing) to support clinical trial quality data
Establish and oversee vector databases (AWS OpenSearch) to support RAG-driven AI workflows
Create batch and streaming ETL/ELT pipelines from the ground up for unstructured clinical data (PDF, DOCX, clinical reports)
Construct and expose data APIs that AI applications can consume
Enhance chunking strategies, embedding generation, and retrieval performance within RAG architectures
Oversee data quality, lineage, and governance across AI/ML data pipelines
Set up and sustain AWS data infrastructure (S3, Lambda, Glue, Athena, Step Functions, DynamoDB)
Partner with Data Scientists and Backend Developers as part of a unified pod team
Requirements
Minimum 7 years of practical, large-scale data engineering experience
Strong background in RAG document ingestion pipelines (chunking, embedding, vector indexing)
Skilled in using AWS OpenSearch as a vector database for RAG workflows
High-level command of Python, along with SQL and Spark SQL
Experience transforming unstructured data (PDF, DOCX) for use in RAG/LLM applications
Working knowledge of AWS Services: S3, Lambda, Glue, Athena, Bedrock, Step Functions, API Gateway, CloudWatch, DynamoDB
Understanding of Docker-based containerization
Ability to develop custom pipelines from the ground up, going beyond simple configuration of pre-built services
Proficiency in English at a B2+ level
Nice to have
Experience within the pharmaceutical or life sciences sector
Exposure to Snowflake and Pinecone (as an alternative vector database)
Understanding of SageMaker processing jobs
Proficiency with CI/CD tools (Jenkins, Git/Bitbucket) and infrastructure-as-code tools (CDK or Terraform)
Familiarity with clinical data standards (CDISC, ADaM, SDTM)
We offer
We connect like-minded people
Delivering innovative solutions to industry leaders, making a global impact
Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
Opportunity to work abroad for up to two months per year
Relocation opportunities within our offices in 55+ countries
Corporate and social events
We invest in your growth
Leadership development, career advising, soft skills and well-being programs
Certifications, including GCP, Azure and AWS
Unlimited access to EPAM's internal learning database
Free English classes with certified teachers
We cover it all
Participation in the Employee Stock Purchase Plan
Monetary bonuses for engaging in the referral program
Comprehensive medical & family care package
Five trust days per year (sick leave without a medical certificate)
Benefits package (sports activities, a variety of stores and services)
EPAM is global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. With over thirty years of expertise in custom software, product and platform engineering, we empower our clients to become AI-Native enterprises, driving measurable value from innovation and digital investments.

Work arrangement
Yes