Senior AI Engineer with Spark, AWS Services

We are seeking a Senior AI Engineer with Spark and AWS Services expertise to join the RBQM Production Pod within the program. You will build and maintain data pipelines that power AI/GenAI applications for Risk-Based Quality Management in clinical trials. This role focuses on RAG document ingestion, vector indexing, and building data APIs for AI applications.
Responsibilities
Design and build RAG document ingestion pipelines (chunking, embedding, vector indexing) for clinical trial quality data
Build and manage vector databases (AWS OpenSearch) for RAG-powered AI workflows
Develop batch and streaming ETL/ELT pipelines from scratch for unstructured clinical data (PDF, DOCX, clinical reports)
Build and expose data APIs for AI application consumption
Optimize chunking strategies, embedding generation, and retrieval performance for RAG architectures
Manage data quality, lineage, and governance for AI/ML data pipelines
Deploy and maintain AWS data infrastructure (S3, Lambda, Glue, Athena, Step Functions, DynamoDB)
Collaborate with Data Scientists and Backend Developers in an integrated pod team
Requirements
5+ years of hands-on data engineering experience at scale
Expertise in RAG document ingestion pipelines (chunking, embedding, vector indexing)
Proficiency in AWS OpenSearch as a vector database for RAG workflows
Advanced proficiency in Python, including SQL and Spark SQL
Skills in unstructured data transformation (PDF, DOCX) for RAG/LLM applications
Familiarity with AWS Services: S3, Lambda, Glue, Athena, Bedrock, Step Functions, API Gateway, CloudWatch, DynamoDB
Knowledge of containerization with Docker
Capability to build custom pipelines from scratch, beyond configuring out-of-the-box services
Proficiency in English at a B2+ level
Nice to have
Background in pharmaceutical or life sciences domain
Familiarity with Snowflake, Pinecone (vector DB alternative)
Knowledge of SageMaker processing jobs
Skills in CI/CD tools (Jenkins, Git/Bitbucket) and infrastructure tools (CDK or Terraform)
Understanding of clinical data standards (CDISC, ADaM, SDTM)
We offer
CONTINUOUS UPSKILLING, LEARNING & DEVELOPMENT
Diversity of tasks and projects
Assessment center for objective review of competency level
Personal development plan
Mentoring programs and leadership development
Certification and professional development support
Access to learning platforms including more than 2,500 internal courses
English courses taught by certified teachers
CORPORATE BENEFITS
Extra leave days
Referral bonuses
COMPENSATION PACKAGE
Competitive compensation paid in USD
Regular salary and performance reviews
MEDICAL & HEALTHCARE
Private health insurance
Well-being events
WORKING ENVIRONMENT
Recreation areas and kitchens
Tea, coffee and snacks
Sports equipment and game consoles
IT Equipment
Microsoft’s Software Assurance Home Use Program (HUP)
Please note that our Talent Attraction Team reviews applications and CVs submitted in English.
EPAM is global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. With over thirty years of expertise in custom software, product and platform engineering, we empower our clients to become AI-Native enterprises, driving measurable value from innovation and digital investments.

Work arrangement
Yes