Agentic AI Engineer - Senior

About Interactly.ai
Interactly.ai is transforming healthcare operations with AI-powered automation across scheduling, lab follow-ups, eligibility checks, prior authorization, and insurance verification. Our multimodal AI agents work across voice, email, chat, fax, and EHR integrations improving patient access, reducing staff burden, and accelerating care delivery.
We serve 2,000+ physicians and healthcare providers, delivering measurable outcomes, 20–30% fewer no-shows and thousands of staff hours saved. Now, we're scaling our AI platform and we want you to help lead that charge.
Role Overview
As an Agentic AI Engineer at Interactly.ai, you'll design, build, and ship LLM-powered, agentic AI systems that run in production at scale. You won't just prototype you'll own pipelines end-to-end, from prompt design and eval frameworks to distributed microservice deployment and monitoring. You'll take full ownership of everything you build, including end-to-end testing, so that what ships is production-ready, reliable, and safe from day one.
Key Responsibilities
Design and build LLM-powered agentic AI pipelines that reason, plan, and execute multi-step workflows with minimal human intervention.
Own tool & function calling integrations connect LLMs to internal APIs, EHR systems, calendars, and third-party services.
Build and maintain rigorous eval frameworks: offline eval sets, regression suites, latency benchmarks, and safety/guardrail tests.
Ship and operate LLMs in production: streaming responses, structured outputs, prompt versioning, cost/latency tracking, and observability.
Architect and maintain distributed microservices that expose LLM APIs async Python (FastAPI), event-driven pipelines, and real-time WebSocket workflows.
Build RAG pipelines: chunking strategies, embeddings, vector stores, retrieval tuning, and PHI-safe handling.
Take full ownership and conduct end-to-end testing for everything you build from unit and integration tests through to production validation and post-deployment monitoring.
Collaborate closely with product, backend, and QA to define AI requirements and ship reliable, safe features.
Qualifications
3+ years of hands-on Gen AI / LLM engineering experience building and shipping real systems, not just research or prototypes.
5+ years of hands-on AI / ML engineering experience overall (Gen AI + pre-Gen AI combined).
Deep LLM expertise in prompt engineering, tool/function calling, structured outputs, chain-of-thought, and agent orchestration (LangChain, LangGraph, or similar).
Proven eval culture that you treat evals as a first-class concern not an afterthought.
Production LLM experience, integrations with OpenAI / Azure OpenAI / AWS Bedrock, streaming, cost optimisation, and monitoring.
Strong software engineering fundamentals in Python (strong), async programming, REST APIs, CI/CD, Docker.
Distributed systems fluency with microservice design, event-driven architecture (Kafka/SQS/RabbitMQ), and LLM API gateway patterns.
End-to-end ownership mindset, you write tests, validate in staging, and don't consider a feature done until it's verified in production.
Preferred Skills
Not required for hiring but these make you stand out:
LLM post-training experience with fine-tuning, RLHF, DPO, or LoRA for domain-adapted or instruction-tuned models.
Automatic prompt optimisation familiarity with tools or techniques like DSPy, TextGrad, or automated prompt search for systematic prompt improvement.
Healthcare industry experience, working knowledge of EHR systems, HL7/FHIR standards, HIPAA compliance, or clinical workflows.
Startup experience where you've worked in a fast-moving, resource constrained environment where you wore multiple hats and shipped quickly.
Scaling & on-prem deployments experience deploying LLMs at scale, including self-hosted / on-prem model serving (vLLM, TGI, Triton) or hybrid cloud architectures.

Work arrangement
No