Lead AI Engineer - Speech & Clinical AI

You will own the model layer for our speech and diagnostic AI systems, taking open-weight ASR and language models and adapting them until they work reliably on real clinical audio and real clinical reasoning tasks. This is a deeply hands-on engineering role. Most of your week is spent on data, training runs, and evaluation. Alongside that, you will be the person the rest of the team turns to when they need to decide whether a problem calls for fine-tuning, retrieval, better prompting, or a different architecture entirely.
The core responsibilities for the job include the following:
Model Fine-Tuning (core of the role):
Fine-tune and adapt ASR models for domain vocabulary, accented and multilingual speech, noisy audio, and multi-speaker consultations.
Fine-tune LLMs for diagnostic and clinical reasoning, symptom extraction, differential generation, structured summarization, and clinical coding.
Run and debug real training jobs end-to-end, and diagnose runs that plateau, diverge, or catastrophically forget.
Decide when fine-tuning is the right answer versus prompting, retrieval, or a change of base model.
Data and Evaluation:
Own training data: curation, labeling strategy, synthetic generation, augmentation, deduplication, and leakage checks.
Build an evaluation that predicts production behavior: WER/CER sliced by accent and speaker, entity-level accuracy on clinical terms, and hallucination and omission rates.
Run clinician-in-the-loop review and convert qualitative complaints into measurable objectives.
Production and Optimization:
Distillation, quantization, streaming and latency optimization, batching, and cost per hour of audio.
Partner with the platform/MLOps on serving, monitoring, drift detection, and retraining cadence.
Technical Leadership (approx. 20-25% of the role):
Set applied-AI direction for the team; run design reviews and make build-vs-buy calls.
Mentor 3-5 engineers on evaluation discipline and experiment hygiene.
Communicate model capability and limitations honestly to product, clinical, and leadership stakeholders.
Fine-Tuning Techniques:
Full fine-tuning and parameter-efficient fine-tuning: LoRA, QLoRA, adapters, and prefix tuning.
SFT (supervised fine-tuning) and instruction tuning.
Preference optimization: DPO, ORPO, KTO, RLHF / PPO.
Continued pre-training / domain-adaptive pre-training.
Learning-rate scheduling, warmup, gradient accumulation, and gradient clipping.
Mixed precision (bf16 / fp16), gradient checkpointing.
Mitigating catastrophic forgetting, overfitting, and distribution shift.
Curriculum design and data mixture/sampling ratio tuning.
Checkpoint selection, early stopping, and model merging (SLERP, TIES) are good to have.
Distributed Training
DeepSpeed (ZeRO stages) and/or PyTorch FSDP.
Multi-GPU and multi-node training.
NCCL basics, sharding strategies, OOM debugging.
Flash Attention, xFormers, and Liger Kernel are good to have.
Speech / ASR:
Whisper, wav2vec2 HuBERT, Conformer, RNN-T, and transducer models.
CTC vs. attention-encoder-decoder vs. transducer architecture trade-offs.
Domain adaptation: vocabulary biasing, contextual biasing, shallow/external LM fusion.
Forced alignment; pronunciation and lexicon handling for clinical and drug terms.
Audio preprocessing: feature extraction (log-mel, MFCC), VAD, resampling, normalization.
Augmentation: SpecAugment, noise, reverberation, speed/pitch perturbation.
Speaker diarization and speaker separation (Pyanote or equivalent).
WER/CER evaluation sliced by accent, speaker, cohort, and clinical term.
Streaming / low-latency ASR inference.
Toolkits: NeMo, ESPnet, SpeechBrain, Kaldi, torchaudio, librosa.
LLM and Clinical NLP:
Fine-tuning open-weight LLMs (Llama, Qwen, Mistral, Gemma, or equivalent) for a specialized domain.
Clinical NER and structured extraction from unstructured notes.
Terminology normalization: SNOMED CT, ICD-10 LOINC, RxNorm, UMLS.
Handling failure modes: hallucination, omission, sycophancy, and long-context degradation.
RAG and agentic pipelines; embedding models and vector stores.
Prompt engineering and structured/constrained output generation.
Clinical document standards: HL7 FHIR and SOAP notes are good to have.
Data Engineering:
Large-scale dataset curation, cleaning, and deduplication.
Labelling strategy, annotation guidelines, and inter-annotator agreement.
Synthetic data generation and quality filtering.
Train/validation/test splitting with leakage detection.
Audio- and text-data pipelines at scale (Spark, Ray, Dask, or equivalent).
Evaluation:
Designing task-specific eval suites and regression harnesses.
Speech metrics: WER, CER, and entity-level and keyword recall.
Generation metrics: rubric-based grading, LLM-as-a-judge (with its caveats), and factuality and omission scoring.
Human/clinician review protocols and annotation tooling.
Statistical significance, confidence intervals, and per-cohort error analysis.
Bias and fairness testing across accent, gender, age, and language.
Knowledge distillation:
Export and runtimes: ONNX, TensorRT, and CTranslate2
Serving stacks: vLLM, Triton Inference Server, TorchServe, TGI.
KV caching, continuous batching, latency, and throughput profiling.
Infrastructure and MLOps:
Cloud GPU compute: AWS / GCP / Azure; SLURM or Kubernetes-based scheduling.
Docker and containerized training/serving.
Experiment tracking: Weights and Biases or MLflow.
Model and dataset versioning, artifact registries.
CI/CD for ML, automated retraining, monitoring, and drift detection.
Git and code review discipline.
Cost awareness and GPU budget management.
Leadership and Soft Skills:
Mentoring 3-5 engineers on applied AI practice.
Running design and code reviews; setting technical standards.
Translating clinical and product requirements into model objectives.
Explaining model behavior, uncertainty, and limitations to non-technical stakeholders.
Written communication: design docs, eval reports, post-mortems.
Pragmatic prioritization and comfort with ambiguity.
Domain and Compliance:
Healthcare / medtech or another regulated domain.
PHI / PII handling, de-identification, HIPAA / GDPR / DPDP.
Clinical validation and SaMD / CE marking / FDA pathways.
Model documentation, audit trails, and reproducibility for regulated release.
Requirements:
8-10 years in software / ML engineering, with 3-4+ years training or adapting deep learning models in production.
Publications, open-source contributions, or benchmark results in speech or clinical NLP are good to have.
Multilingual or code-switched speech modelling is good to have.
Must have:
Languages: Python (primary), production-grade, tested, and reviewable; Bash/shell scripting; SQL; and C++ or Rust for inference optimization (good to have).
Core ML Frameworks: PyTorch (deep, hands-on), Hugging Face Transformers, PEFT, TRL, Datasets, Accelerate, Tokenizers, NumPy, SciPy, pandas, scikit-learn, and JAX/TensorFlow are good to have.

Experience
6-10 yrs