Lead - AI/ML Engineering (Databricks & GenAI)

Role Details: The AI CoE partners with business, technology, and risk stakeholders to identify high-value AI opportunities and deliver scalable, compliant AI solutions. The team drives end-to-end AI adoption, leveraging GenAI and advanced analytics to transform business processes and create measurable business outcomes.
Leads the design, development and operation of secure, reusable and production-grade AI solutions on Azure and Databricks, combining strong software engineering with practical ML, statistical and evaluation competence.
Overall, Job Description
AI Solution Engineering & Architecture
Lead the end-to-end engineering of enterprise AI solutions, from technical design and proof of value through production deployment, scaling, monitoring and ongoing enhancement.
Design modular architectures across application services, AI orchestration, retrieval, model endpoints, data pipelines, enterprise integrations and user-facing channels.
Build reusable AI engineering components and reference implementations for model access, RAG, document intelligence, agent tools, prompt execution, evaluation, feedback and observability.
Make pragmatic build-versus-buy and technology choices while ensuring alignment with enterprise architecture, security, data-governance and platform standards.
Generative AI, RAG & Agent Engineering
Build production GenAI applications using LLMs, structured outputs, context management, tool calling, model routing, guardrails, retries, fallbacks, caching and human-in-the-loop controls.
Engineer RAG and enterprise knowledge solutions covering ingestion, parsing/OCR, metadata, chunking, embeddings, hybrid retrieval, reranking, citations, abstention and permission-aware access.
Design bounded agentic workflows with explicit tool contracts, execution limits, durable state, approval checkpoints, idempotency, failure handling and auditability.
Determine when conventional software, rules, search, predictive ML, GenAI or a combined approach is most appropriate.
Azure Databricks Engineering
Build and integrate AI workloads using Python, SQL, PySpark, Delta Lake, Databricks Workflows, Model Serving, Vector Search, Unity Catalog and MLflow.
Integrate Azure-hosted applications and services with Databricks-governed data, features, models, functions and AI artefacts through secure APIs and approved connectivity patterns.
Use Azure services for application hosting, API management, identity, secrets, networking, monitoring, messaging and CI/CD as required by the solution architecture.
AI Quality, ML & Applied Statistics
Apply ML and statistical fundamentals to define baselines, datasets, evaluation design, metrics, thresholds and release criteria for AI solutions.
Create repeatable evaluation and regression frameworks for retrieval, generation, agents and predictive models, combining deterministic measures, human review and calibrated model-based scoring.
Technical Leadership & Delivery
Lead and mentor AI engineers, conduct architecture and code reviews and remain hands-on for prototypes, complex components and production issues.
Partner with AI Product Managers, data teams, business units, Enterprise Architecture, Information Security, Risk, Model Risk, Compliance and Audit to take solutions safely into production.
Define engineering standards, reusable patterns, acceptance criteria and production-readiness gates and challenge vendors and implementation partners on technical quality.
EDUCATION
Bachelor’s or Master’s degree in Computer Science, Engineering, Information Technology, Data Science, Statistics, Mathematics or a related quantitative discipline.
Strong software-engineering and applied AI competence is more important than research credentials
Preferred: Recognised Azure, Databricks, MLflow, cloud architecture or AI engineering certification.
Essential requirements:
9-12 years of overall technology experience.
6+ years in software, data or platform engineering, including strong hands-on Python development and production-system ownership.
4+ years building AI/ML solutions, with at least 2 years in Generative AI, RAG, NLP, document intelligence or agentic applications.
3+ years with Databricks, Apache Spark or a comparable lakehouse/distributed engineering environment.
3+ years deploying and operating production applications, data pipelines or ML systems on Azure or another major cloud platform.
2+ years leading engineers or complex technical workstreams while remaining actively involved in architecture, coding and reviews.
Demonstrated ownership of at least two AI solutions from problem definition and design through deployment, monitoring and ongoing support.
Technical Skills:
Software & AI Application Engineering
Advanced Python, object-oriented and modular design, typing, asynchronous programming, packaging, dependency management and performance optimisation.
API and service engineering, including REST, streaming, authentication, concurrency, rate limiting, retries, resilience and error handling.
System-design fundamentals covering distributed systems, microservices, event-driven patterns, caching, scalability and graceful degradation.
Git, peer review, unit/integration/contract/regression testing, secure coding, documentation and production incident analysis.
Working familiarity with JavaScript/TypeScript and modern frontend patterns sufficient to design and review end-to-end AI applications.
Generative AI & LLM Engineering
LLM fundamentals, model APIs, prompt/context design, structured generation, function/tool calling and model abstraction.
Model selection and routing based on quality, latency, throughput, cost, security, data residency and operational requirements.
Prompt and application versioning, guardrails, validation, fallbacks, human review and model/provider portability.
Practical judgement on prompting versus RAG versus fine-tuning versus predictive ML or conventional software.
RAG, Document Intelligence & Agents
Document ingestion, parsing, OCR, classification, metadata extraction and document-quality handling.
Chunking, embeddings, keyword/vector/hybrid retrieval, metadata filters, reranking, source traceability, citations and answer abstention.
Permission-aware retrieval and failure isolation across ingestion, retrieval, generation, tool selection and execution.
Agent routers, workflows, tool schemas, state management, checkpoints, approval controls, execution boundaries and compensating actions.
Databricks Engineering
Python, SQL, PySpark, Delta Lake and lakehouse design patterns.
Databricks Runtime, notebooks/repos, Jobs, Workflows and deployment bundles or equivalent release practices.
Batch/streaming pipelines, feature computation, distributed processing and workload cost/performance optimisation.
Model Serving, scalable batch and real-time inference, Vector Search and governed integration with Databricks APIs/SDKs.
MLflow, Unity Catalog & AI Lifecycle
Experiment and application tracking, parameters, metrics, artefacts, model/app packaging and reproducibility.
Models and AI artefacts in Unity Catalog, versions/aliases, promotion, rollback, access control, lineage and auditability.
MLflow tracing, prompt/application versioning, evaluation datasets, scorers, feedback and production monitoring.
Traceability across data, documents, features, code, models, prompts, tools, configurations and evaluation results.
Machine Learning & Applied Statistics
Classification, regression, ranking, clustering, forecasting and anomaly detection fundamentals.
Training/validation/test design, sampling, baselines, feature engineering, model comparison, calibration and threshold optimisation.
Probability, hypothesis testing, confidence intervals, regression, experimental design, A/B testing and practical versus statistical significance.
Leakage, bias, class imbalance, confounding, temporal validation, drift, uncertainty and explainability.
The role requires strong practical understanding and review capability; it is not intended to be a research-scientist position.
AI Evaluation & Observability
Retrieval measures including precision/recall at K, ranking relevance and labelled query-document datasets.
LLM measures including groundedness, correctness, completeness, consistency, robustness, safety and task completion.
Deterministic checks, custom scorers, human evaluation, regression datasets and appropriately calibrated LLM-as-judge methods.
Application and model observability covering traces, latency, errors, throughput, token/compute cost, drift and quality degradation.

Work arrangement
No