Senior Site Reliability Engineer

Senior Site Reliability Engineer, GPU Infrastructure
Remote, EMEA or APAC | Full-time | Competitive base + meaningful equity
The company
I’m hiring for an AI infrastructure company that sources GPU compute from dozens of providers and runs it as one managed platform for AI labs and model builders. They have tens of thousands of GPUs under management and a team of around 50. They’re profitable and backed by some of the best-known names in AI. It’s a confidential search, so I can’t name them here, but I’ll tell you everything on a call.
The role
This isn’t a generalist SRE seat. You’ll run large GPU clusters for distributed training and inference, spread across multiple providers and regions, and work directly with the customers pushing them hardest. When a training run slows down, you’ll be the one tracing it from the network fabric through the kernel to the framework.
What you’ll own
Designing and running multi-provider, multi-region GPU clusters for large training jobs
Acting as the technical contact for customers, from onboarding through to live troubleshooting and tuning
SLOs, capacity planning and reliability across a mixed GPU fleet
Keeping InfiniBand, RoCE and NVLink fabrics healthy
Monitoring that shows what’s happening with GPU utilisation, interconnect throughput and hardware health
Automation for provisioning, health checks, self-healing and driver and firmware updates
Leading incidents where the root cause could sit in hardware, network, orchestration or the ML stack
What you’ll need
You’ve run large GPU clusters (A100, H100, B200 or similar) in production yourself
Production experience with InfiniBand, RoCE or NVLink for distributed training
A good understanding of how big training jobs behave at the systems level: NCCL, CUDA, PyTorch distributed or similar
Strong Linux, including kernel tuning and NVIDIA driver and CUDA management
Kubernetes with GPU workloads, or Slurm and other HPC schedulers (either counts)
Python, Go or Bash for building proper tooling, plus Terraform, Helm, Ansible or similar
GPU monitoring with DCGM and nvidia-smi
Experience leading messy incidents where the cause could be anywhere from firmware to application code
Nice to have
Parallel file systems like VAST, WEKA, Lustre or GPFS
Profiling and tuning distributed training performance
Physical cluster design: racks, power, cooling, network topology and burn-in
Mentoring other infra engineers

Work arrangement
Yes