Distributed Training Engineer

Location: Noida or Bengaluru
About the Role
Our training infrastructure runs distributed jobs across many GPUs at once, and it has to just work no lost checkpoints, no silent slowdowns, no surprises at 2am. As a Training Runtime Engineer, you'll build and harden the distributed training infrastructure our model development depends on, working on some of the hardest reliability and performance problems in the stack.
What You'll Do
Build and maintain distributed training infrastructure, including tensor, pipeline, data, and expert parallelism and FSDP
Implement and harden distributed checkpointing and fault-tolerant recovery for long-running training jobs
Optimize NCCL communication patterns and diagnose distributed-training performance issues
Support multi-tenant job scheduling and isolation for concurrent training runs
Partner with our research team to translate new training recipes into runtime requirements
What You'll Bring
5+ years of experience in distributed systems or ML infrastructure engineering
Strong experience with distributed training frameworks (PyTorch, Megatron-LM, DeepSpeed, or similar)
Hands-on experience with CUDA, NCCL, and RDMA-based communication
Solid grasp of distributed checkpointing and fault-tolerant systems design
Proficiency in C++ and Python
Education
B.Tech in Computer Science or a related field; M.Tech a plus for senior scope.
Nice to Have
Experience at a distributed-database, HPC, or cloud infrastructure company — the reliability and consistency challenges transfer directly
Prior work on a multi-tenant training or scheduling system

Work arrangement
Hybrid