Research Engineer, AI & Machine Learning (PhD Student)

Job Type: Flexible Contract (Remote)
Cobalt is hiring current Masters and PhD students to review reinforcement learning tasks and environments used to test whether frontier AI models can carry out real research and engineering work. Other contributors design the tasks. Your job is to make sure each one is correct, fair, and actually measures what it claims to measure before it is accepted. If you enjoy finding the flaw in an experiment, the bug in a grader, or the loophole in a reward function, this role is built around that skill.
What you will do
Review reinforcement learning tasks end to end, including the task description, environment code, data, dependencies, reference solution, and automated tests or graders, and confirm that each part is correct and consistent with the others.
Run reference solutions and graders yourself to confirm that correct solutions pass, incorrect solutions fail, and results are reproducible. Because reinforcement learning results vary between runs and random seeds, you will check that tests are deterministic or use tolerances that are clearly justified.
Look for ways an agent could pass a task without doing the intended work, such as exploiting a reward signal, reading the answer from the environment, or satisfying a weak test, and document each issue so the task author can fix it.
Review how a frontier AI model attempted each task and judge whether its failure reflects a real gap in capability, or instead reflects ambiguous wording, a hidden assumption, a broken environment, or an unfair grader.
Write clear, specific feedback to task authors and work with them through revisions until the task meets the acceptance standard.
Track recurring problems across tasks and help improve the review checklist and quality standards over time.
Example tasks you might review include implementing proximal policy optimization or soft actor-critic from a paper’s description alone, finding the bug in a flawed reward function or environment, reproducing a reported learning curve and explaining a discrepancy, and running a sound statistical comparison between two agents across multiple random seeds.
Who we are looking for
Currently enrolled in a Masters or PhD program in machine learning, computer science, statistics, robotics, or a closely related field.
Hands-on experience with reinforcement learning through research, a thesis, coursework projects, or industry work. You should be comfortable reading an implementation and saying whether it is correct.
Strong working knowledge of Python and at least one major framework, such as PyTorch or JAX, and experience with reinforcement learning libraries and environment interfaces such as Gymnasium.
Comfort with the Linux command line, Docker, and Git, enough to set up an environment, run a task, and trace a failure.
A solid grasp of experimental design and statistics, including variance across seeds, confidence intervals, and when a difference between two results is meaningful.
Careful, direct written communication. Much of the job is explaining precisely what is wrong and why.
Strong advantages
Experience as a teaching assistant who wrote assignments or autograders, or as a peer reviewer for a conference, workshop, or journal.
A publication or preprint in machine learning, which we value but do not require.
Experience working with healthcare, clinical, or biological data, such as electronic health records, medical imaging, genomics, or laboratory data. Some projects involve these domains, and reviewers who understand them are especially valuable.
Why Cobalt AI
Work on frontier AI evaluation. Your judgment directly determines which tasks are used to measure what leading models can and cannot do.
Build skills that matter for your research. Reviewing many tasks and model attempts gives you a close view of how frontier models are trained and assessed, and where they break.
Fit the work around your degree. Set your own schedule, with flexible weeks of 10 to 40 hours that can scale down during busy academic periods.
Grow with us. Strong reviewers can move into task design, project leadership, and research collaborations on future projects.

Work arrangement
Yes