Staff Machine Learning Engineer, AI Training Data Expert (PhD, Contract))

About the role
Cobalt builds expert data and evaluation infrastructure for AI developers. We are recruiting senior machine learning researchers, including faculty, research scientists, and experienced postdoctoral researchers, for a contract project that tests whether frontier AI models can carry out real research work in a command line environment. You will design research-grade tasks that a leading AI model cannot solve, then review how and why the model fails. Accepted tasks are used to evaluate and train the next generation of frontier models.
What you will do
Design self-contained command line tasks drawn from the practice of machine learning research, such as implementing a method from a paper's description alone, locating the error in a flawed experimental setup, reproducing a reported result and explaining a discrepancy, implementing a custom loss function or optimizer correctly, and running a sound statistical comparison between two models.
Build the task environment, including code, data, and dependencies, write a reference solution, and write automated tests that verify whether a solution is correct. Because results can vary between runs, tests need to be deterministic or use well-justified tolerances.
Run your task against a frontier AI model, review its attempt with the same rigor you would apply to a paper under peer review, and refine the task until the failure reflects a real gap in the model's capability rather than ambiguity or trick wording.
Work with reviewers to bring each task to acceptance.
Who we are looking for
A PhD in machine learning, computer science, statistics, or a closely related field.
Industry or academic experience in machine learning research, in academia or industry. Years spent on doctoral research count toward this total.
At least one publication, ideally peer-reviewed work at a recognized machine learning venue or journal.
Deep working knowledge of Python and at least one major framework, such as PyTorch or JAX.
Fluency in the Linux command line, shell scripting, Docker, and Git.
Experience as a peer reviewer, or in designing course assignments and automated grading, is a strong advantage, because the work draws on the same skills.
Why Cobalt AI:
Advance frontier AI where it counts. Apply your research expertise to the data that frontier labs cannot obtain any other way, where your reasoning directly shapes how the next generation of models works through technical problems.
Grow professionally. Expand your influence through evaluation projects, advisory roles, and research collaborations, while deepening your understanding of how frontier models are trained and assessed.
Work with a top-tier network. Collaborate with researchers from leading institutions and labs on high-impact, flexible work.
Set your own schedule. Flexible 10 to 40 hour weeks that fit around your research position and your life.
Competitive pay. Rates vary by project and are determined by a number of factors, including scope, skillset, and experience.

Work arrangement
No