Principal Distributed Systems Engineer, AI Model Evaluation (PhD)

About the role
Cobalt builds expert data and evaluation infrastructure for AI developers. We are recruiting senior distributed systems engineers for a contract project that evaluates how well frontier AI models perform real systems engineering work in a command line environment. Your job is to design original, realistic systems tasks that a leading AI model fails to solve.
What you will do
Design self-contained command line tasks drawn from real distributed systems work, such as finding a race condition or deadlock in a concurrent service, fixing a consistency bug in a replicated data store, diagnosing a failure from logs and traces across several services, repairing a broken consensus or leader election implementation, tuning a service that degrades under load, and recovering a cluster from a corrupted or partially failed state.
Build the task environment, typically several cooperating processes or containers on a single machine, write a reference solution, and write automated tests that verify whether a solution is correct. Because concurrency and timing bugs are hard to reproduce, tests need to trigger the failure reliably rather than intermittently.
Test your task against a frontier AI model and refine it until the model fails for substantive reasons rather than because of ambiguity, trick wording, or flaky tests.
Respond to reviewer feedback until the task is accepted.
Who we are looking for
A PhD in computer science, distributed systems, or a closely related field.
Academic or industry experience building or operating distributed systems, such as databases, storage systems, messaging systems, schedulers, or large-scale service infrastructure.
At least one publication, either academic (for example a peer-reviewed paper at a recognized systems venue) or professional (for example a conference talk, a detailed engineering article, or a widely used open-source system).
Strong programming skills in at least one systems language, such as Go, Rust, C++, or Java.
Fluency in the Linux command line, shell scripting, Docker, networking tools, and Git.
The ability to write precise task specifications that another engineer could follow without asking questions.
Why Cobalt AI:
Advance frontier AI where it counts. Apply your research expertise to the data that frontier labs cannot obtain any other way, where your reasoning directly shapes how the next generation of models works through technical problems.
Grow professionally. Expand your influence through evaluation projects, advisory roles, and research collaborations, while deepening your understanding of how frontier models are trained and assessed.
Work with a top-tier network. Collaborate with researchers from leading institutions and labs on high-impact, flexible work.
Set your own schedule. Flexible 10 to 40 hour weeks that fit around your research position and your life.
Competitive pay. Rates vary by project and are determined by a number of factors, including scope, skillset, and experience.

Work arrangement
No