Machine Learning Evaluation Specialist

Machine Learning Evaluation Specialist (AI Training)
About The Role
What if your years of hard-earned research expertise could directly shape the future of AI? We're looking for domain experts with deep machine learning knowledge to design evaluation challenges that push state-of-the-art AI systems to their limits.
This isn't reviewing chatbot responses or rating simple outputs. You'll be crafting complex, nuanced problems that only a true domain expert could construct — and that only the most capable AI systems could hope to solve.
Organization: Alignerr
Type: Hourly Contract
Location: Remote
Commitment: 10–40 hours/week
What You'll Do
Design original, expert-level machine learning problems grounded in your area of specialization
Build evaluation tasks that go far beyond standard ML pipelines — requiring genuine, deep domain knowledge
Draw from your own research experience to craft challenges that rigorously stress-test highly capable LLMs
Define precise problem statements, evaluation criteria, and gold-standard solutions
Assess AI-generated solutions for correctness, creativity, and methodological soundness
Document problem difficulty levels, required knowledge, and expected failure modes
Collaborate asynchronously with a global team of researchers and engineers
Who You Are
You hold a graduate-level degree (MS or PhD preferred) in a scientific or technical field that intersects with machine learning
You have strong working knowledge of ML methods — model selection, feature engineering, evaluation metrics, and pipeline design
You're deeply familiar with active, open research problems in your field
You can identify precisely where general ML knowledge breaks down and specialized domain insight becomes essential
You have experience conducting or publishing original research (highly valued)
You communicate complex ideas clearly and precisely in writing
You're self-motivated and thrive working independently on intellectually demanding tasks
Example Domains (Not Exhaustive)
Computational biology, genomics, or bioinformatics
Climate science and environmental modeling
Medical imaging and healthcare ML
Materials science and computational chemistry
Astrophysics and signal processing
Natural language processing for low-resource or specialized corpora
Robotics, control theory, or reinforcement learning in complex environments
Financial modeling and quantitative analysis
Why Join Us
Top-tier compensation — among the highest rates available for contract research work
Work at the frontier — your contributions directly influence how next-generation AI models are measured and improved
Use your expertise meaningfully — this role is built for specialists, not generalists
Full autonomy and flexibility — set your own schedule and work from anywhere
Global collaboration — connect asynchronously with leading researchers and engineers worldwide
Ongoing opportunity — strong contributors are considered for contract extensions and deeper research involvement
Build your profile — establish yourself as a recognized contributor to cutting-edge AI development and safety research

Work arrangement
Yes