Public listing
AI Evaluation Rubric Auditor | $60-$85/Hour
- Work modeYes
- ListedPosted 8 hours ago
- SourceLinkedin
Role overview
About the role
Description supplied by the original job listing.
We are looking for experienced AI evaluation professionals to audit the quality and accuracy of LLM evaluation tasks.
In this role, you will review work completed by AI Evaluation Reviewers, assess their decisions, and ensure that prompts and grading rubrics are fair, accurate, and aligned with evaluation guidelines. You will also provide actionable feedback to reviewers to improve the overall quality and consistency of AI evaluation.
Responsibilities
Audit complete AI evaluation tasks, including prompts, grading rubrics, and model attempts.
Assess whether identified model failures are genuinely justified.
Verify whether the required information was available to the AI agent.
Identify inaccurate, ambiguous, or unfair grading criteria.
Determine whether valid model responses could incorrectly receive failing grades.
Check rubric structure and compliance with evaluation guidelines.
Identify tasks that cannot be reasonably corrected.
Provide clear, constructive feedback to reviewers on their decisions.
Maintain consistent auditing quality and availability.
Requirements
Substantial experience writing, reviewing, or auditing LLM evaluation rubrics.
Previous experience in AI evaluation, RLHF, model training, or AI quality assurance.
Experience providing feedback to writers, reviewers, or evaluators.
Strong analytical and critical-thinking abilities.
Excellent written English and attention to detail.
Ability to analyze lengthy AI agent transcripts and identify specific errors.
Ability to independently verify information rather than blindly trusting reference answers.
Consistent availability and ability to meet project expectations.
Relevant Experience
Candidates with experience in the following areas are encouraged to apply:
LLM Evaluation and Benchmarking
Rubric Design and Auditing
RLHF and Human Feedback
AI Quality Assurance
AI Model Assessment
Reviewer Quality Control
Prompt and Response Evaluation
Technical Content Review
Application Requirement: Please list the AI evaluation projects you have worked on, the organizations or platforms involved, and your specific responsibilities.
Contract Details
Compensation: $60-$85 per hour
Work arrangement: Fully Remote
Schedule: Flexible, with consistent availability expected
Payments: Weekly via Stripe or Wise
Join a team working with leading AI labs to improve the quality, reliability, and evaluation of next-generation AI systems.
About Hirely
Hirely connects qualified professionals with remote opportunities in AI research, business, software engineering, and technology. We are recruiting candidates on behalf of a hiring partner, which manages final candidate selection, onboarding, project assignments, and engagement terms.
Skills
LLM, Auditing, Onboarding, Quality Control
Work arrangement
Yes
Skills from the listing
- LLM
- Auditing
- Onboarding
- Quality Control
Your early-applicant advantage
How quickly JobBeeper finds and shares openings.
Job alerts
Get notified about new openings
Save your preferences, then add a paid plan for matching job alerts.