Public listing

AI Evaluation Rubric Auditor | $60-$85/Hour

Hirely Argentina (Remote)
  • Work modeYes
  • ListedPosted 6 hours ago
  • SourceLinkedin

Role overview

About the role

Description supplied by the original job listing.

We are looking for experienced AI evaluation professionals to audit the quality and accuracy of LLM evaluation tasks.
In this role, you will review work completed by AI Evaluation Reviewers, assess their decisions, and ensure that prompts and grading rubrics are fair, accurate, and aligned with evaluation guidelines.
You will also provide actionable feedback to reviewers to improve the overall quality and consistency of AI evaluation.
Responsibilities
Audit complete AI evaluation tasks, including prompts, grading rubrics, and model attempts.
Assess whether identified model failures are genuinely justified.
Verify whether the required information was available to the AI agent.
Identify inaccurate, ambiguous, or unfair grading criteria.
Determine whether valid model responses could incorrectly receive failing grades.
Check rubric structure and compliance with evaluation guidelines.
Identify tasks that cannot be reasonably corrected.
Provide clear, constructive feedback to reviewers on their decisions.
Maintain consistent auditing quality and availability.
Requirements
Substantial experience writing, reviewing, or auditing LLM evaluation rubrics.
Previous experience in AI evaluation, RLHF, model training, or AI quality assurance.
Experience providing feedback to writers, reviewers, or evaluators.
Strong analytical and critical-thinking abilities.
Excellent written English and attention to detail.
Ability to analyze lengthy AI agent transcripts and identify specific errors.
Ability to independently verify information rather than blindly trusting reference answers.
Consistent availability and ability to meet project expectations.
Relevant Experience
Candidates with experience in the following areas are encouraged to apply:
LLM Evaluation and Benchmarking
Rubric Design and Auditing
RLHF and Human Feedback
AI Quality Assurance
AI Model Assessment
Reviewer Quality Control
Prompt and Response Evaluation
Technical Content Review
Application Requirement: Please list the AI evaluation projects you have worked on, the organizations or platforms involved, and your specific responsibilities.
Contract Details
Compensation: $60-$85 per hour
Work arrangement: Fully Remote
Schedule: Flexible, with consistent availability expected
Payments: Weekly via Stripe or Wise
Join a team working with leading AI labs to improve the quality, reliability, and evaluation of next-generation AI systems.
About Hirely
Hirely connects qualified professionals with remote opportunities in AI research, business, software engineering, and technology. We are recruiting candidates on behalf of a hiring partner, which manages final candidate selection, onboarding, project assignments, and engagement terms.

Skills
LLM, Auditing, Onboarding, Quality Control

Work arrangement
Yes

Skills from the listing

  • LLM
  • Auditing
  • Onboarding
  • Quality Control

Your early-applicant advantage

How quickly JobBeeper finds and shares openings.

16 min from Linkedin publishing this role to us finding it
10 min median time from a role going live to a subscriber being told
6 hours subscribers had this role before this page existed
68,127 roles found in the last 24 hours
See available plans →

Job alerts

Get notified about new openings

Save your preferences, then add a paid plan for matching job alerts.

Create free account
Log in to apply