AI Quality Analyst - English

Hirely is recruiting AI Quality Analysts on behalf of a hiring partner for an AI evaluation project focused on personalized AI interactions.
You'll design multi-turn prompts and evaluate how effectively AI models use relevant personal context to produce grounded, natural, relevant, and helpful responses.
What You'll Do
Design creative multi-turn prompts based on realistic personal scenarios
Execute conversations of typically 1-5 turns to test AI personalization
Evaluate whether AI responses appropriately use relevant context
Assess responses across dimensions including Grounding, Integration, and Helpfulness
Identify incorrect personalization, unsupported inferences, and hallucinations
Evaluate whether personal information is integrated naturally without unnecessary repetition
Compare two AI responses side-by-side and determine which response is more helpful and natural
Identify subtle differences in response quality, usability, and naturalness
Write clear and defensible rationales explaining model rankings
Reference specific conversation turns when documenting evaluation decisions
Provide detailed annotations and constructive feedback
Extract and verify relevant debugging information from model outputs
Follow strict data-hygiene procedures throughout evaluation work
Who We're Looking For
Strong English reading and writing skills
Bachelor's degree or equivalent experience in a relevant field
Strong analytical and critical-thinking skills
Ability to evaluate nuanced and ambiguous AI responses
Strong written communication and ability to explain evaluation decisions clearly
Ability to design creative prompts and multi-turn conversations
Understanding of personalization, contextual reasoning, and AI response quality
Experience in AI evaluation, data annotation, content moderation, or related work is strongly preferred
Background in Computer Science, Linguistics, Journalism, Policy, Law, Ethics, or another analytical field is valuable
Desktop/laptop with a reliable internet connection
Compensation & Terms
Contractor engagement
At least 4 hours/day, up to 40 hours/week
Fully remote
Application Process
Submit your application and profile
Shortlisted candidates receive a Job Interest Form
Complete the assessment within 24 hours
Successful candidates proceed to pre-onboarding requirements
Important Requirement
This project requires candidates to use their primary personal Google account and enable relevant personal data sources for the evaluation. Candidates should only apply if they are comfortable with the project's stated data-access and evaluation requirements.
About Hirely
Hirely connects professionals with remote opportunities from hiring partners across artificial intelligence, software engineering, cloud, data, research and other specialised fields. Hirely acts as a recruitment and talent-sourcing partner for the hiring companies it represents. Final hiring decisions, onboarding, compensation and project assignments are determined by the hiring partner.

Work arrangement
No