AI Data Quality Control (QC) Auditor — Coding Specialist

Role Overview
We’re looking for a Coding Specialist QC Auditor to serve as the final check before coding-focused AI training data reaches the client. You’ll review human-labeled tasks and AI-generated responses, primarily Python work, against project-specific rubrics to catch objective issues in code, logic, formatting, factual accuracy, and instruction-following.
This is a hands-on auditing role, not a guideline-writing or people-management role. You’ll use your technical judgment to assess whether the project can be reliably audited using standardized criteria and apply coding expertise when the work requires deeper evaluation.
Key Responsibilities
Review coding tasks and model responses against standardized, multi-step checklists and project Audit Specs.
Evaluate whether the project’s audit needs can be covered by objective, standardized quality criteria or require coding-specific judgment.
Identify and reject objective failures, including broken code, flawed logic, unsupported claims, missed prompt requirements, PII leaks, formatting errors, and violations of negative constraints.
Review code correctness, multi-file execution logic, edge cases, and technical tradeoffs when the rubric requires subject-matter expertise.
Assess AI-agent behavior when relevant, including reasoning, tool selection, execution flow, runtime failures, and looping across multiple turns.
Write concise, evidence-based feedback that identifies the failed requirement and explains the issue clearly.
Meet volume and accuracy targets in a structured, high-focus workflow.
Stay current with changing project requirements, updated calibration specs, and process changes.
Requirements & Qualifications
Strong Python proficiency; most work will be Python-focused.
Experience writing and reviewing code, ideally in a team environment with peer review.
Solid understanding of computer science fundamentals.
Hands-on experience using coding LLMs.
Exceptional attention to detail and the ability to spot subtle code, logic, formatting, or instruction-following errors.
Comfortable working from a rubric while knowing when technical judgment is needed to assess complex work.
Strong reading comprehension and the ability to quickly evaluate complex prompts, code, and AI-generated responses.
Comfortable working in web-based tools and across multiple tabs to validate work.
Prior experience in software QA, testing, code review, technical data annotation, fact-checking, or QC is preferred.
This is a Contractor role with an expected commitment of 30 hours per week
Preferred Skills
Experience evaluating LLM-generated code.
Ability to read raw unified diffs, trace logic across multiple files, and identify edge cases without relying on IDE linters.
Working knowledge of Docker, virtual environments, and sandbox isolation.
Experience diagnosing runtime failures caused by an AI agent.
Ability to evaluate an agent’s reasoning, tool choices, and behavior across multi-turn tasks.
Full-stack web development experience, including JavaScript and related frameworks.
#HandshakeAI

Work arrangement
Yes