Annotation Quality Program Manager

The role
You'll own the quality bar for shift's egocentric training data: the golden sets, SOPs, and calibration system that every annotator, reviewer and auto-labeling model is measured against. You'll also work with engineering to stand up new annotation workflows in quality control, hand tracking, object identification, and action labelling.
What you'll do
Own the quality bar end-to-end: golden/benchmark sets, acceptance criteria, frame-level tolerance thresholds, and the audit loops that hold them
Author and version annotation SOPs, rubrics, and edge-case rules as the client spec evolves
Measure and report quality: IAA (kappa/alpha, tolerance-based agreement on temporal boundaries), acceptance/rejection, rework, cost-per-accepted-hour — and root-cause disagreements to guideline ambiguity vs. annotator error
Set up audit frameworks for quality control and drive human-in-the-loop operations to detect bad quality vs good quality data
Stand up annotator training and certification, run calibration sessions, and design human-in-the-loop workflows where models pre-label, and humans review, correct, and escalate
Partner with engineering and product on new workflows (hand tracking, object identification, action labeling): label schemas, tool requirements, QC dashboards, and pilot validation before production scaling
What we're looking for
5+ years hands-on video annotation and quality control, including 1+ year with egocentric/first-person video for robotics or embodied AI
3+ years in a lead or QA capacity: calibrating annotators, adjudicating disagreements, owning guidelines others work to
Built 0→1 annotation programs and taken them from pilot to production
Authored (not just followed) SOPs and rubrics, with deep command of quality methodology: IAA frameworks, gold-set creation, sampling strategies, tolerance thresholds
Hands-on with 2+ professional annotation tools (CVAT, Labelbox, Encord, V7, Label Studio, or equivalent), configuring label schemas and QA workflows — including auditing model-generated labels
Bonus
Robotics, mechatronics, or computer-vision engineering background
Managed distributed annotator workforces or external vendors
Multi-modal annotation (3D/depth, joint pose, grasp outcome classification)
Trained or fine-tuned auto-labeling models, or built annotation tooling
Set up quality control and audit frameworks and scaled the operations
Why this role
The quality bar is the product: shift's ground truth becomes the benchmark physical-AI models are measured against
Ground-floor ownership at a funded, fast-growing startup - you build the quality system from scratch
About shift
shift pays everyday workers to record short videos of the tasks they already do (in kitchens, workshops, job sites, bakeries, and warehouses) to help train the next generation of AI and robotics. We’ve paid out over $15M to 25,000+ workers across 15+ countries and are now looking to scale further.

Work arrangement
No