AI Evaluation-Benchmarking--W2 Remote contract

Software Engineer — AI Evaluation, Code Benchmarking & Automation
Role Overview
Help build and scale the tooling used to measure how well AI-powered software development tools and coding agents actually perform. You'll develop evaluation harnesses, automate benchmark runs, and build code benchmarking frameworks that compare AI-generated code against established baselines and real-world engineering outcomes. This is a hands-on engineering role focused on ensuring evaluations are accurate, reproducible, meaningful, and scalable.
Key Responsibilities
Build and integrate evaluation harnesses and automation for software development use cases, including turning real engineering artifacts such as merged pull requests into repeatable benchmark tasks.
Design and execute code benchmarks to evaluate AI-generated code for correctness, functionality, code quality, maintainability, test coverage, performance, and efficiency.
Compare AI-generated solutions against human-written/reference implementations and other AI-generated solutions using consistent benchmark criteria and automated test suites.
Build versioned, repeatable processes to evaluate AI tools, coding agents, models, and harnesses, with reproducible run environments including pinned dependencies, containerized runs, and isolated Git worktrees.
Develop automated testing and benchmarking workflows that validate generated code through unit tests, integration tests, static analysis, performance tests, and other execution-based evaluations.
Validate and calibrate evaluation approaches against human judgment, ensuring that benchmark scores are not only repeatable but also accurately reflect real-world code quality and engineering outcomes.
Support benchmarking across quality, productivity, performance, cost, and latency, including measuring the effort and resources required to achieve successful coding outcomes.
Analyze results across repeated benchmark runs, identifying variance, failure patterns, regressions, performance differences, and cost per successful outcome.
Investigate benchmark failures and evaluation inconsistencies, determine root causes, and improve the reliability and automation of evaluation workflows.
Work closely with engineering and data teams to improve benchmarking infrastructure, evaluation methodologies, datasets, and reporting.
Document benchmark methodology, results, limitations, and findings for both technical engineering teams and leadership audiences.
Required Skills & Experience
Strong software engineering background with hands-on experience building automation, developer tooling, benchmarking, testing, or validation systems.
Strong proficiency in Python, with experience in at least one additional language such as Java, JavaScript, TypeScript, or a similar programming language.
Experience designing or implementing code benchmarks, automated code evaluation, software testing, or comparative analysis of code implementations.
Strong understanding of software engineering fundamentals, including code quality, correctness, maintainability, testing, performance, and debugging.
Solid working knowledge of Git, including branches, history, commits, worktrees, and repository workflows.
Experience with Docker/containerization, APIs, development environments, CI/CD pipelines, and typical software engineering workflows.
Experience building automated test and evaluation pipelines capable of executing and comparing multiple code implementations or solutions.
Familiarity with AI/LLM/agent evaluation concepts, including why a judge can be consistent but incorrect, why a single run can be misleading, evaluation variance, and benchmark contamination.
Experience analyzing benchmark results and determining whether a measurement is technically valid and representative of the underlying engineering outcome.
Strong troubleshooting, analytical, and problem-solving skills with the ability to identify issues in both software implementations and evaluation methodologies.

Work arrangement
Yes