Research Scientist – AI Benchmarks & Evaluations

About Dusker AI
Dusker AI specializes in developing high-quality benchmarks, evaluation frameworks, and real-world tasks to assess the capabilities of frontier AI models and autonomous agents.
Our work focuses on Terminal-Bench, SWE-Bench, coding agents, LLM evaluations, and agentic workflows, helping leading AI organizations measure reasoning, reliability, adaptability, robustness, and safety.
We go beyond traditional benchmark scores to investigate how AI systems perform in complex, real-world environments, identify critical failure modes, and develop more reliable evaluation methodologies.
As we expand our research capabilities, we are establishing a dedicated AI Benchmarks & Evaluations Research Group in Jaipur to advance the science of evaluating frontier AI systems.
Role Description
We are seeking a highly motivated Research Scientist – AI Benchmarks & Evaluations to lead research initiatives focused on designing, developing, and improving evaluation methodologies for large language models (LLMs), AI coding agents, and autonomous systems.
This is a full-time, on-site position based in Jaipur, India.
The Research Scientist will work with engineers, domain experts, and academic collaborators to develop challenging benchmarks, analyze frontier AI model performance, identify systematic failure patterns, and contribute original research to the broader AI community.
The role offers access to real-world benchmark datasets, agent execution traces, evaluation infrastructure, and computing resources to support rigorous experimentation and research.
A key objective will be to develop novel evaluation frameworks and publish meaningful research at leading AI conferences and workshops.
Key Responsibilities
Benchmark Research & Development: Design innovative benchmarks and evaluation tasks for frontier LLMs, AI coding agents, and autonomous systems, including Terminal-Bench and SWE-Bench-style environments.
Evaluation Frameworks: Develop reliable, reproducible evaluation protocols and automated grading systems that minimize reward hacking, benchmark contamination, and evaluation bias.
Frontier Model Evaluation: Conduct experiments on leading AI models and agent frameworks to assess reasoning, tool use, coding proficiency, reliability, and task completion capabilities.
Failure Mode Analysis: Investigate model failures, agent execution traces, error patterns, and performance limitations to identify opportunities for improving AI capabilities.
Research & Publications: Conduct original research, formulate hypotheses, perform controlled experiments, and contribute publications to leading AI research venues such as NeurIPS, ICML, ICLR, ACL, and related conferences.
Evaluation Infrastructure: Build scalable evaluation pipelines using Python, Linux, Docker, Git, and modern ML/LLM tooling.
Dataset Development: Work with expert-generated benchmark tasks, real-world software engineering challenges, and evaluation datasets to ensure quality, diversity, and difficulty.
Academic Collaboration: Collaborate with professors, research laboratories, and academic institutions across India on AI benchmarking and evaluation research.
Research Communication: Prepare technical reports, research papers, experimental documentation, and presentations for internal stakeholders, research partners, and clients.
Cross-Functional Collaboration: Work closely with engineering and research teams to translate experimental findings into actionable improvements in benchmarking methodologies.
Qualifications
Master's or PhD in Computer Science, Artificial Intelligence, Machine Learning, Natural Language Processing, Software Engineering, or a closely related discipline.
Strong research background in machine learning, LLMs, AI agents, model evaluation, or software engineering research.
Demonstrated research contributions through peer-reviewed publications, open-source benchmark projects, evaluation tools, or publicly available datasets.
Advanced proficiency in Python, including scientific computing, statistical analysis, and machine learning libraries.
Practical experience with Linux, Docker, Git, and reproducible software development environments.
Hands-on experience working with large language models, AI coding agents, agent frameworks, or model inference systems.
Ability to design experimental methodologies, construct evaluation metrics, and conduct rigorous statistical analysis.
Experience working with real-world datasets, automated evaluation pipelines, and model performance analysis.
Strong understanding of AI reasoning, robustness, reliability, generalization, and failure analysis.
Excellent research writing, analytical thinking, technical documentation, and presentation skills.
Ability to work independently on research problems while collaborating effectively with multidisciplinary teams.
Willingness to work full-time from our Jaipur office.
Preferred Qualification
Publications at recognized AI/ML research venues such as NeurIPS, ICML, ICLR, ACL, EMNLP, or AAAI.
Contributions to benchmarks such as Terminal-Bench, SWE-Bench, OSWorld, GAIA, or similar agent evaluation initiatives.
Experience with LLM evaluation frameworks, agent execution environments, automated graders, or sandboxed coding environments.
Understanding of benchmark contamination, reward hacking, pass@k evaluation, and statistical significance in model comparisons.
Experience with model serving or evaluation infrastructure such as vLLM, Hugging Face Transformers, or other inference frameworks.
Familiarity with reinforcement learning, tool-using agents, long-horizon tasks, and multi-step reasoning.
Prior collaboration with academic research groups or experience developing publicly released benchmark datasets.
What we offer
Opportunity to become an early member of Dusker AI's dedicated research division.
Access to real-world AI benchmark datasets, agent execution data, and computing resources.
Freedom to explore novel research directions in AI benchmarking and agent evaluation.
Opportunities to publish original research under your own name and contribute to the global AI research community.
Collaboration with experienced engineers, domain experts, and academic researchers.
Direct exposure to challenging problems in frontier AI evaluation and autonomous agent systems.
Opportunity to help shape the future of AI benchmarking and evaluation methodologies.

Work arrangement
No