Ai QA Engineers

AI Quality Assurance (QA) Engineer
Job Type: Full-time, Contract Job
Overview AI Quality Assurance (QA) Engineer.
In this role, you will test, evaluate, and ensure the reliability of customer-facing AI agents and multi-step workflows for a fast-growing B2B SaaS client in the procurement space. This role is perfect for someone with strong backend testing intuition who loves to break and evaluate software systems. You will not follow a rigid manual script. Instead, you will design automated evaluation pipelines, conduct security/adversarial testing, and track errors through complex workflows to ensure our client's AI agents execute accurately and handle edge cases gracefully. If you enjoy discovering prompt vulnerabilities, analyzing performance trade-offs, and building automated assertion checks, we want to hear from you.
What You'll Do System Evaluation & Benchmarking
● Design Evaluation Frameworks: Build and maintain automated testing pipelines and metrics to validate system updates and prevent bugs.
● Establish Test Datasets: Partner with engineering and product teams to create highquality test datasets ("Golden Sets") that reflect real-world workflows.
● Validate Model Performance: Set up automated scoring systems where secondary models evaluate and grade the quality, accuracy, and compliance of primary outputs. Workflow & Integration Auditing
● Verify System Transitions: Test multi-step behaviors to ensure logical flow, correct execution order, and accurate memory retention across tasks.
● Schema & API Verification: Ensure systems reliably output structured, compliant data to prevent errors when connecting to external APIs and databases.
● Performance Tracking: Trace processing bottlenecks and audit complex execution paths to optimize speed and resource usage. Vulnerability, Safety & Security
● Adversarial Testing: Conduct proactive security testing, including prompt-injection and jailbreaking attempts, to prevent unauthorized system actions.
● Data Isolation Audits: Ensure applications strictly respect user privacy boundaries, data filtering rules, and API authentication scopes.
● Grounding Audits: Validate that generated system citations link accurately back to source documents to prevent false information. What We're Looking For Core Traits (Non-Negotiable)
● A "Breaker" Mindset: You have high technical intuition and proactively think of edge cases, system bottlenecks, and how to exploit vulnerabilities rather than waiting for a test to fail.
● Attention to Detail: You possess a meticulous nature that ensures no minor error or edge case goes unnoticed.
● High Level of Initiative: You naturally take ownership of loosely defined quality metrics and turn them into reproducible, automated test suites.
● Stepwise Troubleshooting Instinct: When a system fails or throws an API error, you walk through how to isolate the step, payload, or bottleneck rather than relying on a generic checklist.
● Strong Communication Skills: Direct written and spoken English, allowing you to easily translate complex technical failures into clear, actionable feedback for engineers.
● Critical Thinking: You approach problems with deep analytical rigor, questioning assumptions and thoroughly evaluating system logic. Technical Skills
● Professional QA, Software Engineering, or system evaluation experience with a heavy focus on backend testing, API validation, or software auditing.
● Automated Testing Mastery: Strong experience writing clean, modular automated test suites using modern backend scripting languages and testing frameworks, along with comfort handling asynchronous patterns.
● Hands-on Software & Workflow Experience: Familiarity with data structures, evaluation frameworks, database structures, and prompt behaviors.
● Data Validation & Assertion: Proficiency in verifying structured data outputs, schema compliance, and testing APIs. Personal Projects (Highly Valued)
● You experiment with software evaluation frameworks, safety parameters, or automated tools on your own time.
● You have built, audited, or actively tried to find bugs in automated systems, and can talk concretely about what broke, how you patched it, and what you learned. Note: Formal titles matter less than demonstrated ability. We care far more about what you've built than where you've worked. What We Offer ● Consistent, long-term contract work with a stable, high-growth B2B SaaS client.
● Work Setup: Fully Remote (Work From Home).
The opportunity to shape the quality standards of advanced systems managing critical enterprise workflows. About Data Skill Source Data Skill Source partners with fast-growing SaaS companies to deliver managed services, systems support, and applied automation. We focus on practical, production-ready technology—not demos or experiments that never ship.
How to Apply
Please submit your resume and a short note describing:
1. Backend or automation test suites you have built from scratch.
2. Any personal projects where you built, tested, or experimented with automation, security, or data workflows.
3. Your experience troubleshooting API integrations or complex schema structures.

Work arrangement
Yes