QA Developer, AI Assistant

At Poka, we're changing how the world's industrial teams work — capturing the knowledge of their best people and putting it in every operator's hands.
Our connected worker platform is purpose-built for manufacturing and the wider industrial world, with industrial AI inside the shift, not on top of it. The result: frontline performance that becomes compounding knowledge and measurable impact — every shift, every site, every region.
We're proud to support global leaders like Nestlé, Bosch, Mars, and Danone across 3,000+ sites in 70+ countries. And as a fast-growing part of the IFS family, you'll join a team whose daily work directly shapes the platform behind how modern manufacturing gets done.
Role overview
Our team develops the AI assistant built into Poka. Workers and supervisors use it every day on manufacturing floors. We build it with prompts, tools, information retrieval and agents.
As a quality assurance (QA) specialist on this team, you make sure what we ship works: the assistant's answers and the features around it. You do it with automation, AI and tools you build, not with manual testing. You work with the devs, the product team and the design team to define what "good" means, then you build the evals, automated tests and tools that check it.
How this is different from traditional QA
No manual test passes. If something needs to be checked more than once, you automate it, with code, AI or whatever tool fits.
A large part of what you test has no single answer. The same question can get 50 good answers phrased differently, and the assistant can answer differently from one run to the next. You help define what "good" means and measure it.
You write Python every week.
If you're looking for a role focused on test plans and manual regression testing, this isn't it.
What you'll be responsible for
Automating QA for the team's features and building the tools that make it easier, including ways for non-developers to add evals and analyze the results.
Building and maintaining eval suites based on real usage: actual user questions, expected behavior and the edge cases that show up in production.
Defining with the team what a good answer looks like for our users, writing scoring rubrics and using LLM-as-judge where it makes sense, calibrated against human-labeled samples.
Running regression checks in CI every time we change a prompt, a tool, retrieval or the Claude model version, covering answer quality, cost and speed. You build the guardrails, evals and tests that tell us whether a release gets the green light or not.
Reviewing real conversations and grouping failures into patterns: made-up answers, wrong tool calls, missed context, inappropriate tone. Then working with developers to turn them into fixes.
Contributing to adversarial testing: prompt injection, data leaking between customers, getting the assistant to do things it shouldn't.
Doing timeboxed exploratory testing on features that change what the assistant can decide or do, like new agents, tools or actions, including checking that actions stay safe when something fails or gets retried.
Joining feature kickoffs to help define acceptance criteria, including AI-specific ones.
What you bring
AEC, DEC or BAC in a field related to the position, or equivalent experience
2 to 4 years of relevant experience
A background in test automation or as an SDET
You're comfortable with several programming languages, including Python and TypeScript
Real curiosity about how LLMs behave and why they fail
You're comfortable judging quality when there's no single answer, and explaining your reasoning
Curiosity about our users' world. You can't judge an answer without understanding what a supervisor on a manufacturing floor actually needs.
Assets
You've helped set up an eval harness, even in a personal project
Experience using AI tools to speed up testing
Basic statistics: pass rates, variance, sample size
Experience in security or adversarial testing
Knowledge of the test pyramid and the shift-left approach
B2B experience or experience on industrial platforms
Knowledge of Jira
The technologies you'll use
Python
TypeScript
Claude Code
Eval framework (promptfoo)
AWS
Docker
And many others
We strongly believe that a passion for new technologies, a positive attitude and real team spirit are key to succeeding at Poka.
Above all, we're looking for someone curious and resourceful. You care about the success of the product and your team. You like complex problems, especially the ones without an obvious solution. You want to write clean, high-quality code and keep learning.
If that sounds like you, you're probably the person we're looking for!
What we offer: