Shopify AI LLM QA Engineer (PLAYWRIGHT)

About The Opportunity
A fast-scaling AI-powered e-commerce enablement platform, we build intelligent, LLM-driven QA systems that validate Shopify storefronts, automate regression testing, and ensure flawless generative experiences at scale. Our AI QA engineers work at the intersection ofShopify ecosystems, large language models, and test automation to ship reliable, customer-centric digital experiences across global merchant bases.
Role & Responsibilities
Design and execute AI-powered test cases for Shopify themes, LLM-generated content, and AI-driven checkout flows.
Build and maintain automated test suites using Playwright/Selenium integrated with LLM validation layers for dynamic content.
Collaborate with AI/ML engineers to define test heuristics for LLM outputs—accuracy, hallucination detection, prompt fidelity, and tone alignment.
Leverage synthetic test data and prompt engineering to simulate edge-case user journeys across Shopify stores.
Integrate QA pipelines with CI/CD (GitHub Actions, Jenkins) to enable shift-left testing for AI-enhanced Shopify features.
Monitor test coverage, flakiness, and LLM drift; recommend improvements to test architecture and validation logic.
Skills & Qualifications
Must-Have
Playwright
Selenium
Shopify Liquid
LLM Validation
CI/CD Pipelines
Python
Test Automation Frameworks
GitHub Actions
Preferred
LangChain
OpenAI API
Shopify Admin API
Benefits & Culture Highlights
100% remote, flexible hours with async-first collaboration.
Work directly on AI testing infrastructure for global Shopify merchants.
Access to cutting-edge LLM tooling and testing frameworks—learn while you build.
Skills: engineers,github,selenium,python,automation,api,validation,playwright,shopify,javascript,testing

Work arrangement
Yes