Senior Quality Engineer 2 - AI - SOC Automation

As part of our Concierge Security approach, these insights are paired with expert human analysis to provide proactive protection for our customers. We are looking for passionate and experienced Quality Engineers to join our team and help build and enhance our next-generation testing platforms. In this role, you'll contribute to ensuring the reliability, performance, and quality of systems that are critical to delivering AI-powered security outcomes.
Responsibilities:
Create and extend software frameworks and tools to run, monitor, and report on automated tests.
Have experience building and maintaining Automation frameworks and CI/CD pipelines.
Review testing strategies, approaches, and test plans and provide insight to development teams.
Architect, design, and build software to automate a variety of testing methods (e. g., API validation, performance/load testing, fuzzing, etc. )
Develop and maintain test frameworks for GenAI-based and AI/ML applications, including model validation, LLM prompt-response consistency, hallucination detection, and regression testing for AI inference APIs.
Implement evaluation pipelines to test LLM accuracy, latency, bias, toxicity, and reliability.
Requirements:
Are excited to work at the intersection of AI/ML, Big Data and the Cybersecurity problem space
Have experience deploying services on cloud computing platforms (e. g, AWS, Azure, GCP)
Continuously learn and expand your technical horizons
Work as part of a team of highly skilled software professionals.
Are you an expert using common tools for executing functional, load and fuzz testing (e. g., Locust, CATS).
Have worked on distributed systems and microservices architecture (preferred).
Our Technology Stack: Pytest, Robot, RestAPIs, Python, Playwright, Docker, Kubernetes, AWS (Bedrock, Lambda, SQS, S3 etc. ), LangChain, LLM evaluation frameworks (e. g., Ragas), OpenAPI, Postgres, Linux.

Experience
3-6 yrs