JOBSEARCHER

AI Test Engineer

HaystackDenver, COL6 LeadNovember 8th, 2026
An employer is seeking an experienced AI Test Engineer to lead comprehensive quality assurance and evaluation strategies for advanced AI systems.
The Role: Define end-to-end testing strategies for non-deterministic systems, including golden datasets, LLM evaluation frameworks, regression suites, and human review workflows Develop automated evaluation harnesses to measure accuracy, groundedness, latency, toxicity, bias, and multimodal quality across speech, image, and voice models Perform adversarial and red-team testing to identify vulnerabilities, prompt injections, data leakage, and edge cases across varied languages and accents Build and maintain automated functional, performance, and load testing pipelines integrated into CI/CD quality gates across web and mobile platforms Monitor defect trends, track release-readiness metrics, and communicate quality risks to cross-functional stakeholders
What You'll Need: 12+ years of relevant experience in software testing and quality engineering Proven experience evaluating AI models and non-deterministic systems using modern automated evaluation frameworks Strong background in test automation across browsers, iOS, and Android platforms Familiarity with adversarial testing, red-teaming, and CI/CD quality integration What's On Offer Full-time employment with a fully remote work schedule Opportunity to work at the forefront of AI evaluation and quality engineering Collaborative, engineering-driven environment Apply via Haystack today!