{"schemaVersion":"jobsearcher.job.v1","id":"6de4f7976329dd92cae348cd","url":"https://jobsearcher.com/jobs/6de4f7976329dd92cae348cd","canonicalUrl":"https://jobsearcher.com/jobs/6de4f7976329dd92cae348cd","title":"Lead Consultant - QA(AI) Engineer","description":"Join Delphi - Where Innovation meets transformation\r\nAt Delphi, we believe in creating an environment where our people thrive. Ourhybrid work modelempowers you to choose where you work - whether it's from the office, your home, or a mix of both - so you can prioritize what matters most. We are committed to supporting your personal goals, family, and overall well-being while driving transformative results for our clients.We welcome exceptional talent from anywhere across the globe. Interviews and onboarding are conducted virtually, reflecting our digital-first mindset.Rooted in the region, we specialize in delivering tailored, impactful solutions inData, Advanced Analytics and AI, Infrastructure, Cloud Security, and Application Modernization . Whether it’s enablingpredictive analytics , transforming operations with automation, or driving customer engagement with intelligent platforms, we are the trusted partner for organizations ready to embrace a smarter, more efficient future.We are looking for a hands‑on Senior QA Consultant to lead end‑to‑end testing of AI and generative AI applications across enterprise environments. This role combines strong expertise in traditional QA engineering with modern AI evaluation and validation practices. The ideal candidate will drive quality assurance initiatives for RAG pipelines, multi‑agent systems, OCR and speech‑to‑text solutions while ensuring production‑grade quality outcomes in regulated industries such as finance, healthcare, and insurance. The successful candidate will lead a small QA pod, collaborate closely with engineering, AI/ML, product, and business teams, and act as the client‑facing QA owner for enterprise AI engagements. This role requires strong technical leadership, automation expertise, AI evaluation capabilities, and excellent stakeholder management skills.Experience Requirements10–15 years of experience in Software QA / Test Engineering3–4 years of experience leading QA teams or delivery podsHands‑on experience testing AI / GenAI applications in production environmentsExperience working within regulated domains such as finance, healthcare, insurance, or similar industriesKey Responsibilities\r\nQuality Engineering & DeliveryOwn end‑to‑end quality assurance for AI/GenAI engagements including test strategy, planning, execution, defect management, reporting, and exit criteriaLead and mentor QA pods of 3–6 engineers across automation frameworks, AI testing practices, and delivery executionDesign and execute functional, API, backend, integration, regression, performance, and AI evaluation test casesBuild and maintain scalable automation frameworks for web, mobile, and API testingDrive continuous quality improvements through reusable accelerators, frameworks, datasets, and evaluation toolsAI Application Testing & EvaluationDefine AI‑specific test coverage including prompt validation, hallucination detection, groundedness checks, retrieval quality, tool‑call correctness, latency, bias, and safety testingDesign and execute testing strategies for RAG architectures, multi‑agent workflows, OCR systems, and speech‑to‑text applicationsBuild synthetic datasets covering edge cases, adversarial scenarios, persona‑based workflows, and multimodal inputsImplement LLM‑as‑a‑Judge evaluation pipelines using customizable scoring rubrics and multi‑model evaluationsProduce detailed AI quality scorecards and benchmarking reports across accuracy, relevancy, faithfulness, and safety metricsAutomation, CI/CD & ToolingIntegrate AI evaluation and automation testing into Jenkins and Azure DevOps CI/CD pipelinesBuild regression dashboards, quality gates, and automated reporting frameworksPerform API testing, backend validation, database testing, and event‑driven workflow validationConduct load and performance testing using modern performance engineering toolsSupport cross‑browser, parallel, and scalable automation execution environmentsClient & Stakeholder ManagementAct as the client‑facing QA lead for enterprise engagementsConduct quality reviews, risk assessments, and stakeholder reporting sessionsTranslate technical quality metrics into business‑focused outcomes and recommendationsCollaborate with engineering, AI/ML, product, and leadership teams to ensure successful delivery outcomesRequired Technical Skills -\r\nCore QA & AutomationFunctional Testing, Exploratory Testing, Requirements Traceability, and Test Design TechniquesAPI Testing using REST, GraphQL, Postman/Newman, RestAssured, and RequestsBackend and Database Testing across SQL Server, MySQL, Oracle, PostgreSQL, and ETL/Data PipelinesAutomation Frameworks using Selenium, Playwright, PyTest, and AppiumPerformance Testing using JMeter, k6, or LocustProgramming & Development -Strong programming expertise in Python or JavaKnowledge of OOP concepts, design patterns, data structures, and unit testing frameworksAbility to build utilities, custom assertions, CLI tools, and reusable automation componentsAI & Generative AI TestingHands‑on experience testing AI/GenAI applications in enterprise environmentsKnowledge of LLM platforms including OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, and Vertex AIExperience with orchestration frameworks such as LangChain, LlamaIndex, and Semantic KernelUnderstanding of multi‑agent systems, memory handling, routing, tool calling, and guardrailsExpertise in RAG evaluation, vector databases, embeddings, retrieval quality, and hallucination testingFamiliarity with AI evaluation platforms such as RAGAS, DeepEval, Galileo, Arize Phoenix, Patronus, Okareo, or similar toolsExperience validating OCR systems and speech‑to‑text solutions using WER/CER and degradation testing methodologiesBuild, DevOps & PlatformsAzure DevOps, Jenkins, JIRA/Xray or Zephyr, GitHub/GitLab/BitbucketCI/CD pipelines, regression dashboards, and quality gate implementationExposure to Power Platform tools including Power Apps, Power BI, and Power Automate is a plusDomain & Compliance ExpertiseExperience working in regulated industry environments such as banking, financial services, healthcare, insurance, or life sciencesUnderstanding of compliance requirements including data privacy, PII/PHI handling, explainability, and AI governance practicesBehavioral & Consulting SkillsStrong leadership and mentoring capabilitiesExcellent verbal and written communication skillsClient‑centric and solution‑oriented mindsetStrong stakeholder management and collaboration abilitiesOwnership mindset with proactive problem‑solving approachAbility to work effectively in fast‑paced consulting environmentsContinuous learner with strong interest in emerging AI testing methodologies and toolsPreferred QualificationsBachelor’s degree in Computer Science, Engineering, or a related fieldMaster’s degree preferredCertifications such as ISTQB Advanced, Certified Agile Tester, AWS/Azure AI Certifications, or equivalentExperience with adversarial testing and responsible AI frameworks such as NIST AI RMF or ISO 42001Exposure to MLOps / LLMOps observability and production monitoringContributions to QA or AI communities through blogs, speaking sessions, or open source initiativesWhat Success Looks LikeHigh‑quality delivery outcomes with measurable reduction in defect leakageSuccessful implementation of reusable AI QA accelerators and automation frameworksStrong stakeholder and client feedback across engagementsGrowth and technical enablement of QA pod members under leadershipContinuous improvement of AI testing practices, evaluation frameworks, and delivery standardsWhat we offer\r\nAt Delphi, we are dedicated to creating an environment where you can thrive, both professionally and personally. Ourcompetitive compensation package, performance‑based incentives,and health benefits are designed to ensure you’re well‑supported. We believe in your continuous growth and offercompany‑sponsored certifications, training programs , and skill‑building opportunities to help you succeed.We foster a culture of inclusivity and support, withremote workoptions and a fully supported work‑from‑home setup to ensure your comfort and productivity. Our positive and inclusive culture includes team activities, wellness and mental health programs to ensure you feel supported.#J-18808-Ljbffr","company":"Remote","rawCompany":"remote","city":"Millbrae","state":"CA","isRemote":false,"isActive":false,"createdAt":"2026-11-04T03:16:14.449Z","occupations":[{"code":"15-1253.00","title":"Software Quality Assurance Analysts and Testers","slug":"software-quality-assurance-analysts-and-testers"},{"code":"11-3051.01","title":"Quality Control Systems Managers","slug":"quality-control-systems-managers"},{"code":"19-4099.01","title":"Quality Control Analysts","slug":"quality-control-analysts"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Lead Consultant - QA(AI) Engineer","description":"Join Delphi - Where Innovation meets transformation\r\nAt Delphi, we believe in creating an environment where our people thrive. Ourhybrid work modelempowers you to choose where you work - whether it's from the office, your home, or a mix of both - so you can prioritize what matters most. We are committed to supporting your personal goals, family, and overall well-being while driving transformative results for our clients.We welcome exceptional talent from anywhere across the globe. Interviews and onboarding are conducted virtually, reflecting our digital-first mindset.Rooted in the region, we specialize in delivering tailored, impactful solutions inData, Advanced Analytics and AI, Infrastructure, Cloud Security, and Application Modernization . Whether it’s enablingpredictive analytics , transforming operations with automation, or driving customer engagement with intelligent platforms, we are the trusted partner for organizations ready to embrace a smarter, more efficient future.We are looking for a hands‑on Senior QA Consultant to lead end‑to‑end testing of AI and generative AI applications across enterprise environments. This role combines strong expertise in traditional QA engineering with modern AI evaluation and validation practices. The ideal candidate will drive quality assurance initiatives for RAG pipelines, multi‑agent systems, OCR and speech‑to‑text solutions while ensuring production‑grade quality outcomes in regulated industries such as finance, healthcare, and insurance. The successful candidate will lead a small QA pod, collaborate closely with engineering, AI/ML, product, and business teams, and act as the client‑facing QA owner for enterprise AI engagements. This role requires strong technical leadership, automation expertise, AI evaluation capabilities, and excellent stakeholder management skills.Experience Requirements10–15 years of experience in Software QA / Test Engineering3–4 years of experience leading QA teams or delivery podsHands‑on experience testing AI / GenAI applications in production environmentsExperience working within regulated domains such as finance, healthcare, insurance, or similar industriesKey Responsibilities\r\nQuality Engineering & DeliveryOwn end‑to‑end quality assurance for AI/GenAI engagements including test strategy, planning, execution, defect management, reporting, and exit criteriaLead and mentor QA pods of 3–6 engineers across automation frameworks, AI testing practices, and delivery executionDesign and execute functional, API, backend, integration, regression, performance, and AI evaluation test casesBuild and maintain scalable automation frameworks for web, mobile, and API testingDrive continuous quality improvements through reusable accelerators, frameworks, datasets, and evaluation toolsAI Application Testing & EvaluationDefine AI‑specific test coverage including prompt validation, hallucination detection, groundedness checks, retrieval quality, tool‑call correctness, latency, bias, and safety testingDesign and execute testing strategies for RAG architectures, multi‑agent workflows, OCR systems, and speech‑to‑text applicationsBuild synthetic datasets covering edge cases, adversarial scenarios, persona‑based workflows, and multimodal inputsImplement LLM‑as‑a‑Judge evaluation pipelines using customizable scoring rubrics and multi‑model evaluationsProduce detailed AI quality scorecards and benchmarking reports across accuracy, relevancy, faithfulness, and safety metricsAutomation, CI/CD & ToolingIntegrate AI evaluation and automation testing into Jenkins and Azure DevOps CI/CD pipelinesBuild regression dashboards, quality gates, and automated reporting frameworksPerform API testing, backend validation, database testing, and event‑driven workflow validationConduct load and performance testing using modern performance engineering toolsSupport cross‑browser, parallel, and scalable automation execution environmentsClient & Stakeholder ManagementAct as the client‑facing QA lead for enterprise engagementsConduct quality reviews, risk assessments, and stakeholder reporting sessionsTranslate technical quality metrics into business‑focused outcomes and recommendationsCollaborate with engineering, AI/ML, product, and leadership teams to ensure successful delivery outcomesRequired Technical Skills -\r\nCore QA & AutomationFunctional Testing, Exploratory Testing, Requirements Traceability, and Test Design TechniquesAPI Testing using REST, GraphQL, Postman/Newman, RestAssured, and RequestsBackend and Database Testing across SQL Server, MySQL, Oracle, PostgreSQL, and ETL/Data PipelinesAutomation Frameworks using Selenium, Playwright, PyTest, and AppiumPerformance Testing using JMeter, k6, or LocustProgramming & Development -Strong programming expertise in Python or JavaKnowledge of OOP concepts, design patterns, data structures, and unit testing frameworksAbility to build utilities, custom assertions, CLI tools, and reusable automation componentsAI & Generative AI TestingHands‑on experience testing AI/GenAI applications in enterprise environmentsKnowledge of LLM platforms including OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, and Vertex AIExperience with orchestration frameworks such as LangChain, LlamaIndex, and Semantic KernelUnderstanding of multi‑agent systems, memory handling, routing, tool calling, and guardrailsExpertise in RAG evaluation, vector databases, embeddings, retrieval quality, and hallucination testingFamiliarity with AI evaluation platforms such as RAGAS, DeepEval, Galileo, Arize Phoenix, Patronus, Okareo, or similar toolsExperience validating OCR systems and speech‑to‑text solutions using WER/CER and degradation testing methodologiesBuild, DevOps & PlatformsAzure DevOps, Jenkins, JIRA/Xray or Zephyr, GitHub/GitLab/BitbucketCI/CD pipelines, regression dashboards, and quality gate implementationExposure to Power Platform tools including Power Apps, Power BI, and Power Automate is a plusDomain & Compliance ExpertiseExperience working in regulated industry environments such as banking, financial services, healthcare, insurance, or life sciencesUnderstanding of compliance requirements including data privacy, PII/PHI handling, explainability, and AI governance practicesBehavioral & Consulting SkillsStrong leadership and mentoring capabilitiesExcellent verbal and written communication skillsClient‑centric and solution‑oriented mindsetStrong stakeholder management and collaboration abilitiesOwnership mindset with proactive problem‑solving approachAbility to work effectively in fast‑paced consulting environmentsContinuous learner with strong interest in emerging AI testing methodologies and toolsPreferred QualificationsBachelor’s degree in Computer Science, Engineering, or a related fieldMaster’s degree preferredCertifications such as ISTQB Advanced, Certified Agile Tester, AWS/Azure AI Certifications, or equivalentExperience with adversarial testing and responsible AI frameworks such as NIST AI RMF or ISO 42001Exposure to MLOps / LLMOps observability and production monitoringContributions to QA or AI communities through blogs, speaking sessions, or open source initiativesWhat Success Looks LikeHigh‑quality delivery outcomes with measurable reduction in defect leakageSuccessful implementation of reusable AI QA accelerators and automation frameworksStrong stakeholder and client feedback across engagementsGrowth and technical enablement of QA pod members under leadershipContinuous improvement of AI testing practices, evaluation frameworks, and delivery standardsWhat we offer\r\nAt Delphi, we are dedicated to creating an environment where you can thrive, both professionally and personally. Ourcompetitive compensation package, performance‑based incentives,and health benefits are designed to ensure you’re well‑supported. We believe in your continuous growth and offercompany‑sponsored certifications, training programs , and skill‑building opportunities to help you succeed.We foster a culture of inclusivity and support, withremote workoptions and a fully supported work‑from‑home setup to ensure your comfort and productivity. Our positive and inclusive culture includes team activities, wellness and mental health programs to ensure you feel supported.#J-18808-Ljbffr","datePosted":"2026-11-04T03:16:14.449Z","dateModified":"2026-11-04T03:16:14.449Z","hiringOrganization":{"@type":"Organization","name":"Remote","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Millbrae","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"6de4f7976329dd92cae348cd"},"url":"https://jobsearcher.com/jobs/6de4f7976329dd92cae348cd"}}