QA Analyst (AI Systems)

About CodeRoadCodeRoad provides end-to-end software development services, helping businesses scale with ideal infrastructure solutions. From staff augmentation to dedicated IT teams and general software engineering, our nearshore technology services empower businesses to thrive in an ever-evolving digital landscape.About the RoleWe’re looking for a QA Analyst specialized in modern AI systems. You’ll be responsible for validating not only APIs and performance, but also the behavior and reliability of AI-driven outputs. This role goes beyond traditional QA — you’ll test intelligent systems, not just deterministic ones.Key ResponsibilitiesDesign and execute test strategies for AI-powered APIs and workflowsValidate agent inputs/outputs and ensure consistency and reliabilityPerform API testing, exploratory testing, and automated testingRun performance, load, and bulk testing scenariosCollaborate with engineers to identify edge cases in AI behaviorContribute to testing frameworks for LLM-based systemsRequirements3+ years of experience in QA (Automation + Manual).Strong API Testing: Expert level with Postman, Insomnia, or RestAssured.Automation Specialist: Proficient in Playwright or Selenium for end-to-end flows.Performance & Scale: Experience with Load/Bulk testing using Locust or JMeter.AI/LLM Validation: Familiarity with testing Agentic workflows, validating AI payloads, and using tools like LangSmith.Observability: Exposure to OpenTelemetry for tracing and system monitoring.Analytical Rigor: Ability to design test cases for non-linear, agentic AI tasks.Advanced English: Fluent for documentation and global collaboration.Nice to HaveExperience testing AI/ML systemsFamiliarity with LangSmith or similar toolsExperience with MLFlow or model evaluation frameworksWhat You’ll Love100% RemoteHolidays offPaid Time OffHealth insurance assistance programCompetitive USD compensationStrong team culture and collaborative environmentOngoing training and growth opportunities