AI Quality Associate

AI Quality AssociateThe AI Quality Associate focuses on validating the quality of AI-generated outputs, agent behaviors, and AI-assisted workflows. This impactful role will be critical in building benchmark scenarios, defining scoring rubrics, evaluating business usefulness, and identifying failure patterns that conventional pass or fail software testing would not catch.This position will ideally be based in one of our US offices and is a hybrid role with the flexibility to work remotely 2 days per week (also eligible for remote candidates).This position is not eligible for immigration sponsorship.What You'll Do:AI Output EvaluationDesign and execute structured evaluations for AI-enabled features and workflowsAssess outputs for groundedness, instruction adherence, consistency, usefulness, tone, control compliance, and riskIdentify hallucinations, unsupported assertions, missing logic, and unsafe recommendationsBenchmark & Rubric DevelopmentBuild and maintain golden datasets, benchmark prompts, comparison sets, and scorecardsDevelop rubrics that allow quality to be measured consistently across releases and changesWorkflow & Model Change ValidationCompare performance across prompt versions, workflow revisions, tools, and modelsSupport release decisions with evidence on quality regression or improvementBusiness & Domain PartnershipWork closely with Finance SMEs, product managers, and engineers to determine what acceptable looks like in real business contextsHelp define human-review thresholds and escalation patterns for higher-risk use casesProduction FeedbackAnalyze reviewer feedback, override patterns, and live quality signals to improve evaluation coverage over timeYou Have:4+ years of experience in QA, analytics, business process validation, AI evaluation, operations, or similar rolesStrong writing, analysis, and pattern-recognition skillsExperience evaluating outputs against nuanced criteria rather than only binary correctnessExperience with finance, accounting, FP&A, transaction services, or business process design preferredAbility to work with structured rubrics, scenario libraries, and evidence-based reviews – you are thoughtful, precise, and highly discerningAbility to spot subtle output problems others missComfort with ambiguity but disciplined in scoring and documentationComfort collaborating across Engineering and business teams, with a focus on trust, usefulness, and business realityBachelor's degree preferredYou Are:Excited to be part of a growing team, with a focus on driving future growthPassionate about delivering exceptional client serviceKnowledgeable about the relationship between Private Equity sponsors and their portfolio companies, particularly within the office of the CFOWilling to get your hands dirty in the details of a project while simultaneously seeing the whole pictureComfortable managing projects with multiple complex workstreams while also focused on your single workstream (you are managing and doing)A self-starter with a strong work ethicFull of entrepreneurial spirit and comfortable in a fluid, flat organizationAble to effectively communicate complex issues and solutions, and raise issues to senior team members when necessaryA team player, able to work with team members across all levelsSomeone who enjoys mentoring others and doing meaningful workA leader of others; you lead by exampleThe annual salary for this role ranges from: $110,000 to $145,000 + benefits + bonus.Actual compensation packages are determined by evaluating a wide array of factors unique to each candidate, including but not limited to geographic location, skill set, years and depth of experience, education, certifications, cost of labor and internal equity.Accordion is an Equal Opportunity Employer. We are committed to building a team that represents a variety of backgrounds, perspectives, and skills. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, veteran status, or any other status protected under federal, state, or local law.