{"schemaVersion":"jobsearcher.job.v1","id":"9faf2e49a1b6d3e9ac39e928","url":"https://jobsearcher.com/jobs/9faf2e49a1b6d3e9ac39e928","canonicalUrl":"https://jobsearcher.com/jobs/9faf2e49a1b6d3e9ac39e928","title":"Software Engineer, Agent Optimization & Evaluation","description":"For our client, we are seeking a Software Engineer, Agent Optimization & Evaluation to join the team of a leader in the Media space.This role will focus on building the infrastructure and tooling needed to evaluate and improve AI agent performance at scale. You will help shape systems that measure quality, latency, reliability, and cost, while enabling data-driven decisions about routing, benchmarking, and regression detection. The position partners closely with cross-functional teams to integrate agent capabilities and support broader product and engineering goals. It is a strong fit for an engineer who enjoys combining large-scale systems work with applied AI evaluation.Location: Remote - US based candidates only, no visa sponsorship availableCompensation: $109,800 – $325,000 annuallyResponsibilities Design and build the agent optimization harness Develop evaluation environments to measure agent performance Build systems for automated evaluation and regression detection Design model routing strategies for optimization Define metrics to measure developer productivity Build tooling for analyzing agent behavior Collaborate with cross-functional teams to integrate agent capabilitiesQualifications 5-12+ years of engineering experience Extensive experience in large-scale production software systems Experience designing evaluation and benchmarking systems for AI or complex distributed systems Strong understanding of modern LLMs and AI agents Proven ability to optimize systems for quality, latency, reliability, and cost Excellent software engineering fundamentals Strong cross-functional collaboration skillsBenefits Comprehensive healthcare benefits and income replacement programs 401(k) with employer match Global benefit programs for lifestyle support Family planning support Gender-affirming care Mental health and coaching benefits Flexible vacation and paid volunteer time off Generous paid parental leaveOur client is an equal opportunity employer. We encourage you to apply even if you don’t meet every qualification—your background could be exactly what this team needs.","company":"Ladders","rawCompany":"ladders","city":"Denver","state":"CO","isRemote":false,"isActive":false,"createdAt":"2026-10-24T09:04:42.636Z","occupations":[{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"},{"code":"17-2112.02","title":"Validation Engineers","slug":"validation-engineers"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541810","title":"Advertising Agencies","slug":"advertising-agencies"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Software Engineer, Agent Optimization & Evaluation","description":"For our client, we are seeking a Software Engineer, Agent Optimization & Evaluation to join the team of a leader in the Media space.This role will focus on building the infrastructure and tooling needed to evaluate and improve AI agent performance at scale. You will help shape systems that measure quality, latency, reliability, and cost, while enabling data-driven decisions about routing, benchmarking, and regression detection. The position partners closely with cross-functional teams to integrate agent capabilities and support broader product and engineering goals. It is a strong fit for an engineer who enjoys combining large-scale systems work with applied AI evaluation.Location: Remote - US based candidates only, no visa sponsorship availableCompensation: $109,800 – $325,000 annuallyResponsibilities Design and build the agent optimization harness Develop evaluation environments to measure agent performance Build systems for automated evaluation and regression detection Design model routing strategies for optimization Define metrics to measure developer productivity Build tooling for analyzing agent behavior Collaborate with cross-functional teams to integrate agent capabilitiesQualifications 5-12+ years of engineering experience Extensive experience in large-scale production software systems Experience designing evaluation and benchmarking systems for AI or complex distributed systems Strong understanding of modern LLMs and AI agents Proven ability to optimize systems for quality, latency, reliability, and cost Excellent software engineering fundamentals Strong cross-functional collaboration skillsBenefits Comprehensive healthcare benefits and income replacement programs 401(k) with employer match Global benefit programs for lifestyle support Family planning support Gender-affirming care Mental health and coaching benefits Flexible vacation and paid volunteer time off Generous paid parental leaveOur client is an equal opportunity employer. We encourage you to apply even if you don’t meet every qualification—your background could be exactly what this team needs.","datePosted":"2026-10-24T09:04:42.636Z","dateModified":"2026-10-24T09:04:42.636Z","hiringOrganization":{"@type":"Organization","name":"Ladders","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Denver","addressRegion":"CO","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"9faf2e49a1b6d3e9ac39e928"},"url":"https://jobsearcher.com/jobs/9faf2e49a1b6d3e9ac39e928"}}