{"schemaVersion":"jobsearcher.job.v1","id":"250d77892f987c8a5f380bf8","url":"https://jobsearcher.com/jobs/250d77892f987c8a5f380bf8","canonicalUrl":"https://jobsearcher.com/jobs/250d77892f987c8a5f380bf8","title":"AI Evaluation & Benchmark Engineer","description":"AI & Machine Learning Generative Systems Hybrid · San Francisco, CA Full-Time$180,000 – $230,000 USD · 0.15% – 0.3% EquityEnsure Theme Framer models maintain the highest benchmarked accuracy in the industry by building automated sandbox testing frameworks.What You Will DoBuild automated sandbox runners that execute and verify generated code across 10,000+ unit test suites.Create synthetic benchmark datasets for Next.js, Tailwind, and full-stack API generation.Identify model failure modes and partner with research teams on dataset filtering and RL feedback.What We Are Looking For4+ years experience in Python, automated testing frameworks, and LLM evaluation tooling.Strong knowledge of continuous integration (CI) and containerized execution.Bonus QualificationsExperience with HumanEval, SWE-bench, or custom code evaluation suites.Primary Technology Stack\r\nPython Docker PyTest PostgreSQL FastAPI GitHub Actions100% Employer-Covered Health & DentalMeaningful Equity OwnershipRemote-First Setup Stipend ($2,500)Flexible Time Off & Mental Health DaysAnnual Company Engineering Retreats$2,000 / Year Continuous Learning BudgetShape The Future Of Autonomous Software\r\nExplore all open roles across compiler infrastructure, generative AI, and design systems.#J-18808-Ljbffr","company":"Themeframer","rawCompany":"themeframer","city":"Millbrae","state":"CA","isRemote":false,"isActive":false,"createdAt":"2026-11-04T03:16:17.067Z","occupations":[{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"},{"code":"15-1253.00","title":"Software Quality Assurance Analysts and Testers","slug":"software-quality-assurance-analysts-and-testers"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"AI Evaluation & Benchmark Engineer","description":"AI & Machine Learning Generative Systems Hybrid · San Francisco, CA Full-Time$180,000 – $230,000 USD · 0.15% – 0.3% EquityEnsure Theme Framer models maintain the highest benchmarked accuracy in the industry by building automated sandbox testing frameworks.What You Will DoBuild automated sandbox runners that execute and verify generated code across 10,000+ unit test suites.Create synthetic benchmark datasets for Next.js, Tailwind, and full-stack API generation.Identify model failure modes and partner with research teams on dataset filtering and RL feedback.What We Are Looking For4+ years experience in Python, automated testing frameworks, and LLM evaluation tooling.Strong knowledge of continuous integration (CI) and containerized execution.Bonus QualificationsExperience with HumanEval, SWE-bench, or custom code evaluation suites.Primary Technology Stack\r\nPython Docker PyTest PostgreSQL FastAPI GitHub Actions100% Employer-Covered Health & DentalMeaningful Equity OwnershipRemote-First Setup Stipend ($2,500)Flexible Time Off & Mental Health DaysAnnual Company Engineering Retreats$2,000 / Year Continuous Learning BudgetShape The Future Of Autonomous Software\r\nExplore all open roles across compiler infrastructure, generative AI, and design systems.#J-18808-Ljbffr","datePosted":"2026-11-04T03:16:17.067Z","dateModified":"2026-11-04T03:16:17.067Z","hiringOrganization":{"@type":"Organization","name":"Themeframer","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Millbrae","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"250d77892f987c8a5f380bf8"},"url":"https://jobsearcher.com/jobs/250d77892f987c8a5f380bf8"}}