{"schemaVersion":"jobsearcher.job.v1","id":"3a3134ece23f84c7aa4df012","url":"https://jobsearcher.com/jobs/3a3134ece23f84c7aa4df012","canonicalUrl":"https://jobsearcher.com/jobs/3a3134ece23f84c7aa4df012","title":"Software Engineer - Benchmark Auditor","description":"About The JobMercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.Position: SWE-Bench Task AuditorType: ContractCompensation: $70–$90/hourLocation: RemoteRole ResponsibilitiesEvaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks.Assess repository-level tasks, reference patches, test harnesses, and grading integrity.Provide clear, rubric-based written feedback to improve AI model training.Audit reference patches, test runners, and Docker isolation to detect answer leakage and reward hacking.Work independently and asynchronously to meet deadlines and enhance AI model performance.QualificationsMust-Have3+ years professional software engineering experience.Real open-source contribution or maintainer experience (merged PRs, committer/maintainer roles).Strong ability to audit reference patches, test runners, and Docker isolation.Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++).PreferredFamiliarity with SWE-Bench (Verified) or similar repository benchmarks.Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.).Prior code-review or task-grading experience.Application Process (Takes 20–30 mins to complete)Upload resumeAI interview based on your resumeSubmit formResources & SupportFor details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcomeFor any help or support, reach out to: support@mercor.comPS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.","company":"Mercor","rawCompany":"mercor","city":"Millbrae","state":"CA","isRemote":false,"isActive":false,"createdAt":"2026-10-19T10:11:13.333Z","occupations":[{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1253.00","title":"Software Quality Assurance Analysts and Testers","slug":"software-quality-assurance-analysts-and-testers"},{"code":"15-1251.00","title":"Computer Programmers","slug":"computer-programmers"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Software Engineer - Benchmark Auditor","description":"About The JobMercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.Position: SWE-Bench Task AuditorType: ContractCompensation: $70–$90/hourLocation: RemoteRole ResponsibilitiesEvaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks.Assess repository-level tasks, reference patches, test harnesses, and grading integrity.Provide clear, rubric-based written feedback to improve AI model training.Audit reference patches, test runners, and Docker isolation to detect answer leakage and reward hacking.Work independently and asynchronously to meet deadlines and enhance AI model performance.QualificationsMust-Have3+ years professional software engineering experience.Real open-source contribution or maintainer experience (merged PRs, committer/maintainer roles).Strong ability to audit reference patches, test runners, and Docker isolation.Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++).PreferredFamiliarity with SWE-Bench (Verified) or similar repository benchmarks.Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.).Prior code-review or task-grading experience.Application Process (Takes 20–30 mins to complete)Upload resumeAI interview based on your resumeSubmit formResources & SupportFor details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcomeFor any help or support, reach out to: support@mercor.comPS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.","datePosted":"2026-10-19T10:11:13.333Z","dateModified":"2026-10-19T10:11:13.333Z","hiringOrganization":{"@type":"Organization","name":"Mercor","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Millbrae","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"3a3134ece23f84c7aa4df012"},"url":"https://jobsearcher.com/jobs/3a3134ece23f84c7aa4df012"}}