{"schemaVersion":"jobsearcher.job.v1","id":"1e59055da63c0b9435bfb40a","url":"https://jobsearcher.com/jobs/1e59055da63c0b9435bfb40a","canonicalUrl":"https://jobsearcher.com/jobs/1e59055da63c0b9435bfb40a","title":"Software Engineering Task Auditor (SWE-Bench)","description":"Software Engineering Task Auditor (SWE-Bench)United States - Remote | Independent contractor | US$70-US$90/hourAbout the opportunityHumanitApp is coordinating applications for an external hiring provider. This is not employment with HumanitApp. The project helps an AI research lab assess whether software-engineering benchmark tasks are correct, reproducible and fairly graded.In practical terms, you will review a coding task, the proposed fix and the tests used to judge it. Your feedback should explain whether the fix solves the right problem, whether the tests are reliable and whether another engineer can reproduce the result.What you will do- Review repository-level coding tasks and reference patches for correctness.- Check test runners and Docker environments for reliable, reproducible execution.- Identify answer leakage, misleading tests and ways a solution could game the scoring.- Write concise, evidence-based feedback using the project's review rubric.What you bring- At least 3 years of professional software-engineering experience.- Real open-source contribution or maintainer experience, such as merged pull requests.- Python proficiency plus at least one of Java, Go, TypeScript or C++.- Experience reviewing patches, running tests and working with Docker isolation.- Clear technical writing and careful code-review judgment.Helpful, but not requiredExperience with SWE-Bench or similar repository benchmarks; contributions to Python projects such as Django, Flask, scikit-learn, SymPy or pytest; previous task-grading experience.Location and engagementThis opening is for candidates working remotely in the United States. It is an independent-contractor engagement, not W-2 employment. The advertised rate is US$70-US$90 per hour. Work can be completed on your own schedule; project scope, duration and final terms are confirmed by the hiring provider. The provider currently states that H-1B and STEM OPT candidates cannot be supported.How to applyApply through LinkedIn. You may also email your CV and relevant open-source work to team@humanitapp.com with the subject \"SWE-Bench Task Auditor\". Please do not include confidential employer or client material.The external hiring provider makes screening and selection decisions. Applying does not guarantee selection or paid work. HumanitApp does not charge an application fee. Qualified applicants are considered without regard to legally protected characteristics; reasonable accommodations are available on request.","company":"Humanitapp","rawCompany":"humanitapp","city":"Denver","state":"CO","isRemote":false,"isActive":false,"createdAt":"2026-10-24T09:05:28.948Z","occupations":[{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1253.00","title":"Software Quality Assurance Analysts and Testers","slug":"software-quality-assurance-analysts-and-testers"},{"code":"15-1251.00","title":"Computer Programmers","slug":"computer-programmers"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541690","title":"Other Scientific and Technical Consulting Services","slug":"other-scientific-and-technical-consulting-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Software Engineering Task Auditor (SWE-Bench)","description":"Software Engineering Task Auditor (SWE-Bench)United States - Remote | Independent contractor | US$70-US$90/hourAbout the opportunityHumanitApp is coordinating applications for an external hiring provider. This is not employment with HumanitApp. The project helps an AI research lab assess whether software-engineering benchmark tasks are correct, reproducible and fairly graded.In practical terms, you will review a coding task, the proposed fix and the tests used to judge it. Your feedback should explain whether the fix solves the right problem, whether the tests are reliable and whether another engineer can reproduce the result.What you will do- Review repository-level coding tasks and reference patches for correctness.- Check test runners and Docker environments for reliable, reproducible execution.- Identify answer leakage, misleading tests and ways a solution could game the scoring.- Write concise, evidence-based feedback using the project's review rubric.What you bring- At least 3 years of professional software-engineering experience.- Real open-source contribution or maintainer experience, such as merged pull requests.- Python proficiency plus at least one of Java, Go, TypeScript or C++.- Experience reviewing patches, running tests and working with Docker isolation.- Clear technical writing and careful code-review judgment.Helpful, but not requiredExperience with SWE-Bench or similar repository benchmarks; contributions to Python projects such as Django, Flask, scikit-learn, SymPy or pytest; previous task-grading experience.Location and engagementThis opening is for candidates working remotely in the United States. It is an independent-contractor engagement, not W-2 employment. The advertised rate is US$70-US$90 per hour. Work can be completed on your own schedule; project scope, duration and final terms are confirmed by the hiring provider. The provider currently states that H-1B and STEM OPT candidates cannot be supported.How to applyApply through LinkedIn. You may also email your CV and relevant open-source work to team@humanitapp.com with the subject \"SWE-Bench Task Auditor\". Please do not include confidential employer or client material.The external hiring provider makes screening and selection decisions. Applying does not guarantee selection or paid work. HumanitApp does not charge an application fee. Qualified applicants are considered without regard to legally protected characteristics; reasonable accommodations are available on request.","datePosted":"2026-10-24T09:05:28.948Z","dateModified":"2026-10-24T09:05:28.948Z","hiringOrganization":{"@type":"Organization","name":"Humanitapp","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Denver","addressRegion":"CO","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"1e59055da63c0b9435bfb40a"},"url":"https://jobsearcher.com/jobs/1e59055da63c0b9435bfb40a"}}