{"schemaVersion":"jobsearcher.job.v1","id":"8e125f44ae60c8d5d0901a78","url":"https://jobsearcher.com/jobs/8e125f44ae60c8d5d0901a78","canonicalUrl":"https://jobsearcher.com/jobs/8e125f44ae60c8d5d0901a78","title":"Remote Senior Software Engineer AI Evaluation & Benchmarks (Python)","description":"Before Applying This role is open to contractors in accepted locations only. Please confirm your country is on the list before applying — we're unable to process applications from unlisted locations. List of accepted countries and locations.\r\nFor US applicants: This is a 1099 independent contractor role. It is not compatible with F-1 OPT, STEM OPT, or any visa status that requires W-2 employment, guaranteed hours, or employer sponsorship. We are unable to provide offer letters or employment verification for this role.\r\nWhat You'll Be Doing Design and build the coding benchmarks and evaluation pipelines used to test frontier AI models on real software engineering work:\r\nDesign coding benchmarks that evaluate frontier models on real-world programming tasks — reasoning, debugging, and production-quality code\r\nBuild and maintain scalable data pipelines for evaluation workflows\r\nAnalyze model-generated code for correctness, reliability, and edge-case failures\r\nConstruct structured evaluation scenarios across large repos and multi-language environments\r\nProvide detailed technical feedback on model performance and failure patterns\r\nContribute to evaluation frameworks that set the bar for how coding ability is measured\r\nEnd result: benchmarks that meaningfully separate what frontier models can and can't do — and shape how the next generation is trained and improved.\r\nAI coding evaluation in one line: Design task ? build harness ? run model ? analyze failures ? feed findings back into the benchmark ? evaluations that actually distinguish strong models from weak ones.\r\nWhat You'll Need 4+ years of professional software engineering experience (non-negotiable)\r\nExpert Python — clean, performant, well-tested code\r\nHands-on experience working in large, complex codebases\r\nProven experience designing and implementing LLM coding benchmarks and evaluation data pipelines\r\nStrong command of Git and modern development workflows\r\nTrack record at a high-growth tech company or top-tier software organization\r\nStrong written English communication\r\nIdentity verification: Applicants will be required to verify their identity and confirm they have valid documentation to work as an independent contractor in their country of residence.\r\nNice to have Senior or Lead-level profile with a history of technical ownership\r\nBachelor's or Master's in CS, ML, or related field (or equivalent professional experience)\r\nProficiency in additional languages: JavaScript, Go, C++, or others\r\nCI/CD experience and writing robust unit tests (pytest, Mocha, JUnit)\r\nBackground in security engineering or significant open-source contributions\r\nFamiliarity with AI/ML evaluation methodologies or model benchmarking\r\nLogistics Location: Fully remote — work from anywhere on the accepted locations list\r\nCompensation: $80–$100/hr based on location and seniority\r\nContract length: 3 months, with potential for extension\r\nHours: Full-time availability preferred — hours vary by project and are not guaranteed week to week\r\nEngagement: 1099 independent contractor\r\nPayment: Weekly via PayPal or Stripe\r\n?? Important: Hours are project-dependent and can vary week to week. We recommend keeping other work options open alongside this engagement rather than relying on it as your sole source of income.","company":"GrabJobs","rawCompany":"grabjobs","city":"St Louis","state":"MO","isRemote":true,"isActive":false,"createdAt":"2026-07-16T01:47:58.009Z","occupations":[{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1251.00","title":"Computer Programmers","slug":"computer-programmers"},{"code":"15-1254.00","title":"Web Developers","slug":"web-developers"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Remote Senior Software Engineer AI Evaluation & Benchmarks (Python)","description":"Before Applying This role is open to contractors in accepted locations only. Please confirm your country is on the list before applying — we're unable to process applications from unlisted locations. List of accepted countries and locations.\r\nFor US applicants: This is a 1099 independent contractor role. It is not compatible with F-1 OPT, STEM OPT, or any visa status that requires W-2 employment, guaranteed hours, or employer sponsorship. We are unable to provide offer letters or employment verification for this role.\r\nWhat You'll Be Doing Design and build the coding benchmarks and evaluation pipelines used to test frontier AI models on real software engineering work:\r\nDesign coding benchmarks that evaluate frontier models on real-world programming tasks — reasoning, debugging, and production-quality code\r\nBuild and maintain scalable data pipelines for evaluation workflows\r\nAnalyze model-generated code for correctness, reliability, and edge-case failures\r\nConstruct structured evaluation scenarios across large repos and multi-language environments\r\nProvide detailed technical feedback on model performance and failure patterns\r\nContribute to evaluation frameworks that set the bar for how coding ability is measured\r\nEnd result: benchmarks that meaningfully separate what frontier models can and can't do — and shape how the next generation is trained and improved.\r\nAI coding evaluation in one line: Design task ? build harness ? run model ? analyze failures ? feed findings back into the benchmark ? evaluations that actually distinguish strong models from weak ones.\r\nWhat You'll Need 4+ years of professional software engineering experience (non-negotiable)\r\nExpert Python — clean, performant, well-tested code\r\nHands-on experience working in large, complex codebases\r\nProven experience designing and implementing LLM coding benchmarks and evaluation data pipelines\r\nStrong command of Git and modern development workflows\r\nTrack record at a high-growth tech company or top-tier software organization\r\nStrong written English communication\r\nIdentity verification: Applicants will be required to verify their identity and confirm they have valid documentation to work as an independent contractor in their country of residence.\r\nNice to have Senior or Lead-level profile with a history of technical ownership\r\nBachelor's or Master's in CS, ML, or related field (or equivalent professional experience)\r\nProficiency in additional languages: JavaScript, Go, C++, or others\r\nCI/CD experience and writing robust unit tests (pytest, Mocha, JUnit)\r\nBackground in security engineering or significant open-source contributions\r\nFamiliarity with AI/ML evaluation methodologies or model benchmarking\r\nLogistics Location: Fully remote — work from anywhere on the accepted locations list\r\nCompensation: $80–$100/hr based on location and seniority\r\nContract length: 3 months, with potential for extension\r\nHours: Full-time availability preferred — hours vary by project and are not guaranteed week to week\r\nEngagement: 1099 independent contractor\r\nPayment: Weekly via PayPal or Stripe\r\n?? Important: Hours are project-dependent and can vary week to week. We recommend keeping other work options open alongside this engagement rather than relying on it as your sole source of income.","datePosted":"2026-07-16T01:47:58.009Z","dateModified":"2026-07-16T01:47:58.009Z","hiringOrganization":{"@type":"Organization","name":"GrabJobs","sameAs":"https://jobsearcher.com"},"jobLocationType":"TELECOMMUTE","applicantLocationRequirements":{"@type":"Country","name":"US"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"St Louis","addressRegion":"MO","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"8e125f44ae60c8d5d0901a78"},"url":"https://jobsearcher.com/jobs/8e125f44ae60c8d5d0901a78"}}