{"schemaVersion":"jobsearcher.job.v1","id":"e8bb4783c8d2efd6485d0b07","url":"https://jobsearcher.com/jobs/e8bb4783c8d2efd6485d0b07","canonicalUrl":"https://jobsearcher.com/jobs/e8bb4783c8d2efd6485d0b07","title":"Research Engineer - Data and Evaluation","description":"For our client, we are seeking a Research Engineer - Data and Evaluation to join the team of a leader in the Information Technology space.This role focuses on designing and scaling the data and evaluation systems that help researchers measure model performance in complex coding and agentic environments. You will build executable task setups, create robust scoring logic, and investigate failure modes to distinguish model limitations from environment issues. The work is highly collaborative and will partner closely with RL researchers and engineers to improve training data quality, validation methods, and downstream results.Location: RemoteCompensation: $150,000 – $350,000 annuallyResponsibilities Design and create challenging evaluations for coding and long-horizon agents. Develop executable environments with realistic task setups and clear success criteria. Create and test scoring logic and verifiers, addressing reward hacks and misleading progress measures. Trace model failures through outputs and code to differentiate between model limitations and environmental issues. Build and scale RL data pipelines that generate and curate complex coding and agentic tasks. Collaborate with RL researchers to assess the impact of task and training data modifications on performance.Qualifications Strong software engineering skills in Python, particularly in building data pipelines and evaluation systems. Hands-on experience with language models or agents, including understanding their training data and evaluation impacts. Ability to analyze complex data and model outputs to identify failure modes and create testable measurements. Proven track record of managing technical projects from experimentation to validation, collaborating with researchers and engineers. Experience building technical evaluations or agent environments, with skills in debugging model traces and scoring code.Benefits Flexible work arrangements, including remote or in-person options in San Francisco. Visa sponsorship and relocation assistance for international candidates. Quarterly team off-sites, hackathons, and opportunities for professional development. Equity incentives aligned with the company's growth and impact. Work alongside a talented, mission-driven team passionate about advancing AI technology.Our client is an equal opportunity employer. We encourage you to apply even if you don’t meet every qualification—your background could be exactly what this team needs.","company":"Ladders","rawCompany":"ladders","city":"Denver","state":"CO","isRemote":false,"isActive":false,"createdAt":"2026-11-17T11:28:47.765Z","occupations":[{"code":"15-1221.00","title":"Computer and Information Research Scientists","slug":"computer-and-information-research-scientists"},{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-2051.00","title":"Data Scientists","slug":"data-scientists"}],"industries":[{"code":"541690","title":"Other Scientific and Technical Consulting Services","slug":"other-scientific-and-technical-consulting-services"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Research Engineer - Data and Evaluation","description":"For our client, we are seeking a Research Engineer - Data and Evaluation to join the team of a leader in the Information Technology space.This role focuses on designing and scaling the data and evaluation systems that help researchers measure model performance in complex coding and agentic environments. You will build executable task setups, create robust scoring logic, and investigate failure modes to distinguish model limitations from environment issues. The work is highly collaborative and will partner closely with RL researchers and engineers to improve training data quality, validation methods, and downstream results.Location: RemoteCompensation: $150,000 – $350,000 annuallyResponsibilities Design and create challenging evaluations for coding and long-horizon agents. Develop executable environments with realistic task setups and clear success criteria. Create and test scoring logic and verifiers, addressing reward hacks and misleading progress measures. Trace model failures through outputs and code to differentiate between model limitations and environmental issues. Build and scale RL data pipelines that generate and curate complex coding and agentic tasks. Collaborate with RL researchers to assess the impact of task and training data modifications on performance.Qualifications Strong software engineering skills in Python, particularly in building data pipelines and evaluation systems. Hands-on experience with language models or agents, including understanding their training data and evaluation impacts. Ability to analyze complex data and model outputs to identify failure modes and create testable measurements. Proven track record of managing technical projects from experimentation to validation, collaborating with researchers and engineers. Experience building technical evaluations or agent environments, with skills in debugging model traces and scoring code.Benefits Flexible work arrangements, including remote or in-person options in San Francisco. Visa sponsorship and relocation assistance for international candidates. Quarterly team off-sites, hackathons, and opportunities for professional development. Equity incentives aligned with the company's growth and impact. Work alongside a talented, mission-driven team passionate about advancing AI technology.Our client is an equal opportunity employer. We encourage you to apply even if you don’t meet every qualification—your background could be exactly what this team needs.","datePosted":"2026-11-17T11:28:47.765Z","dateModified":"2026-11-17T11:28:47.765Z","hiringOrganization":{"@type":"Organization","name":"Ladders","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Denver","addressRegion":"CO","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"e8bb4783c8d2efd6485d0b07"},"url":"https://jobsearcher.com/jobs/e8bb4783c8d2efd6485d0b07"}}