{"schemaVersion":"jobsearcher.job.v1","id":"fa8dfaef5d1b9da30971eb6b","url":"https://jobsearcher.com/jobs/fa8dfaef5d1b9da30971eb6b","canonicalUrl":"https://jobsearcher.com/jobs/fa8dfaef5d1b9da30971eb6b","title":"Software Engineering Expert, AI Agent Evaluation","description":"Put your software engineering expertise to work at the frontier of AI.Bring the expertise you know best and help train AI agents that do real technical work, on your own schedule, from anywhere. You'll turn problems from your own research or professional practice into hands-on challenges that an AI agent has to solve at the command line. You'll find exactly where it breaks and set the expert standard it's measured against. Workada connects skilled contributors with meaningful AI work, and your judgment is what makes that work reliable. We're building a community of software engineers, both PhDs and seasoned practitioners, to test how far today's AI agents can go on real problems in software engineering. We'd like you to be part of it.What you'll doYou'll take problems from your own work and turn each one into a self-contained task that an AI agent has to solve in a Linux terminal. For every task, you'll do four things:Write a clear brief.Set up the environment with the files and tools the agent needs.Build a reference solution.Write automated checks on the final result.The strongest tasks are the ones today's best agents still get wrong. Your firsthand knowledge of where real software engineering work goes wrong is what makes them valuable. That might be a race condition that only shows under load, a migration that silently drops rows, a query planner that picks the wrong index, or a frontend build that breaks on a dependency bump.Specialties we're looking forIf your background is in software engineering, there's likely a place for you. We're building depth across:Algorithms: data structures, optimization, and algorithm designSystems: operating systems, distributed systems, networking, and concurrencyDatabases: query engines, storage, indexing, and transactionsData engineering: pipelines, ETL, streaming, and orchestrationFrontend: web frameworks, browsers, performance, and accessibilityLanguages: compilers, interpreters, type systems, and developer toolingWho we're looking forAcademics and practitioners are equally welcome. No AI or machine-learning background is needed. We're after your software engineering expertise and your comfort working in a terminal.RequirementsYou hold, or are completing, a PhD in computer science, software engineering, or a closely related field, or you have 5+ years of professional experience in software engineering.You've done technical or computational work in your field and are comfortable in a Linux terminal: shell commands, scripting in Python or Bash, and installing packages.You've used Docker, or can pick it up quickly. We share templates and worked examples.You can draw on your own research, projects, code, or files as source material for your tasks.You have a precise, adversarial eye and enjoy finding the exact point where a workflow breaks, then explaining it cleanly.You're honest about uncertainty and write instructions that leave no room for misreading.You're based in the US or Canada.Experience with production systems, or tools such as Git, CI pipelines, Postgres, Kafka, Spark, or React is a plus.Materials you can useAlmost any part of your technical work can seed a task. For example:Scripts, code, and pipelines from your research or practiceRepositories, build configurations, and failing test suitesSchemas, migrations, and query plansDatasets and the processing behind themBugs, edge cases, and failure modes you've had to debug yourselfWhat we offer$100 to $150 per hour, based on your field and experienceWeekly payouts through StripeRemote work from anywhere in the US or CanadaA flexible scheduleReal people to work with: a 30-minute interview with someone in your field, and a Lead you can reach when questions come upWe're building this group now for projects starting in the coming weeks. If you're a match, we'll share the start date for your field as soon as we have it. We won't ask you to hold time before work is ready.Ready to apply? Join a community of software engineers setting the standard for AI agents.Workada is an Equal Opportunity Employer. We welcome applicants from all backgrounds.","company":"Workada","rawCompany":"workada","city":"Denver","state":"CO","isRemote":false,"isActive":false,"createdAt":"2026-11-06T07:16:59.116Z","occupations":[{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"},{"code":"15-1251.00","title":"Computer Programmers","slug":"computer-programmers"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Software Engineering Expert, AI Agent Evaluation","description":"Put your software engineering expertise to work at the frontier of AI.Bring the expertise you know best and help train AI agents that do real technical work, on your own schedule, from anywhere. You'll turn problems from your own research or professional practice into hands-on challenges that an AI agent has to solve at the command line. You'll find exactly where it breaks and set the expert standard it's measured against. Workada connects skilled contributors with meaningful AI work, and your judgment is what makes that work reliable. We're building a community of software engineers, both PhDs and seasoned practitioners, to test how far today's AI agents can go on real problems in software engineering. We'd like you to be part of it.What you'll doYou'll take problems from your own work and turn each one into a self-contained task that an AI agent has to solve in a Linux terminal. For every task, you'll do four things:Write a clear brief.Set up the environment with the files and tools the agent needs.Build a reference solution.Write automated checks on the final result.The strongest tasks are the ones today's best agents still get wrong. Your firsthand knowledge of where real software engineering work goes wrong is what makes them valuable. That might be a race condition that only shows under load, a migration that silently drops rows, a query planner that picks the wrong index, or a frontend build that breaks on a dependency bump.Specialties we're looking forIf your background is in software engineering, there's likely a place for you. We're building depth across:Algorithms: data structures, optimization, and algorithm designSystems: operating systems, distributed systems, networking, and concurrencyDatabases: query engines, storage, indexing, and transactionsData engineering: pipelines, ETL, streaming, and orchestrationFrontend: web frameworks, browsers, performance, and accessibilityLanguages: compilers, interpreters, type systems, and developer toolingWho we're looking forAcademics and practitioners are equally welcome. No AI or machine-learning background is needed. We're after your software engineering expertise and your comfort working in a terminal.RequirementsYou hold, or are completing, a PhD in computer science, software engineering, or a closely related field, or you have 5+ years of professional experience in software engineering.You've done technical or computational work in your field and are comfortable in a Linux terminal: shell commands, scripting in Python or Bash, and installing packages.You've used Docker, or can pick it up quickly. We share templates and worked examples.You can draw on your own research, projects, code, or files as source material for your tasks.You have a precise, adversarial eye and enjoy finding the exact point where a workflow breaks, then explaining it cleanly.You're honest about uncertainty and write instructions that leave no room for misreading.You're based in the US or Canada.Experience with production systems, or tools such as Git, CI pipelines, Postgres, Kafka, Spark, or React is a plus.Materials you can useAlmost any part of your technical work can seed a task. For example:Scripts, code, and pipelines from your research or practiceRepositories, build configurations, and failing test suitesSchemas, migrations, and query plansDatasets and the processing behind themBugs, edge cases, and failure modes you've had to debug yourselfWhat we offer$100 to $150 per hour, based on your field and experienceWeekly payouts through StripeRemote work from anywhere in the US or CanadaA flexible scheduleReal people to work with: a 30-minute interview with someone in your field, and a Lead you can reach when questions come upWe're building this group now for projects starting in the coming weeks. If you're a match, we'll share the start date for your field as soon as we have it. We won't ask you to hold time before work is ready.Ready to apply? Join a community of software engineers setting the standard for AI agents.Workada is an Equal Opportunity Employer. We welcome applicants from all backgrounds.","datePosted":"2026-11-06T07:16:59.116Z","dateModified":"2026-11-06T07:16:59.116Z","hiringOrganization":{"@type":"Organization","name":"Workada","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Denver","addressRegion":"CO","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"fa8dfaef5d1b9da30971eb6b"},"url":"https://jobsearcher.com/jobs/fa8dfaef5d1b9da30971eb6b"}}