Software Engineering Expert, AI Agent Evaluation
Put your software engineering expertise to work at the frontier of AI.Bring the expertise you know best and help train AI agents that do real technical work, on your own schedule, from anywhere. You'll turn problems from your own research or professional practice into hands-on challenges that an AI agent has to solve at the command line. You'll find exactly where it breaks and set the expert standard it's measured against. Workada connects skilled contributors with meaningful AI work, and your judgment is what makes that work reliable. We're building a community of software engineers, both PhDs and seasoned practitioners, to test how far today's AI agents can go on real problems in software engineering. We'd like you to be part of it.
What you'll do:
You'll take problems from your own work and turn each one into a self-contained task that an AI agent has to solve in a Linux terminal. For every task, you'll do four things:
Write a clear brief.
Set up the environment with the files and tools the agent needs.
Build a reference solution.
Write automated checks on the final result.
The strongest tasks are the ones today's best agents still get wrong. Your firsthand knowledge of where real software engineering work goes wrong is what makes them valuable. That might be a race condition that only shows under load, a migration that silently drops rows, a query planner that picks the wrong index, or a frontend build that breaks on a dependency bump.
Specialties we're looking for
If your background is in software engineering, there's likely a place for you. We're building depth across:
Algorithms: data structures, optimization, and algorithm design
Systems: operating systems, distributed systems, networking, and concurrency
Databases: query engines, storage, indexing, and transactions
Data engineering: pipelines, ETL, streaming, and orchestration
Frontend: web frameworks, browsers, performance, and accessibility
Languages: compilers, interpreters, type systems, and developer tooling
Who we're looking for Academics and practitioners are equally welcome. No AI or machine-learning background is needed. We're after your software engineering expertise and your comfort working in a terminal.
Requirements:
- You hold, or are completing, a PhD in computer science, software engineering, or a closely related field, or you have 5+ years of professional experience in software engineering.
- You've done technical or computational work in your field and are comfortable in a Linux terminal: shell commands, scripting in Python or Bash, and installing packages.
- You've used Docker, or can pick it up quickly. We share templates and worked examples.
- You can draw on your own research, projects, code, or files as source material for your tasks.
- You have a precise, adversarial eye and enjoy finding the exact point where a workflow breaks, then explaining it cleanly.
- You're honest about uncertainty and write instructions that leave no room for misreading.
- You're based in the US or Canada.
- Experience with production systems, or tools such as Git, CI pipelines, Postgres, Kafka, Spark, or React is a plus.
- Materials you can use
- Almost any part of your technical work can seed a task. For example:
- Scripts, code, and pipelines from your research or practice
- Repositories, build configurations, and failing test suites
- Schemas, migrations, and query plans
- Datasets and the processing behind them
- Bugs, edge cases, and failure modes you've had to debug yourself
What we offer:
- $100 to $150 per hour, based on your field and experience
- Weekly payouts through StripeRemote work from anywhere in the US or Canada
- A flexible schedule
Real people to work with: a 30-minute interview with someone in your field, and a Lead you can reach when questions come upWe're building this group now for projects starting in the coming weeks. If you're a match, we'll share the start date for your field as soon as we have it. We won't ask you to hold time before work is ready.
Ready to apply? Join a community of software engineers setting the standard for AI agents.
Workada is an Equal Opportunity Employer. We welcome applicants from all backgrounds.