AI Engineer - Data and Evals
Glean Technologies builds AI products and agents, and this role focuses on making them more reliable and useful through disciplined measurement. As an AI Engineer - Data and Evals, you will design and run evaluation systems that connect offline benchmarks to production behavior, then translate results into engineering improvements. This is a hybrid position in the San Francisco Bay Area, CA, with4 days per weekin the Mountain View or San Francisco office. The base salary range isUSD 200,000 - 280,000 per year, with compensation determined by factors such as location, level, and relevant experience.
What you’ll do:
Buildevalsets and benchmarksfrom expert-curated examples, production data, synthetic data, and past failures, ensuring coverage of edge cases and key user slices. Create and scale aneval platformandagent observability, including experiment pipelines, trace capture and replay, and dashboards that support reproducible, affordable runs. Developagentic LLM verifiersfor correctness, groundedness, citation quality, and task success, then calibrate them againsthuman labelsusing clear rubrics. Run evaluations across the lifecycle, fromCI and pre-releasesuites to sampled production traffic, and communicate outcomes withconfidence intervals .Buildenterprise simulation environmentsusing simulated users and realistic company data to test multi-step agent workflows. Diagnose failures byslice and trace, then convert eval results into concrete fixes while balancingqualityagainstlatencyandcost .What you bring5+ yearsof software engineering experience with strong Go and/or Pythonand backend fundamentals, including APIs, data modeling, testing, and distributed pipelines.
Type:
Scriptis a plus. Experience creatingdatasets and benchmarksfor LLM use cases such asRAGor agents, including evalset curation andsynthetic data generation .Familiarity with eval methodologies and tradeoffs, including deterministic metrics, LLM-as-a-judge, pointwise vs pairwise grading, human labeling, and judge calibration. Experience building eval or experiment infrastructure that runs reliably at scale, includingCI/CD integrationsand observability tooling such asOpenTelemetry .Strong analytical skills: ability to work through traces and failure cases, writeSQL, and use data and metrics to explain quality movement.
Nice to have:
experience with simulation environments and/or model post-training techniques. Tools you may use Go , Python , TypeScript , SQLOpenTelemetry , CI/CDCompensation and benefits Base salary range:$200,000 - $280,000annually Medical, Vision, and Dental coverage Generous time-off policy 401(k) contribution opportunity Home office improvement stipend Annual education and wellness stipends Healthy lunches daily Certain roles may be eligible for variable compensation, equity, and benefitsAI-first mindset at Glean
During the interview process, you will complete a briefAI-focused exercise or discussionto help Glean understand how you think about, design, and apply AI to drive impact in your role.