{"schemaVersion":"jobsearcher.job.v1","id":"2daee813ef6bcbefbb2264a2","url":"https://jobsearcher.com/jobs/2daee813ef6bcbefbb2264a2","canonicalUrl":"https://jobsearcher.com/jobs/2daee813ef6bcbefbb2264a2","title":"AI Engineer - Data and Evals","description":"Glean Technologies builds AI products and agents, and this role focuses on making them more reliable and useful through disciplined measurement. As an AI Engineer - Data and Evals, you will design and run evaluation systems that connect offline benchmarks to production behavior, then translate results into engineering improvements.This is a hybrid position in theSan Francisco Bay Area, CA , with4 days per weekin the Mountain View or San Francisco office. The base salary range isUSD 200,000 - 280,000 per year , with compensation determined by factors such as location, level, and relevant experience.What you’ll doBuildevalsets and benchmarksfrom expert-curated examples, production data, synthetic data, and past failures, ensuring coverage of edge cases and key user slices.Create and scale aneval platformandagent observability , including experiment pipelines, trace capture and replay, and dashboards that support reproducible, affordable runs.Developagentic LLM verifiersfor correctness, groundedness, citation quality, and task success, then calibrate them againsthuman labelsusing clear rubrics.Run evaluations across the lifecycle, fromCI and pre-releasesuites to sampled production traffic, and communicate outcomes withconfidence intervals .Buildenterprise simulation environmentsusing simulated users and realistic company data to test multi-step agent workflows.Diagnose failures byslice and trace , then convert eval results into concrete fixes while balancingqualityagainstlatencyandcost .What you bring5+ yearsof software engineering experience with strongGo and/or Pythonand backend fundamentals, including APIs, data modeling, testing, and distributed pipelines.TypeScriptis a plus.Experience creatingdatasets and benchmarksfor LLM use cases such asRAGor agents, including evalset curation andsynthetic data generation .Familiarity with eval methodologies and tradeoffs, including deterministic metrics,LLM-as-a-judge , pointwise vs pairwise grading, human labeling, and judge calibration.Experience building eval or experiment infrastructure that runs reliably at scale, includingCI/CD integrationsand observability tooling such asOpenTelemetry .Strong analytical skills: ability to work through traces and failure cases, writeSQL , and use data and metrics to explain quality movement.Nice to have:experience with simulation environments and/or model post-training techniques.Tools you may useGo ,Python ,TypeScript ,SQLOpenTelemetry ,CI/CDCompensation and benefitsBase salary range:$200,000 - $280,000annuallyMedical, Vision, and Dental coverageGenerous time-off policy401(k) contribution opportunityHome office improvement stipendAnnual education and wellness stipendsHealthy lunches dailyCertain roles may be eligible for variable compensation, equity, and benefitsAI-first mindset at Glean\r\nDuring the interview process, you will complete a briefAI-focused exercise or discussionto help Glean understand how you think about, design, and apply AI to drive impact in your role.#J-18808-Ljbffr","company":"Glean Technologies","rawCompany":"glean technologies","city":"Millbrae","state":"CA","isRemote":false,"isActive":true,"createdAt":"2026-11-04T02:23:19.111Z","occupations":[{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"},{"code":"15-2051.00","title":"Data Scientists","slug":"data-scientists"}],"industries":[{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"AI Engineer - Data and Evals","description":"Glean Technologies builds AI products and agents, and this role focuses on making them more reliable and useful through disciplined measurement. As an AI Engineer - Data and Evals, you will design and run evaluation systems that connect offline benchmarks to production behavior, then translate results into engineering improvements.This is a hybrid position in theSan Francisco Bay Area, CA , with4 days per weekin the Mountain View or San Francisco office. The base salary range isUSD 200,000 - 280,000 per year , with compensation determined by factors such as location, level, and relevant experience.What you’ll doBuildevalsets and benchmarksfrom expert-curated examples, production data, synthetic data, and past failures, ensuring coverage of edge cases and key user slices.Create and scale aneval platformandagent observability , including experiment pipelines, trace capture and replay, and dashboards that support reproducible, affordable runs.Developagentic LLM verifiersfor correctness, groundedness, citation quality, and task success, then calibrate them againsthuman labelsusing clear rubrics.Run evaluations across the lifecycle, fromCI and pre-releasesuites to sampled production traffic, and communicate outcomes withconfidence intervals .Buildenterprise simulation environmentsusing simulated users and realistic company data to test multi-step agent workflows.Diagnose failures byslice and trace , then convert eval results into concrete fixes while balancingqualityagainstlatencyandcost .What you bring5+ yearsof software engineering experience with strongGo and/or Pythonand backend fundamentals, including APIs, data modeling, testing, and distributed pipelines.TypeScriptis a plus.Experience creatingdatasets and benchmarksfor LLM use cases such asRAGor agents, including evalset curation andsynthetic data generation .Familiarity with eval methodologies and tradeoffs, including deterministic metrics,LLM-as-a-judge , pointwise vs pairwise grading, human labeling, and judge calibration.Experience building eval or experiment infrastructure that runs reliably at scale, includingCI/CD integrationsand observability tooling such asOpenTelemetry .Strong analytical skills: ability to work through traces and failure cases, writeSQL , and use data and metrics to explain quality movement.Nice to have:experience with simulation environments and/or model post-training techniques.Tools you may useGo ,Python ,TypeScript ,SQLOpenTelemetry ,CI/CDCompensation and benefitsBase salary range:$200,000 - $280,000annuallyMedical, Vision, and Dental coverageGenerous time-off policy401(k) contribution opportunityHome office improvement stipendAnnual education and wellness stipendsHealthy lunches dailyCertain roles may be eligible for variable compensation, equity, and benefitsAI-first mindset at Glean\r\nDuring the interview process, you will complete a briefAI-focused exercise or discussionto help Glean understand how you think about, design, and apply AI to drive impact in your role.#J-18808-Ljbffr","datePosted":"2026-11-04T02:23:19.111Z","dateModified":"2026-11-04T02:23:19.111Z","hiringOrganization":{"@type":"Organization","name":"Glean Technologies","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Millbrae","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"2daee813ef6bcbefbb2264a2"},"url":"https://jobsearcher.com/jobs/2daee813ef6bcbefbb2264a2"}}