JOBSEARCHER

Staff Software Engineer, Agent Eval Platform

Overview: In this role you design and build the evaluation and observability stack for enterprise AI agents. You will shape how agent performance is measured across multi-step, stateful interactions and use that signal to train and improve behavior. You’ll work within a cross-functional engineering team at Moveworks, part of the ServiceNow ecosystem, to enable scalable, reliable eval harnesses and realistic simulations. This is an opportunity to tackle novel problems at scale, turning complex agent behavior into actionable telemetry and improvements.
Responsibilities: Architect and own end-to-end evaluation orchestration for multi-turn agent scenarios, including setup, transcripts collection, validation, scoring, and run managementLead agent observability and tracing initiatives using OpenTelemetry to create a queryable trajectory model and robust trace propagationDevelop and maintain a stateful simulation environment (fake enterprise systems, data injection, sandbox isolation) to ensure hermetic, repeatable runsConsolidate disparate eval workflows into a single orchestration service with a clear source of truth and reliable SLOsDefine and implement reliability standards for the evaluation harness and enable self-serve usage by multiple teamsCollaborate with teams to translate eval signals into improvements and safer release gates
Key requirements: 8+ years building production backend or infrastructure systemsStrong in Python or Go (ideally both) Experience designing and operating systems that handle real traffic at scaleComfort with ambiguity; interest in turning fuzzy agent behavior into measurable signalsExperience with distributed systems, orchestration/runtimes, observability tooling, and data-intensive pipelinesKnowledge of gRPC/protobuf service designcomfort with ambiguityPython asyncioGo concurrencyOpenTelemetryspan data modeltrace context propagation