Staff Software Engineer - ML Observability

Overview As a Staff Engineer on Datadog’s LLM Observability team, you will shape the direction and build foundational features for monitoring, explaining, and improving AI systems in production. You’ll work with cross-functional partners to prototype scalable observability tools for LLMs and GenAI, influencing architecture and mentoring engineers. You’ll stay close to customer pain points to guide product priorities and drive innovation in AI observability. Compensation / Benefitscompetitive global benefitscontinuous professional developmenthealthcaredentalparental planningmental health benefits`,`401(k) with match`,`paid time off`,`fitness reimbursements`,`employee stock purchase plan ResponsibilitiesDesign and implement LLM observability featuresPrototype and scale new product features for generative AI systemsCollaborate cross-functionally with engineering, product, UX, and applied science teamsDevelop and extend tools for tracing, evaluating, and debugging LLMsInfluence architecture decisions and mentor engineers in building resilient, high-performance systemsAlign work with customer pain points to guide prioritiesStay current with ML and observability trends to drive innovation Key requirementsBS/MS/PhD in Computer Science, Engineering or related field or equivalent experienceDeep understanding of distributed systems and scalable backend architecturesHands-on experience building and shipping LLM-powered or GenAI applicationsKnowledge of model internals, inference pipelines, evaluation techniques, and prompt engineeringProduct-minded with ability to thrive in ambiguous, fast-changing spacesStrong communication, rigor in code, and commitment to maintainable softwareExperience with observability tools/platformscross-functional collaborationambiguous-space adaptabilitystrong communicationLLM/GenAI applicationsdistributed systemsinference pipelines