JOBSEARCHER

AI DevOps / Observability Engineer

We are seeking an experienced AI DevOps/Observability Engineer to support production operations for Generative AI and LLM agent applications. The role focuses on reliability, observability, automated evaluation, performance optimization, and production readiness.
Key Responsibilities: Implement LLM tracing, logging, telemetry, and observability for AI/agent workflows. Build production dashboards for infrastructure, application, and AI performance metrics. Define SLIs/SLOs, actionable alerts, runbooks, and production-readiness standards. Develop automated LLM evaluation/testing pipelines for quality, safety, hallucination, faithfulness, and relevance. Analyze prompts, token usage, latency, model performance, cost, and accuracy. Support CI/CD, deployments, incident management, and production reliability.
Required Skills: LLM/Agent Evaluation: Ragas, DeepEval, TruLens or similar. LLM Observability/Tracing: LangSmith, Langfuse, Phoenix, Arize, Open Telemetry. Monitoring: Datadog, Prometheus/Grafana, New Relic, or cloud-native monitoring. SRE/Operations: SLI/SLO definition, alerting, incident management, runbooks. Test Automation: Automated testing integrated with CI/CD pipelines. AI/Model Analysis: Prompt benchmarking, regression testing, latency and cost optimization. Python: Advanced Python, async programming, APIs, LangChain/LlamaIndex. Cloud & DevOps: AWS/GCP/Azure, Docker, Kubernetes, CI/CD.
Preferred Qualifications: 3+ years operationalizing LLM/Generative AI applications in production. Experience with Vector Databases such as Pinecone, Milvus, or Chroma. Experience monitoring and optimizing RAG pipelines. Background in SRE, MLOps, or AI production operations.