AI DevOps / Observability Engineer
We are seeking an experienced AI DevOps/Observability Engineer to support production operations for Generative AI and LLM agent applications. The role focuses on reliability, observability, automated evaluation, performance optimization, and production readiness.
Key Responsibilities:
Implement LLM tracing, logging, telemetry, and observability for AI/agent workflows.
Build production dashboards for infrastructure, application, and AI performance metrics.
Define SLIs/SLOs, actionable alerts, runbooks, and production-readiness standards.
Develop automated LLM evaluation/testing pipelines for quality, safety, hallucination, faithfulness, and relevance.
Analyze prompts, token usage, latency, model performance, cost, and accuracy.
Support CI/CD, deployments, incident management, and production reliability.
Required Skills:
LLM/Agent Evaluation: Ragas, DeepEval, TruLens or similar.
LLM Observability/Tracing: LangSmith, Langfuse, Phoenix, Arize, Open
Telemetry.
Monitoring: Datadog, Prometheus/Grafana, New Relic, or cloud-native monitoring.
SRE/Operations: SLI/SLO definition, alerting, incident management, runbooks.
Test Automation: Automated testing integrated with CI/CD pipelines.
AI/Model Analysis: Prompt benchmarking, regression testing, latency and cost optimization.
Python: Advanced Python, async programming, APIs, LangChain/LlamaIndex.
Cloud & DevOps: AWS/GCP/Azure, Docker, Kubernetes, CI/CD.
Preferred Qualifications:
3+ years operationalizing LLM/Generative AI applications in production.
Experience with Vector Databases such as Pinecone, Milvus, or Chroma.
Experience monitoring and optimizing RAG pipelines.
Background in SRE, MLOps, or AI production operations.