Staff Observability Platform Engineer
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
🚀 Staff Observability Platform Engineer | AI/GPU Infrastructure | Remote/Hybrid📍 Hybrid — Seattle, Houston, or New YorkHave you personally operated an observability backend at real production scale — not just consumed dashboards built by another team? We're looking for someone who can quantify that scale and explain the engineering decisions behind it.
What you'll do:
🔹 Design, build, and operate large-scale metrics, logging, and tracing platforms 🔹 Own observability backend architecture in distributed Kubernetes and AI/GPU environments 🔹 Operate and scale Mimir, Thanos, Victoria Metrics, Cortex, Loki, or Elasticsearch at production scale 🔹 Design Prometheus-based architectures (remote write, high availability, retention, global querying) 🔹 Build OpenTelemetry Collector pipelines (receivers, processors, exporters, sampling) 🔹 Identify and remediate high-cardinality metrics 🔹 Establish observability standards across engineering teams
What we're looking for:
✅ Personal ownership (not just consumption) of a metrics/logs backend at meaningful scale ✅ Ability to quantify that scale: active series, ingestion rate, TB/day, retention, cluster size ✅ Real experience making cardinality, retention, ingestion, and storage-cost trade-offs ✅ Strong hands-on production Kubernetes experience (multi-cluster, networking, autoscaling) ✅ Strong production code-reading/review ability (Go and/or Python) ✅ Understanding of retries, timeouts, backpressure, circuit breaking, and failure handling
Nice to have:
🌟 Large-scale GPU fleets, NVIDIA DCGM, InfiniBand/RoCE/NVLink, Slurm/HPC 🌟 Custom Prometheus exporters, custom OpenTelemetry components 🌟 AI/ML infrastructure or distributed training platforms📍 Hybrid — Seattle, Houston, or New York