Staff Observability Platform Engineer

🚀 Staff Observability Platform Engineer | AI/GPU Infrastructure | Remote/Hybrid📍 Hybrid — Seattle, Houston, or New YorkHave you personally operated an observability backend at real production scale — not just consumed dashboards built by another team? We're looking for someone who can quantify that scale and explain the engineering decisions behind it.What you'll do: 🔹 Design, build, and operate large-scale metrics, logging, and tracing platforms 🔹 Own observability backend architecture in distributed Kubernetes and AI/GPU environments 🔹 Operate and scale Mimir, Thanos, VictoriaMetrics, Cortex, Loki, or Elasticsearch at production scale 🔹 Design Prometheus-based architectures (remote write, high availability, retention, global querying) 🔹 Build OpenTelemetry Collector pipelines (receivers, processors, exporters, sampling) 🔹 Identify and remediate high-cardinality metrics 🔹 Establish observability standards across engineering teamsWhat we're looking for: ✅ Personal ownership (not just consumption) of a metrics/logs backend at meaningful scale ✅ Ability to quantify that scale: active series, ingestion rate, TB/day, retention, cluster size ✅ Real experience making cardinality, retention, ingestion, and storage-cost trade-offs ✅ Strong hands-on production Kubernetes experience (multi-cluster, networking, autoscaling) ✅ Strong production code-reading/review ability (Go and/or Python) ✅ Understanding of retries, timeouts, backpressure, circuit breaking, and failure handlingNice to have: 🌟 Large-scale GPU fleets, NVIDIA DCGM, InfiniBand/RoCE/NVLink, Slurm/HPC 🌟 Custom Prometheus exporters, custom OpenTelemetry components 🌟 AI/ML infrastructure or distributed training platforms📍 Hybrid — Seattle, Houston, or New York