Software Engineer - Cloud Infrastructure & Distributed Systems
About Us:
GMI Cloud is the AI-native inference cloud, built for production AI. Backed by Headline VC and one of only seven cloud providers worldwide designated an NVIDIA Reference Platform Cloud Partner, we run NVIDIA H100, H200, B300, GB300, Vera Rubin GPUs in our own data centers, with GPU regions across North America, Europe, and Asia-Pacific. We own the full stack end to end: inference APIs and Model-as-a-Service on top, a Kubernetes-based orchestration layer with autoscaling, load balancing, and multi-region deployment in the middle, and dedicated and on-demand GPU compute on owned hardware underneath.
About This Role:
We are hiring a Software Engineer, Cloud Infrastructure & Distributed Systems to build the infrastructure that runs GMI’s global inference engine and Model-as-a-Service (MaaS) platform. You will own the systems layer underneath every AI request we serve: the Kubernetes platform, GPU fleet orchestration, traffic routing, observability, and the high-concurrency services that tie them together. This role is for engineers who think in systems, not components. You will design and operate cloud-native, multi-region platforms that handle bursty, latency-sensitive AI workloads at production scale, and you will make them reliable, efficient, and observable. We are not just running clusters. We are building the operating system for AI inference at scale.
Key Responsibilities:
Design, build, and operate the Kubernetes-based platform that runs GMI’s dedicated inference and MaaS across multiple GPU clusters in the U.S. and Asia.
Architect high-concurrency, low-latency distributed services — API gateways, request routing, load balancing, rate limiting, and queueing — that stay correct and degrade gracefully under extreme load.
Build multi-cluster and multi-region traffic management, including failover, capacity-aware scheduling, and global load distribution across heterogeneous GPU fleets.
Own GPU resource orchestration on Kubernetes: scheduling, autoscaling, bin-packing, node lifecycle, and health management to maximize utilization and minimize cost per token.
Develop operators, controllers, and automation (CRDs, GitOps/CD pipelines, infrastructure-as-code) that make deployments repeatable, safe, and fast.
Build the observability foundation — metrics, tracing, logging, and alerting — and drive SLOs, incident response, and post-incident improvements.
Drive system-level performance work across concurrency control, the request lifecycle, networking, storage, and compute coordination.
Set engineering standards for reliability, correctness, and long-term maintainability in a fast-moving AI ecosystem.
Required Qualifications:
5+ years of backend or infrastructure engineering, with deep expertise in distributed system design.
Proven experience building and operating high-concurrency, large-scale distributed systems in production.
Strong, hands-on Kubernetes experience in production: workloads, networking, storage, scheduling, autoscaling, and debugging cluster-level issues. Experience writing operators or controllers is a strong plus.
Solid cloud infrastructure fundamentals: Linux, networking (L4/L7 load balancing, DNS, service mesh), containers, and at least one major cloud (GCP, AWS, Azure) or bare-metal data center environment.
Strong understanding of system design tradeoffs across scalability, consistency, availability, and performance.
Proficiency in Go and/or Python; familiarity with C++ or Rust is a plus.
Familiarity with distributed system patterns such as load balancing, rate limiting, queueing, caching, and multi-region deployment.
Experience with infrastructure-as-code and CI/CD (Terraform, Helm, ArgoCD, or similar).Strong debugging, profiling, and performance optimization skills, and clear communication about complex systems end to end.
Preferred Qualifications:
Experience with LLM inference systems or serving frameworks (vLLM, SGLang, TensorRT-LLM, Triton, or similar), including deploying and scaling them on Kubernetes. Experience building or operating a Model-as-a-Service platform, inference API, or other developer-facing AI API at scale (e.g., OpenAI-compatible endpoints, usage metering, multi-tenant isolation).Experience serving multimodal workloads — image, video, audio, or vision-language models — and their distinct latency, memory, and throughput profiles. Familiarity with GPU infrastructure: NVIDIA GPU Operator, device plugins, MIG, InfiniBand/RDMA, NCCL, or large-scale GPU cluster operations. Knowledge of inference-specific optimizations such as KV-cache-aware routing, prefill/decode disaggregation, continuous batching, or speculative decoding. Experience with observability stacks (Prometheus, Grafana, OpenTelemetry) and real-time data or streaming systems (Kafka or similar).
Why Join:
GMI CloudWork on real production AI infrastructure: our own GPU clusters, real customers, real traffic — not a sandbox.
High ownership and direct impact: a small, senior team where your designs ship and define how the platform evolves.
Close partnership with NVIDIA and early access to the latest GPU platforms and inference technologies.
A seat at the frontier of AI infrastructure, shaping how models are served globally.