Site Reliability Engineer, AI Infrastructure (Remote) - 59419267332

Location: Remote (LATAM or South Africa)Type: Full-time, PSTReports to: Head of Infrastructure / SREAbout The CompanyWe operate state-of-the-art AI Factories across Europe and the US, running large-scale NVIDIA GPU clusters (H100, H200, B200, B300) on bare metal for frontier AI workloads. We design, build, and operate the full stack: datacenter power and cooling, InfiniBand fabrics, SLURM and Kubernetes orchestration, storage, and the control plane that turns raw iron into reliable compute for our customers.Role OverviewWe are hiring a Senior Site Reliability Engineer (SRE) to own the reliability of our GPU training and inference clusters from the US West Coast. You will serve as the on-call anchor for US hours, drive incident response on multi-thousand GPU fabrics, and push our platform toward higher availability, faster recovery, and cleaner operations. This is a hands-on role with significant production impact from week one.Key ResponsibilitiesI. Cluster Operations & HardeningProduction SLURM Management: Operate and harden production SLURM clusters running large-scale distributed training and inference jobs.Hardware Health: Own the health of NVIDIA HGX and DGX nodes, including GPU, NVLink, NVSwitch, and BMC diagnostics.Fabric Tuning: Debug and tune NVIDIA Quantum InfiniBand fabrics (NDR and HDR), including Subnet Manager, topology, adaptive routing, SHARP, and congestion issues.Root Cause Analysis: Drive deep-dive RCA on GPU failures, XID errors, ECC events, thermal throttling, and link flaps.II. Automation & ObservabilitySystems Automation: Write robust automation in Python, Go, or Bash to replace manual tasks, improve MTTR, and scale operations efficiently.Observability Stack: Build and maintain observability for GPU fleets using Prometheus, Grafana, DCGM, node exporter, and custom exporters.Capacity & Rollouts: Contribute to capacity planning, firmware rollout strategy, and cluster bring-up for new sites.III. Collaboration & Incident ResponseWorkload Optimization: Partner with customer workload teams on NCCL tuning, job scheduling policy, QoS, and fairshare.Operational Excellence: Lead post-mortems, write comprehensive runbooks, and improve change management processes across global regions.On-Call Leadership: Participate in the on-call rotation for US hours and handle escalations from international sites when necessary.Who You AreRequired Experience & SkillsDeep SRE/HPC Background: 5+ years in SRE, systems engineering, or HPC operations.SLURM Expertise: Extensive production experience with SLURM at scale (accounting/slurmdbd, prolog/epilog scripts, cgroups, GRES, topology awareness).NVIDIA Stack Mastery: Hands-on experience with NVIDIA datacenter GPUs, driver stacks, CUDA runtime, Fabric Manager, nvidia-smi, DCGM, and GPU Direct RDMA.Networking: Operational experience with InfiniBand fabrics at 100G or higher (OpenSM/UFM, ibdiagnet, perfquery, and fabric troubleshooting).Linux Administration: Expert-level Linux admin skills (Ubuntu/RHEL family), including kernel tuning, systemd, networking, and PXE provisioning.Automation: Solid scripting skills in Python and Bash, plus working knowledge of Ansible or Terraform.Nice to HaveDistributed Training: Experience with NCCL internals, PyTorch distributed, or Megatron-style training stacks.Cluster Management: Familiarity with BCM (Base Command Manager), Run:ai, or similar managers.Kubernetes: Experience running Kubernetes on bare metal with GPU, Network, and MPI Operators.Storage: Exposure to high-performance storage like Lustre, WEKA, VAST, or BeeGFS.Infrastructure Context: Prior work in an AI cloud, neocloud, HPC center, or hyperscaler environment.Why Join Us?Frontier Infrastructure: You will touch clusters that train world-class models, working with the most advanced hardware available.Engineering Culture: We maintain a flat structure with direct access to leadership and a culture built around technical craftsmanship and ownership.Remote-First: Full remote flexibility with occasional travel for team summits and datacenter site visits.Competitive Package: Comprehensive US benefits including performance bonuses, equity participation, and 401(k) eligibility.