JOBSEARCHER

Senior HPC Engineer - Fleet Engineering

Lambda LabsSan Jose, CAL6 LeadSeptember 22nd, 2026
Overview: In this role you will build and operate the monitoring and automation foundation for Lambda’s AI compute clusters. You’ll manage large-scale HPC deployments, drive reliability across InfiniBand networks and GPU ecosystems, and codify operations to scale with demand. You’ll collaborate with on-site teams and cross-functional engineers to reduce incident impact and improve stability. This is a hands-on role at scale, shaping how Lambda delivers ubiquitous compute for researchers and enterprises. Compensation / Benefitsgenerous cash & equity compensationhealth, dental, and vision coveragewellness and commuter stipends 401k with 2% company matchflexible paid time off
Responsibilities: Build and operate monitoring and alerting for cluster health (fabric, GPU, power/thermal, job signals) to detect and respond proactivelyRemotely deploy and configure large-scale HPC clusters for AI workloads using automationAutomate cluster lifecycle (OS, firmware, drivers, networking) as code (Ansible, Terraform) Create runbooks and automated remediations for common cluster failure modes for safe execution by Support teamsTroubleshoot and resolve cluster issues across InfiniBand/RoCE, NCCL, GPU-direct, fabric, switching, and power; collaborate with on-site deployment teamsParticipate in on-call rotations and lead incident response for cluster-level problemsContribute to and maintain Standard Operating Procedures and feed requirements back to other engineering teams on simplification and stability
Key requirements: 7+ years in Site Reliability Engineering, HPC Engineering, DevOps, or similarStrong understanding of modern AI infrastructure, including GPU architectures and hardware performance optimizationStrong Linux experience in distributed environmentsExperience configuring/troubleshooting InfiniBand, RoCE, CLOS fabrics, 100GbE, Ethernet, GPU-direct, NCCLProficiency with Python and Go; experience improving internal tooling with SWE teamsExperience with monitoring/alerting tools (Prometheus, Grafana, Clickhouse) Proficiency with automation/configuration management tools (Ansible, Terraform) Excellent problem-solving and troubleshooting skills; high attention to detailPassion for continuous improvement and innovationproblem-solvingattention to detailcontinuous improvement mindsetInfiniBand (IB) and RoCE networkingNCCL and GPU-directCLOS fabrics