{"schemaVersion":"jobsearcher.job.v1","id":"c7ae806696b3d60df61fc609","url":"https://jobsearcher.com/jobs/c7ae806696b3d60df61fc609","canonicalUrl":"https://jobsearcher.com/jobs/c7ae806696b3d60df61fc609","title":"Senior HPC Engineer - Fleet Engineering","description":"Overview\nIn this role you will build and operate the monitoring and automation foundation for Lambda’s AI compute clusters. You’ll manage large-scale HPC deployments, drive reliability across InfiniBand networks and GPU ecosystems, and codify operations to scale with demand. You’ll collaborate with on-site teams and cross-functional engineers to reduce incident impact and improve stability. This is a hands-on role at scale, shaping how Lambda delivers ubiquitous compute for researchers and enterprises.\n\nCompensation / Benefitsgenerous cash & equity compensationhealth, dental, and vision coveragewellness and commuter stipends401k with 2% company matchflexible paid time off\nResponsibilitiesBuild and operate monitoring and alerting for cluster health (fabric, GPU, power/thermal, job signals) to detect and respond proactivelyRemotely deploy and configure large-scale HPC clusters for AI workloads using automationAutomate cluster lifecycle (OS, firmware, drivers, networking) as code (Ansible, Terraform)Create runbooks and automated remediations for common cluster failure modes for safe execution by Support teamsTroubleshoot and resolve cluster issues across InfiniBand/RoCE, NCCL, GPU-direct, fabric, switching, and power; collaborate with on-site deployment teamsParticipate in on-call rotations and lead incident response for cluster-level problemsContribute to and maintain Standard Operating Procedures and feed requirements back to other engineering teams on simplification and stability\nKey requirements7+ years in Site Reliability Engineering, HPC Engineering, DevOps, or similarStrong understanding of modern AI infrastructure, including GPU architectures and hardware performance optimizationStrong Linux experience in distributed environmentsExperience configuring/troubleshooting InfiniBand, RoCE, CLOS fabrics, 100GbE, Ethernet, GPU-direct, NCCLProficiency with Python and Go; experience improving internal tooling with SWE teamsExperience with monitoring/alerting tools (Prometheus, Grafana, Clickhouse)Proficiency with automation/configuration management tools (Ansible, Terraform)Excellent problem-solving and troubleshooting skills; high attention to detailPassion for continuous improvement and innovationproblem-solvingattention to detailcontinuous improvement mindsetInfiniBand (IB) and RoCE networkingNCCL and GPU-directCLOS fabrics","company":"Lambda Labs","rawCompany":"lambda labs","city":"San Jose","state":"CA","isRemote":false,"isActive":true,"createdAt":"2026-09-22T03:47:33.874Z","occupations":[{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"},{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1244.00","title":"Network and Computer Systems Administrators","slug":"network-and-computer-systems-administrators"}],"industries":[{"code":"518210","title":"Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services","slug":"computing-infrastructure-providers-data-processing-web-hosting-and-related-services"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541513","title":"Computer Facilities Management Services","slug":"computer-facilities-management-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Senior HPC Engineer - Fleet Engineering","description":"Overview\nIn this role you will build and operate the monitoring and automation foundation for Lambda’s AI compute clusters. You’ll manage large-scale HPC deployments, drive reliability across InfiniBand networks and GPU ecosystems, and codify operations to scale with demand. You’ll collaborate with on-site teams and cross-functional engineers to reduce incident impact and improve stability. This is a hands-on role at scale, shaping how Lambda delivers ubiquitous compute for researchers and enterprises.\n\nCompensation / Benefitsgenerous cash & equity compensationhealth, dental, and vision coveragewellness and commuter stipends401k with 2% company matchflexible paid time off\nResponsibilitiesBuild and operate monitoring and alerting for cluster health (fabric, GPU, power/thermal, job signals) to detect and respond proactivelyRemotely deploy and configure large-scale HPC clusters for AI workloads using automationAutomate cluster lifecycle (OS, firmware, drivers, networking) as code (Ansible, Terraform)Create runbooks and automated remediations for common cluster failure modes for safe execution by Support teamsTroubleshoot and resolve cluster issues across InfiniBand/RoCE, NCCL, GPU-direct, fabric, switching, and power; collaborate with on-site deployment teamsParticipate in on-call rotations and lead incident response for cluster-level problemsContribute to and maintain Standard Operating Procedures and feed requirements back to other engineering teams on simplification and stability\nKey requirements7+ years in Site Reliability Engineering, HPC Engineering, DevOps, or similarStrong understanding of modern AI infrastructure, including GPU architectures and hardware performance optimizationStrong Linux experience in distributed environmentsExperience configuring/troubleshooting InfiniBand, RoCE, CLOS fabrics, 100GbE, Ethernet, GPU-direct, NCCLProficiency with Python and Go; experience improving internal tooling with SWE teamsExperience with monitoring/alerting tools (Prometheus, Grafana, Clickhouse)Proficiency with automation/configuration management tools (Ansible, Terraform)Excellent problem-solving and troubleshooting skills; high attention to detailPassion for continuous improvement and innovationproblem-solvingattention to detailcontinuous improvement mindsetInfiniBand (IB) and RoCE networkingNCCL and GPU-directCLOS fabrics","datePosted":"2026-09-22T03:47:33.874Z","dateModified":"2026-09-22T03:47:33.874Z","hiringOrganization":{"@type":"Organization","name":"Lambda Labs","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"San Jose","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"c7ae806696b3d60df61fc609"},"url":"https://jobsearcher.com/jobs/c7ae806696b3d60df61fc609"}}