{"schemaVersion":"jobsearcher.job.v1","id":"342e498f862818404adfc2dd","url":"https://jobsearcher.com/jobs/342e498f862818404adfc2dd","canonicalUrl":"https://jobsearcher.com/jobs/342e498f862818404adfc2dd","title":"GPU Cluster Engineer, Systems & Platform","description":"Sciforium is an AI infrastructure company developing next-generation multimodal AI models and a proprietary, high-efficiency serving platform. Backed by multi-million-dollar funding and direct sponsorship from AMD with hands-on support from AMD engineers the team is scaling rapidly to build the full stack powering frontier AI models and real-time applications.\nAbout the Role\nWe are looking for a GPU Cluster Engineer to own the entire software stack of our GPU clusters — from kernel tuning and GPU drivers up through schedulers, containers, and ML frameworks. While our Hardware Operations team keeps the physical machines healthy and connected, you define what a production-ready node looks like in software: you author the images, playbooks, and pipelines that take a freshly provisioned server to a fully validated GPU node, and you keep the fleet consistent, upgradable, and fast. You will serve two demanding customer groups — our foundation model training teams and our model serving/product teams — ensuring both run on correctly configured, well-managed, high-performance infrastructure.\nKey Responsibilities\nOS Bring-Up & Node Lifecycle Engineering\nGolden Images & Automated Bring-Up: Own the node software definition — versioned OS images, kernel tuning (NUMA, hugepages, IRQ affinity, cgroups), GPU/NIC driver stacks — and the automated pipeline that takes a node from base OS to production-ready.\nValidation & Burn-In: Build automated acceptance suites (DCGM diagnostics, nccl-tests/RCCL tests, bandwidth and topology checks, HPL) that gate every node before it enters a scheduler pool.\nFleet Maintenance: Execute rolling kernel/driver/toolkit upgrades with minimal disruption to running workloads; enforce configuration consistency, detect drift, and maintain the driver CUDA/ROCm framework compatibility matrix across the fleet.\nSelf-Healing Operations: Automate detection of unhealthy nodes (Xid/ECC errors, link flaps, thermal throttling), with cordon/drain/reboot/re-image workflows and clean handoff to Hardware Operations for physical repair or RMA.\nConfiguration Management & Automation\nInfrastructure as Code: Manage all node and cluster configuration through Ansible/SaltStack playbooks in Git, with peer-reviewed changes, CI validation, and canary rollouts before fleet-wide deployment.\nProvisioning Pipelines: Build and maintain image/provisioning tooling (PXE, MaaS, Packer, or similar) so new or re-imaged nodes are reproducible, not hand-crafted.\nOperational Tooling: Develop Python/Bash tooling for cluster operations, health reporting, and workflow automation.\nOrchestration & Scheduling (Kubernetes & Slurm)\nKubernetes for Serving: Deploy and operate GPU-enabled Kubernetes for inference workloads — NVIDIA GPU Operator, device plugins, node feature discovery, topology-aware scheduling, and MIG/MPS partitioning where appropriate.\nTraining Schedulers: Operate Slurm (or Run:AI) for multi-node training — partitions, QoS, preemption, accounting, and container integration (enroot/pyxis).\nContainer Platform: Maintain base images, registries, and the NVIDIA Container Toolkit / ROCm container stack; keep training and serving images lean, current, and reproducible.\nGPU Driver & ML Stack Engineering\nDriver & Runtime Lifecycle: Build, deploy, and debug the full accelerator stack — NVIDIA (CUDA toolkit, cuDNN, NCCL, Fabric Manager) and AMD (ROCm, RCCL) — including kernel modules (DKMS), GPUDirect RDMA/Storage, and the RDMA software stack (MOFED/DOCA).\nFramework Environments: Maintain curated, optimized PyTorch and JAX environments with sane dependency and version management for researchers and production services.\nDistributed Performance: Tune NCCL/RCCL across NVLink/NVSwitch and InfiniBand/RoCE fabrics, ensure topology-aware job placement, and run continuous communication/throughput benchmarks to catch regressions.\nAdvanced Debugging & Observability\nEscalation Point: Own the hard problems — NCCL hangs and timeouts, CUDA memory leaks, ROCm kernel crashes, straggler nodes, and unexplained throughput drops.\nObservability: Own software-layer monitoring (DCGM exporter, Prometheus/Grafana, alerting) plus job-level GPU utilization and cluster efficiency reporting.\nQualifications\nMust-Haves:\n5+ years in systems/infrastructure engineering with significant GPU cluster, HPC, or large-scale ML infrastructure experience.\nBachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field.\nDeep Linux internals expertise: kernel modules/DKMS, systemd, cgroups, NUMA, and system performance tuning.\nHands-on experience with NVIDIA (CUDA) and/or AMD (ROCm) driver and runtime stacks on modern accelerators (H200/B200, MI325x/MI355x class), including kernel-level debugging.\nProduction Kubernetes experience with GPU workloads, plus working knowledge of HPC schedulers (Slurm/Run:AI) — or the reverse (deep Slurm, working K8s).\nStrong configuration management experience (Ansible or SaltStack) with Git-based, code-reviewed infrastructure workflows.\nProvisioning and image tooling experience (Packer, MaaS, Foreman, Terraform, or similar) for automated, reproducible node builds.\nClient-side experience with distributed filesystems (Lustre, GPFS, Weka) and checkpoint I/O optimization.\nContainer fluency: Docker/containerd and the NVIDIA Container Toolkit or ROCm equivalent.\nProficiency in Python and Bash for automation and tooling.\nWorking knowledge of NCCL and RDMA networking (InfiniBand/RoCE, GPUDirect) and of PyTorch/JAX runtime behavior.\nNice-to-Haves:\nExperience directly supporting foundation model training teams — multi-node job failure debugging, checkpoint pipeline tuning, and framework-level performance triage — ideally in a startup or research-heavy environment.\nExperience deploying and tuning inference/serving stacks (vLLM, Triton Inference Server, TensorRT-LLM) for latency and throughput targets.\nGPU/system profiling tools: Nsight Systems/Compute, rocprof, perf, eBPF.\nBenefits include\nMedical, dental, and vision insurance\n401k plan\nDaily lunch, snacks, and beverages\nFlexible time off\nCompetitive salary and equity\nEqual opportunity\nSciforium is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status.\nCompensation Range: $150K - $220K","company":"Sciforium","rawCompany":"sciforium","city":"Millbrae","state":"CA","isRemote":false,"isActive":false,"createdAt":"2026-08-09T10:51:41.159Z","occupations":[{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"},{"code":"15-1244.00","title":"Network and Computer Systems Administrators","slug":"network-and-computer-systems-administrators"},{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"518210","title":"Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services","slug":"computing-infrastructure-providers-data-processing-web-hosting-and-related-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"GPU Cluster Engineer, Systems & Platform","description":"Sciforium is an AI infrastructure company developing next-generation multimodal AI models and a proprietary, high-efficiency serving platform. Backed by multi-million-dollar funding and direct sponsorship from AMD with hands-on support from AMD engineers the team is scaling rapidly to build the full stack powering frontier AI models and real-time applications.\nAbout the Role\nWe are looking for a GPU Cluster Engineer to own the entire software stack of our GPU clusters — from kernel tuning and GPU drivers up through schedulers, containers, and ML frameworks. While our Hardware Operations team keeps the physical machines healthy and connected, you define what a production-ready node looks like in software: you author the images, playbooks, and pipelines that take a freshly provisioned server to a fully validated GPU node, and you keep the fleet consistent, upgradable, and fast. You will serve two demanding customer groups — our foundation model training teams and our model serving/product teams — ensuring both run on correctly configured, well-managed, high-performance infrastructure.\nKey Responsibilities\nOS Bring-Up & Node Lifecycle Engineering\nGolden Images & Automated Bring-Up: Own the node software definition — versioned OS images, kernel tuning (NUMA, hugepages, IRQ affinity, cgroups), GPU/NIC driver stacks — and the automated pipeline that takes a node from base OS to production-ready.\nValidation & Burn-In: Build automated acceptance suites (DCGM diagnostics, nccl-tests/RCCL tests, bandwidth and topology checks, HPL) that gate every node before it enters a scheduler pool.\nFleet Maintenance: Execute rolling kernel/driver/toolkit upgrades with minimal disruption to running workloads; enforce configuration consistency, detect drift, and maintain the driver CUDA/ROCm framework compatibility matrix across the fleet.\nSelf-Healing Operations: Automate detection of unhealthy nodes (Xid/ECC errors, link flaps, thermal throttling), with cordon/drain/reboot/re-image workflows and clean handoff to Hardware Operations for physical repair or RMA.\nConfiguration Management & Automation\nInfrastructure as Code: Manage all node and cluster configuration through Ansible/SaltStack playbooks in Git, with peer-reviewed changes, CI validation, and canary rollouts before fleet-wide deployment.\nProvisioning Pipelines: Build and maintain image/provisioning tooling (PXE, MaaS, Packer, or similar) so new or re-imaged nodes are reproducible, not hand-crafted.\nOperational Tooling: Develop Python/Bash tooling for cluster operations, health reporting, and workflow automation.\nOrchestration & Scheduling (Kubernetes & Slurm)\nKubernetes for Serving: Deploy and operate GPU-enabled Kubernetes for inference workloads — NVIDIA GPU Operator, device plugins, node feature discovery, topology-aware scheduling, and MIG/MPS partitioning where appropriate.\nTraining Schedulers: Operate Slurm (or Run:AI) for multi-node training — partitions, QoS, preemption, accounting, and container integration (enroot/pyxis).\nContainer Platform: Maintain base images, registries, and the NVIDIA Container Toolkit / ROCm container stack; keep training and serving images lean, current, and reproducible.\nGPU Driver & ML Stack Engineering\nDriver & Runtime Lifecycle: Build, deploy, and debug the full accelerator stack — NVIDIA (CUDA toolkit, cuDNN, NCCL, Fabric Manager) and AMD (ROCm, RCCL) — including kernel modules (DKMS), GPUDirect RDMA/Storage, and the RDMA software stack (MOFED/DOCA).\nFramework Environments: Maintain curated, optimized PyTorch and JAX environments with sane dependency and version management for researchers and production services.\nDistributed Performance: Tune NCCL/RCCL across NVLink/NVSwitch and InfiniBand/RoCE fabrics, ensure topology-aware job placement, and run continuous communication/throughput benchmarks to catch regressions.\nAdvanced Debugging & Observability\nEscalation Point: Own the hard problems — NCCL hangs and timeouts, CUDA memory leaks, ROCm kernel crashes, straggler nodes, and unexplained throughput drops.\nObservability: Own software-layer monitoring (DCGM exporter, Prometheus/Grafana, alerting) plus job-level GPU utilization and cluster efficiency reporting.\nQualifications\nMust-Haves:\n5+ years in systems/infrastructure engineering with significant GPU cluster, HPC, or large-scale ML infrastructure experience.\nBachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field.\nDeep Linux internals expertise: kernel modules/DKMS, systemd, cgroups, NUMA, and system performance tuning.\nHands-on experience with NVIDIA (CUDA) and/or AMD (ROCm) driver and runtime stacks on modern accelerators (H200/B200, MI325x/MI355x class), including kernel-level debugging.\nProduction Kubernetes experience with GPU workloads, plus working knowledge of HPC schedulers (Slurm/Run:AI) — or the reverse (deep Slurm, working K8s).\nStrong configuration management experience (Ansible or SaltStack) with Git-based, code-reviewed infrastructure workflows.\nProvisioning and image tooling experience (Packer, MaaS, Foreman, Terraform, or similar) for automated, reproducible node builds.\nClient-side experience with distributed filesystems (Lustre, GPFS, Weka) and checkpoint I/O optimization.\nContainer fluency: Docker/containerd and the NVIDIA Container Toolkit or ROCm equivalent.\nProficiency in Python and Bash for automation and tooling.\nWorking knowledge of NCCL and RDMA networking (InfiniBand/RoCE, GPUDirect) and of PyTorch/JAX runtime behavior.\nNice-to-Haves:\nExperience directly supporting foundation model training teams — multi-node job failure debugging, checkpoint pipeline tuning, and framework-level performance triage — ideally in a startup or research-heavy environment.\nExperience deploying and tuning inference/serving stacks (vLLM, Triton Inference Server, TensorRT-LLM) for latency and throughput targets.\nGPU/system profiling tools: Nsight Systems/Compute, rocprof, perf, eBPF.\nBenefits include\nMedical, dental, and vision insurance\n401k plan\nDaily lunch, snacks, and beverages\nFlexible time off\nCompetitive salary and equity\nEqual opportunity\nSciforium is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status.\nCompensation Range: $150K - $220K","datePosted":"2026-08-09T10:51:41.159Z","dateModified":"2026-08-09T10:51:41.159Z","hiringOrganization":{"@type":"Organization","name":"Sciforium","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Millbrae","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"342e498f862818404adfc2dd"},"url":"https://jobsearcher.com/jobs/342e498f862818404adfc2dd"}}