Software Engineer - Distributed Training Infrastructure

Clockwork.io is a Silicon Valley startup that delivers state-of-the-art AI compute acceleration. We are founded by Stanford researchers and veteran systems engineers with a shared belief: distributed systems powering modern AI require a new approach to managing time, reliability, and performance. Unlike traditional solutions that rely on specialized hardware or embedded telemetry in switches, Clockwork’s system brings insane visibility, resilience, acceleration and efficiency to the network layer entirely through software. As AI workloads continue to scale in size, urgency, and impact, networks must evolve to keep up. Clockwork exists to make that evolution possible.About UsClockwork.io – A Software-Driven Revolution in AI NetworkingClockwork Systems was founded by Stanford researchers and veteran systems engineers who share a vision for redefining the foundations of distributed computing. As AI workloads grow increasingly complex, traditional infrastructure struggles to meet the demands of performance, reliability, and precise coordination. Clockwork is pioneering a software-driven approach to AI networking, delivering deterministic time, ultra-low latency, and seamless scalability for modern distributed systems.To learn more, visit www.clockwork.io.About the RoleWe are looking for an experienced software engineer to help build, optimize, and maintain large-scale distributed training infrastructure based on the PyTorch ecosystem. This role focuses on production-grade training workflows involving multi-GPU and multi-node orchestration, high-performance communication layers, and advanced parallelism strategies.You’ll work alongside infrastructure and machine learning teams to ensure training jobs are efficient, scalable, and resilient.What You will doDevelop and support distributed PyTorch training jobs using torch.distributed / c10dIntegrate and maintain frameworks like Megatron-LM, DeepSpeed, and related LLM training stacksDiagnose and resolve distributed training issues (e.g., NCCL hangs, OOM, checkpoint corruption)Optimize performance across communication, I/O, and memory bottlenecksImplement fault tolerance, checkpointing, and recovery mechanisms for long-running jobsWrite tooling and scripts to streamline training workflows and experiment managementCollaborate with ML engineers to ensure compatibility with orchestration and container environments (e.g., Slurm, Kubernetes)What We’re Looking ForDeep experience with PyTorch and torch.distributed (c10d)Hands-on experience with at least one of: Megatron-LM, DeepSpeed, or FairScaleProficiency in Python and Linux shell scriptingExperience with multi-node GPU clusters using Slurm, Kubernetes, or similarStrong understanding of NCCL, collective communication, and GPU topologyFamiliarity with debugging tools and techniques for distributed systemsPreferred SkillsExperience scaling LLM training across 8+ GPUs and multiple nodesKnowledge of tensor, pipeline, and data parallelismFamiliarity with containerized training environments (Docker, Singularity)Exposure to HPC environments or cloud GPU infrastructureExperience with training workload orchestration tools or custom job launchersComfort with large-scale checkpointing, resume/restart logic, and model I/O⸻Bonus SkillsProfiling tools: PyTorch Profiler, Nsight, nvprof, or equivalentExperience with performance tuning in distributed training environmentsContributions to ML infrastructure open-source projectsFamiliarity with storage, networking, or RDMA/GPU Direct technologiesUnderstanding of observability in ML pipelines (metrics, logs, dashboards)EnjoyChallenging projects.A friendly and inclusive workplace culture.Competitive compensation.A great benefits package.Catered lunch Clockwork is assembling world class teams to build cutting edge software. We look for bright people from all walks of life and we grow together. All qualified applicants will receive consideration for employment without regard to race, color, ancestry, religion, age, sex, sexual orientation, gender identity, national origin, or protected veteran status and will not be discriminated against on the basis of disability.