Senior Python Engineer (Agent & Infrastructure)
About the company:
We’re working with a top AI infrastructure platform that’s on track to 3X last year’s ARR. Their software sits underneath how ML teams run their work. It schedules jobs, manages GPU fleets and moves experiments from laptop to cluster, across cloud, on-prem and Kubernetes. Teams from fast-moving startups to large enterprises run production workloads on it.
About the role:
You’ll own the execution layer of the platform: the components that spin up containers, manage GPUs and tunnel connections so everything works across very different environments. The role sits between software engineering and DevOps. You’ll write Python that orchestrates infrastructure, manages Docker containers, talks to the Kubernetes API and handles low-level networking.
Responsibilities:
- Agent development: design and optimize the agent, a Python service that pulls jobs, sets up environments and executes ML pipelines
- Kubernetes integration: write logic that talks directly to the K8s APIs, manage pod lifecycles and handle CRDsResource management: implement dynamic allocation of GPU, CPU and memory, and container orchestration
- Systems programming: build robust daemons and services that interact with OS-level primitives (systemd, signals, I/O streams)
- Networking: troubleshoot and optimize TCP/IP connections, DNS resolution and firewall traversal so users stay connected
Requirements:
- 8+ years of development experience, with a strong focus on systems programming
- Kubernetes mastery: a deep understanding of the architecture, beyond writing YAML. You know how to write code that controls K8sContainer internals: extensive Docker experience, including building and maintaining images
- Python for systems: automation, daemons or CLI tools, using libraries like subprocess, socket and asyncio
- Networking fundamentals: HTTP/S, WebSockets, TCP/IP, proxies and reverse proxiesOS knowledge: strong Linux internals and shell scripting
Nice to have:
- GPU hardware management (NVIDIA drivers, CUDA, NVIDIA Container Toolkit)
- Building Kubernetes operators or controllers (Kopf or Operator SDK)
- Background in HPC, Slurm or MPIGo, for specific K8s components