Senior AI Infrastructure Engineer
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
*About StackYak*
StackYak is building the infrastructure layer for running AI workloads in production. We bring together GPU compute, model serving, networking, and cloud infrastructure so teams can deploy AI without assembling and operating the underlying stack themselves.
We're looking for a *Senior AI Inference Engineer* who knows how to make models run fast, reliably, and efficiently on GPUs.
*What you'll work on*
You'll own the technical depth of our inference stack: how models are loaded, served, optimized, observed, and scaled across different GPU hardware.
You'll work hands-on with systems such as *vLLM, SGLang, TensorRT-LLM, CUDA and ROCm*, digging below the API layer into what actually determines inference performance.
You'll:
- Deploy and operate production model-serving infrastructure using vLLM and similar runtimes
- Optimize *TTFT, inter-token latency, throughput, GPU utilization, memory usage, and cost*
- Tune batching, KV cache behavior, parallelism, quantization, model configuration, and runtime parameters
- Benchmark models across different NVIDIA and AMD GPU architectures
- Diagnose GPU memory, driver, runtime, networking, and performance bottlenecks
- Build reliable patterns for multi-GPU and multi-node inference
- Improve how StackYak provisions, configures, monitors, and operates inference workloads
- Help determine the right hardware and serving configuration for different models and customer workloads
- Build the telemetry and tooling needed to understand inference performance in production
- Work closely with the rest of the engineering team as we turn this operational knowledge into the StackYak platform
*What we're looking for*
We're particularly interested in engineers who have *actually operated LLM inference infrastructure*, not just consumed hosted model APIs.
You should have strong experience with several of:
- *vLLM, SGLang, TensorRT-LLM, TGI, Triton*, or comparable inference runtimes
- NVIDIA GPUs, CUDA, NCCL and the surrounding software stack
- GPU profiling, benchmarking, memory management, and performance optimization
- Quantization and model formats such as FP8, INT8, AWQ or GPTQ
- Tensor parallelism, pipeline parallelism, and distributed inference
- Docker and Linux systems engineering
- Python and the modern ML ecosystem
- Production observability and performance troubleshooting
Experience with *AMD GPUs/ROCm*, high-performance networking, RDMA/InfiniBand/RoCE, or operating GPU infrastructure in data centers is especially useful.
You do not need to be an ML researcher. We're looking for someone who understands the *systems engineering of inference*: what happens between receiving a request and getting tokens out of a GPU, and how to make every part of that path better.
This is a hands-on engineering role at an early-stage company. You'll have significant influence over how StackYak's inference infrastructure is designed and how we run models across an increasingly diverse fleet of GPU hardware.
Pay: $160,000.00 - $200,000.00 per year
Work Location: In person