LLM Inference Engineer
LLM Inference Engineer
Employment Type: Full-Time (FTE)
Team: Founding Team
Experience Level: New Graduate to Senior
Location: Remote -SFO, CACompensation and Benefits
Base Salary: $120,000–$200,000
Equity: 0.25%–2.0%, with four-year vesting and a one-year cliff
Compensation offers flexibility to trade off base salary for additional equity. Benefits include health, dental, vision, and flexible time off.
Role Overview:
We are looking for an Inference Engineer to optimize the performance and cost of large language model (LLM) serving engines, including vLLM and SGLang. The role focuses on profiling inference workloads, improving GPU memory utilization, optimizing batching and caching, developing GPU kernels, and benchmarking performance on both NVIDIA and AMD GPUs. The ideal candidate combines strong systems engineering and GPU programming skills with hands-on experience modifying inference engines and demonstrating measurable performance improvements. Candidates ranging from new graduates to senior engineers are welcome, including those with relevant research, internships, thesis work, or open-source contributions.
Key Responsibilities:
Profile LLM inference workloads to identify performance bottlenecks involving execution time, GPU memory, latency, and throughput.
Develop and optimize inference-serving mechanisms, including continuous batching, KV cache, prefix caching, PagedAttention, speculative decoding, and chunked prefill.
Optimize scheduling and GPU resource utilization, including workload distribution across GPUs and prefill/decode execution.
Develop and optimize GPU kernels and runtime components using CUDA, Triton, ROCm, HIP, and related technologies.
Optimize model serving across NVIDIA and AMD GPU architectures.
Benchmark inference performance using metrics such as time-to-first-token (TTFT), inter-token latency (ITL/TPOT), tokens per second, throughput, and p99 latency.
Use profiling tools, hardware measurements, benchmark harnesses, and load testing to identify bottlenecks and validate improvements.
Support quantization and memory optimization techniques, including FP8, INT4, AWQ, and GPTQ.Bring newly released open-source models into the serving environment and optimize their inference performance.
Contribute production-ready code and improvements to open-source projects such as vLLM, SGLang, and related inference infrastructure.
Required Qualifications:
Strong background in Computer Science, Engineering, Machine Learning, Systems, or a related field.
Hands-on experience working with LLM inference or serving engines, such as vLLM, SGLang, TensorRT-LLM, llama.cpp, TGI, or FlashInfer.
Understanding of inference-serving internals, including KV cache, continuous batching, prefix caching, PagedAttention, speculative decoding, and prefill/decode execution.
Experience with performance profiling, benchmarking, and quantitative evaluation of inference workloads.
Strong systems programming skills in C++, Rust, CUDA, or Python, with an understanding of distributed systems, concurrency, memory management, schedulers, compilers, or high-performance computing (HPC).Experience with GPU programming and kernel optimization using technologies such as CUDA, Triton, ROCm, HIP, or CUTLASS.Ability to investigate performance bottlenecks, implement optimizations, and demonstrate measurable results.
Experience contributing code to production systems or open-source projects is valuable.
Preferred Qualifications:
Experience optimizing inference on NVIDIA H100/A100 or AMD MI300X/MI250 GPUs.
Knowledge of quantization techniques, including FP8, INT4, AWQ, and GPTQ.Experience with mixture-of-experts (MoE) serving and distributed inference.
Contributions to open-source projects, including vLLM, SGLang, TensorRT-LLM, or related GPU and compiler projects.
Experience with profiling tools such as Nsight, PyTorch Profiler, or rocprof.
Publications or research in systems and machine learning venues such as MLSys, OSDI, SOSP, or ASPLOS.Experience developing benchmark infrastructure, load-testing systems, or performance regression detection.
Relevant internships, thesis projects, research papers, or personal GitHub projects demonstrating inference optimization skills.
Key Skills:
LLM Inference | vLLM/SGLang | CUDA/ROCm | GPU Kernel Optimization | Performance Profiling | KV Cache