AI Inference Optimization Engineer
Job Title: AI Inference Optimization Engineer
Job Location: Santa Clara, CA - OnsiteJob Type - Long Term Contract Role (2+ years)
Job Description:
The engineer will optimize LLM inference across GPU hardware, serving engines, and distributed deployments. This is a senior performance engineering role focused on achieving agreed latency, throughput, model quality, and cost targets. The engineer will use controlled benchmarks and profiling to distinguish model, runtime, memory, and communication bottlenecks.
Key responsibilities:
- -Establish reproducible inference baselines using representative models, input and output lengths, concurrency levels, and traffic patterns. Record hardware, software, and configuration versions.
- Evaluate and tune vLLM, SGLang, TensorRT-LLM, Triton Inference Server, and NVIDIA Dynamo where appropriate to the deployment architecture.
- Measure time to first token, inter-token latency, end-to-end latency, tail latency, throughput, GPU memory use, and resource efficiency under sustained and burst load.
- Optimize batching, KV-cache allocation and reuse, scheduling, model loading, and parallelism strategies. Evaluate distributed and disaggregated serving where supported and beneficial.
- Assess quantization approaches such as FP8 and NVFP4 on compatible hardware and software. Use representative quality evaluations and document accuracy and performance tradeoffs.
- Profile CPU and GPU execution, memory transfers, kernel behavior, and communication with appropriate tools. Diagnose out-of-memory conditions and poor utilization.
- Investigate multi-node serving issues with platform and network teams, including NCCL, RDMA paths, topology, and transport dependencies where used by the serving stack.
- Create performance regression and soak tests, recommend production settings, and provide validated deployment artifacts, benchmark reports, and rollback guidance to delivery teams.
- Required experience and skills-7+ years in performance engineering, ML systems, GPU software, or related systems engineering, including hands-on model serving and optimization.
- Strong Python and practical C++ skills, with knowledge of GPU execution, memory hierarchy, numerical precision, and parallel computing.
- Hands-on experience with at least one production LLM serving stack such as vLLM, SGLang, TensorRT-LLM, or Triton.
- Understanding of transformer inference, prefill and decode phases, KV-cache behavior, batching, quantization, and model parallelism.
- Ability to design fair benchmarks, control variables, interpret latency distributions, and verify that performance changes preserve required output quality.
- Strong Linux, container, and debugging skills, with the ability to isolate runtime, model, driver, hardware, and communication issues.
- Preferred experience
- Experience with NVIDIA Nsight Systems or Nsight Compute, CUDA profiling, and targeted kernel or memory-transfer optimization.
- Experience with NVIDIA Dynamo, multi-GPU or multi-node inference, speculative decoding, and prefix caching where supported.
- Experience validating new GPU architectures, low-precision formats, and compatibility across model, driver, runtime, and serving engine versions.
Recommended certifications Preferred: NVIDIA-Certified Professional Generative AI LLMs (NCP-GENL). Additional infrastructure or operations credentials: NVIDIA-Certified Professional AI Infrastructure (NCP-AII) or AI Operations (NCP-AIO). Foundational: NCA-AIIO.Certifications are preferred credentials. Candidates should demonstrate the corresponding skills through practical delivery and troubleshooting experience.
Expected deliverables and success measures:
Deliver reproducible benchmark suites, profiling evidence, quality comparisons, and recommended serving configurations. Success is demonstrated by improvements against agreed baselines, stable sustained-load behavior, and clearly documented tradeoffs without unvalidated quality regression.