Software Engineer, Inference
Overview:
In this role you will help design and operate Sierra's inference architecture for self-hosted models and third-party providers. You will shape how inference traffic flows, manage capacity and latency, and ensure cost-efficient, reliable production performance. You'll collaborate with frontier labs and providers to scale AI infrastructure and support models throughout their lifecycle. This is a systems-first, AI-focused role at scale that blends distributed infrastructure with cutting-edge inference techniques. Compensation / Benefits Flexible (unlimited) paid time off Medical, dental, vision benefits for you and family Life insurance and disability benefits Retirement plan (country-dependent) Parental leave Fertility and family building benefits through Carrot
Responsibilities:
- Design and shape Sierra's inference architecture across models, infrastructure, and providers
- Build and operate self-hosted inference on GPU infrastructure, including containers and inference engines
- Develop routing, failover, capacity management, and quota systems for low-latency, reliable inference at scale
- Work with Applied Research to optimize latency, throughput, and cost via techniques like speculative decoding and serving-engine optimizations
- Collaborate with inference providers to tune engines and push the serving stack forward
- Support broader model lifecycle by contributing to post-training infrastructure in partnership with Models and Agent Runtime teams
Key requirements:
- Deep systems thinking and distributed systems fundamentals
- Experience designing, building, and operating large-scale production systems
- Strong judgment on latency, reliability, capacity, and cost tradeoffs
- Experience owning complex infrastructure end-to-end from architecture to production operations
- Curiosity about applying systems expertise to AI infrastructure and quick learning of evolving tech
- Ownership mindset
- Analytical problem solving
- Collaborative cross-team communication
- Experience with large-scale production systems
- Distributed systems fundamentalsML infrastructure, MLOps, or production inference systems (nice to have)