Member of Technical Staff, Inference Systems

About the Company

Our client is a seed-stage, stealth-mode startup building a high-performance AI inference platform from the ground up. The founding team brings deep AI-infrastructure experience and is tackling the hardest problems in LLM serving — scheduling, KV cache management, request routing, and the runtime systems that power model inference at scale — with Rust at the core of the stack. This is a ground-floor opportunity to shape the architecture of a fast, reliable inference system without legacy constraints.

• Recently founded
• Small founding team
• Industry: AI infrastructure / LLM inference

The Role

You'll join a small, fast-moving team building a new inference system from scratch. This is a role for a systems engineer who lives and breathes inference internals — attention, KV cache, batching, scheduling — and wants to own the whole stack rather than a narrow slice.

What you'll be doing

• Build a new inference runtime from scratch in Rust, owning batching, scheduling, request routing, and the full serving stack.
• Design and implement KV cache management, prefix caching, and optimizations that cut latency and cost per token.
• Scale serving across GPUs and nodes, tackling multi-GPU and multi-node challenges directly.
• Profile, benchmark, and ship performance improvements across the entire inference pipeline.
• Work closely with the founding team on the core architectural decisions that define the platform.

Tech stack: Rust, Python, PyTorch, C++, Go, vLLM, SGLang, TensorRT-LLM, CUDA, Triton, NCCL

Requirements

• 2–10 years of experience as a backend or distributed systems engineer.
• Hands-on experience building, operating, or optimizing LLM inference or serving systems at the engine, router, or runtime layer — beyond just calling hosted APIs.
• A deep working knowledge of transformer inference internals: attention, KV cache, batching, scheduling, and where the real bottlenecks are.
• Performance-critical backend or distributed systems experience where latency, throughput, and cost were first-order concerns.
• Hands-on time with a production inference engine (vLLM, SGLang, or TensorRT-LLM) plus strong systems-language skills (Rust, C++, Go, or systems-level Python/PyTorch).
• If you haven't used Rust yet, you should be able to become productive in it within a few weeks of joining.

Nice to Haves

• Time on an inference team at a model provider, accelerator vendor, or research lab, or open-source contributions to vLLM, SGLang, or Dynamo.
• A CS or systems degree from a strong program.
• Production Rust experience, CUDA/Triton kernel work, multi-GPU or multi-node serving (NCCL, NVLink, RDMA), prefix caching, speculative decoding, or prefill/decode disaggregation.

Why Join

• Own the architecture of a brand-new inference system with no legacy constraints.
• Solve the hardest problems in LLM serving, from KV cache management to request routing.
• A performance engineer's dream: latency and cost per token are the whole game.
• High visibility on a small, elite team where your code powers the core engine.

Details

Location: Palo Alto, CA

Work policy: Full-time, on-site five days a week

Compensation: $230K–$350K + equity

Visa sponsorship: Open to visa transfers (OPT, H-1B) and new sponsorships (new H-1B, TN)

Employment type: Full-time