Open Source Inference Stack



Introduction
Teams that serve their own models need two things that are hard to build at the same time. They need one front door for every client and every team, with keys, budgets and usage tracking. They also need each request to land on the model replica that can answer it fastest, without one team’s backlog blocking another.
This stack pairs three open source projects that each do one of those jobs. LiteLLM is the gateway, llm-d is the router and vLLM is the inference engine.
You can try the whole stack on a laptop with one docker compose up, with no GPU and no Kubernetes. See Get Started.
LiteLLM
The gateway: one API, virtual keys, teams and spend tracking
LiteLLM is an open source LLM gateway. Clients talk to it in the OpenAI or Anthropic format, and it handles who is allowed to call which model.
- One API for every client. OpenAI Chat Completions and Anthropic Messages both work against the same self-hosted model, so Claude Code and OpenAI SDK users share one endpoint.
- Virtual keys and teams. Each key belongs to a team, with its own model access, budget and rate limits.
- Spend and usage tracking. Every request is attributed to a key and a team, including for self-hosted models.
llm-d
The router: sends each request to the best replica
llm-d is a Kubernetes-native router for self-hosted inference. It sits between the gateway and the model servers.
- Cache-aware routing. It prefers the replica that already holds the prompt prefix in its cache, which cuts time to first token.
- Fairness between tenants. With flow control on, requests are queued per tenant and served in turn, so a busy team cannot starve a quiet one.
- Priority bands. Traffic can be marked premium, standard or best effort. Under load the router serves the highest band first and rejects a request only when its band’s queue is full.
Flow control in llm-d. Diagram from the llm-d docs.
vLLM
The engine: serves the model
vLLM is a high-throughput inference server with an OpenAI-compatible API. llm-d reads its queue and cache metrics to decide where each request goes.
Integration
LiteLLM already knows which team a key belongs to. It passes that to llm-d as two request headers, and llm-d uses them to queue the request.
| Header | Meaning | Set from |
|---|---|---|
x-llm-d-inference-fairness-id | The tenant the request belongs to. Requests with the same ID share one queue | The LiteLLM team on the key |
x-llm-d-inference-objective | The name of an llm-d InferenceObjective, which carries the priority | A per-key or per-model setting |
Clients do not send these headers. The gateway sets them after it authenticates the key, so a client cannot claim another team’s place in the queue. The llm-d header reference lists both.
Why This Stack?
- Each layer does one job. The gateway owns identity and accounting, the router owns placement, the engine owns tokens per second.
- Open source end to end. llm-d and vLLM are Apache 2.0. LiteLLM is MIT outside its
enterprisedirectory. All three are self-hosted. - Runs where your GPUs are. Nothing leaves your cluster.
Getting Started
Follow Get Started to run the stack locally in a few minutes, then move to a real cluster with the llm-d LiteLLM guide.
Questions/Feedback?
- LiteLLM: GitHub issues
- llm-d: GitHub issues