dynamo

| Docs | Roadmap | Recipes | Examples | Containers | Digest | Design Proposals | How to Contribute | Slack |

English | 简体中文

The open-source, datacenter-scale inference stack. Dynamo is the orchestration layer above inference engines — it doesn't replace SGLang, TensorRT-LLM, or vLLM, it turns them into a coordinated multi-node inference system. Disaggregated serving, intelligent routing, multi-tier KV caching, and automatic scaling work together to maximize throughput and minimize latency for LLM, reasoning, multimodal, and video generation workloads.

Built in Rust for performance, Python for extensibility.

Events are updated automatically. Subscribe to our public calendar.

• You're serving LLMs across multiple GPUs or nodes and need to coordinate them
• You want KV-aware routing to avoid redundant prefill computation
• You need to independently scale prefill and decode (disaggregated serving)
• You want automatic scaling that meets latency SLAs at minimum total cost of ownership (TCO)
• You need fast cold-starts when spinning up new replicas

If you're running a single model on a single GPU, your inference engine alone is probably sufficient.

Feature support at a glance:

Full Feature Matrix → — LoRA, request migration, speculative decoding, and feature interactions.

Most inference engines optimize a single GPU or a single node. Dynamo is the orchestration layer above them — it turns a cluster of GPUs into a coordinated inference system.

Architecture Deep Dive →

• Zero-config deploy (DGDR) (beta): Specify model, HW, and SLA in one YAML — AISimulate performance models estimate candidate configurations, Planner optimizes the topology, and Dynamo deploys
• Agentic inference: Per-request hints for priority, expected output length, and speculative prefill, plus session metadata for tracing and SGLang subagent KV isolation. LangChain + NeMo Agent Toolkit integrations
• Multimodal E/P/D: Disaggregated encode/prefill/decode with embedding cache — 30% faster TTFT on image workloads
• Video generation: Native FastVideo + SGLang Diffusion support — real-time 1080p on single B200
• K8s Inference Gateway plugin: KV-aware routing inside the standard Kubernetes gateway
• Storage-tier KV offload: S3/Azure blob support + global KV events for cluster-wide cache visibility

Dynamo can expose traffic through two Kubernetes request routing topologies. Both expose an OpenAI-compatible API and support the same backends, disaggregated serving, and KV-aware routing.

Request flow for the Dynamo-native path is client → Frontend → Router → workers. Request flow for the Gateway API path is client → Gateway → EPP → Frontend sidecar (direct) → workers.

See the Gateway API Inference Extension (GAIE) guide for the Gateway API setup, supported features, and configuration.

This repo ships agent skills: if you work with an AI coding agent (Claude Code, Codex, Cursor), clone the repo and ask it to deploy, troubleshoot, benchmark, or optimize a Dynamo deployment. The skills activate automatically; no setup required.

Choose either vLLM or SGLang:

# Pull a prebuilt container
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.0

# Inside the container — start frontend and worker
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 &
python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --discovery-backend file &

# Once the worker is ready, send a request
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [{"role": "user", "content": "Hello!"}],
  "max_tokens": 100
}' | jq

# Pull a prebuilt container
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.6.0

# Inside the container — start frontend and worker
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 &
python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --discovery-backend file &

# Once the worker is ready, send a request
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [{"role": "user", "content": "Hello!"}],
  "max_tokens": 100
}' | jq

Also available: tensorrtllm-runtime:1.6.0.

Install uv (curl -LsSf https://astral.sh/uv/install.sh | sh), then choose your backend:

uv pip install --prerelease=allow "ai-dynamo[vllm]"

uv pip install --prerelease=allow "ai-dynamo[sglang]"

Note: TensorRT-LLM requires pip with --extra-index-url https://pypi.nvidia.com. See the install guide for TRT-LLM-specific instructions.

Then start the frontend and the matching worker as shown above. See the full installation guide for system dependencies and backend-specific notes.

For production multi-node clusters, install the Dynamo Platform and deploy with a single manifest:

# Zero-config deploy: specify model + SLA, Dynamo handles the rest
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: my-model
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  sla:
    ttft: 200.0   # ms
    itl: 20.0     # ms
  autoApply: true

Pre-built recipes for common models:

See recipes/ for the full list. Cloud-specific guides: AWS EKS · Google GKE · Azure AKS · Amazon ECS

For contributors who want to build and develop locally. See the full build guide for details.

# Install system deps (Ubuntu 24.04)
sudo apt install -y build-essential libhwloc-dev libudev-dev pkg-config libclang-dev protobuf-compiler python3-dev cmake

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh && source $HOME/.cargo/env

# Create venv and build
uv venv dynamo && source dynamo/bin/activate
uv pip install pip 'maturin[patchelf]'
cd lib/bindings/python && maturin develop --uv && cd $PROJECT_ROOT
uv pip install -e lib/gpu_memory_service
uv pip install -e .

VSCode/Cursor users: see the .devcontainer for a pre-configured dev environment.

Dynamo is built in the open with an OSS-first development model. We welcome contributions of all kinds.

• Contribution Guide — How to contribute code, docs, and recipes
• Design Proposals — RFCs for major features, tracked as dep:* labeled GitHub issues
• Office Hours — Biweekly calls
• Community Meetings (Youtube) – Weekly (Tue 9:00 AM PT) development community meetings
• Slack — Chat with the team and community
• Dynamo Day Recordings — Deep dives from production users

• [03/15] Dynamo 1.0 is here — production-ready with strong community adoption
• [03/15] NVIDIA Blackwell Ultra sets new inference records in MLPerf
• [03/15] NVIDIA Blackwell leads on SemiAnalysis InferenceMax benchmarks
• [12/05] Moonshot AI's Kimi K2 achieves 10x inference speedup with Dynamo on GB200
• [12/02] Mistral AI runs Mistral Large 3 with 10x faster inference using Dynamo
• [11/20] Dell integrates PowerScale with NIXL for 19x faster TTFT

cargo run -p dynamo-llm --bin generate-frontend-openapi

• Support Matrix — Hardware, OS, CUDA, and backend versions
• Feature Matrix — Detailed backend compatibility
• Release Artifacts — Containers, wheels, Helm charts
• Service Discovery — K8s-native vs etcd vs file-based discovery
• Benchmarking Guide — Compare deployment topologies with AIPerf