Shimmy — Local Inference, OpenAI-Compatible
🔒 Local GPU inference — one binary, 100% Rust 🚀
Shimmy is independently maintained and free forever. Sponsorship funds certification, compatibility work, and releases.
Shimmy will be free forever. No asterisks. No "free for now." No pivot to paid.
What Is Shimmy?
Shimmy is a single-binary OpenAI-compatible inference server for GGUF models. Point your existing AI tools at Shimmy and they just work — locally, privately, and free.
Shimmy is the server. Airframe is the engine. Under the hood, Shimmy runs on Airframe (v0.4.4), a pure-Rust WebGPU (WGSL) transformer engine. No C++ toolchain, no Python runtime, no backend flags. 26 models certified across 12 families. Version history: CHANGELOG · Airframe CHANGELOG.
Why this matters:
- No Python runtime or C++ toolchain — Rust only, top to bottom
- F32 accumulation precision with deterministic output (same model + seed + params → same output)
- WGSL compute shaders via WebGPU — NVIDIA, AMD, Intel, integrated GPUs, Apple Silicon
- Model spec auto-derived from GGUF metadata — no hardcoded per-model constants
- YaRN RoPE scaling for extended context via
SHIMMY_MAX_CTX(see Extended Context)
🎯 Supported Models
12 model families · 26 certified model/quant combinations — every model below passes Shimmy's 3-box certification regimen (MATH + INFERENCE + DETERMINISM) against the certification ledger. Certification applies to the named model/quant combination; architecture recognition does not automatically mean certification. GGUF files load as-is; no recompilation, no hardcoded per-model constants.
| Family | Model | Quants |
|---|---|---|
| Llama | Llama-3.2-1B-Instruct | Q4_K_M · Q6_K |
| Llama-3.2-3B-Instruct | Q4_K_M | |
| Llama-3.1-8B-Instruct | Q4_K_M | |
| TinyLlama-1.1B-Chat | Q4_0 · Q5_K_M · Q6_K | |
| Qwen3 | Qwen3-0.6B | Q4_K_M |
| Qwen3-1.7B | Q4_K_M | |
| Qwen3-4B | Q4_K_M | |
| Qwen3-4B-Thinking | Q4_K_M | |
| Qwen3-8B | Q4_K_M | |
| Qwen2 | Qwen2-0.5B-Instruct | Q4_K_M |
| Qwen2-1.5B-Instruct | Q4_K_M | |
| Qwen2-7B-Instruct | Q4_K_M | |
| Qwen3.5 | Qwen3.5-9B | Q4_K_M |
| Phi-3 | Phi-3.5-mini-Instruct | Q4_K_M |
| Phi-3-mini-4k-Instruct | Q4_0 | |
| Phi-2 | Phi-2 | Q4_K_M |
| Gemma-2 | Gemma-2-2B-it | Q4_K_M |
| Gemma-2-9B-it | Q4_K_M (supported; cert: see v2-roadmap) | |
| Gemma-4 | Gemma-4-12B-coder | Q4_K_M |
| Gemma-4-E4B | Q4_K_M | |
| DeepSeek-R1 | DeepSeek-R1-0528-Qwen3-8B | Q4_K_M |
| Ministral | Ministral-3-14B-Reasoning | Q4_K_M |
| StarCoder2 | StarCoder2-3B | Q4_K_M |
SafeTensors format (.safetensors) is supported for model loading via safetensors_native. Full Airframe-native inference for SafeTensors remains roadmap work; see docs/v2-roadmap.md.
Features
- ⚡ TurboShimmy INT4 KV Cache — About 7× lower KV-cache memory in tested configurations. Run Llama-3.2-3B on 4 GB GPUs.
- 🚀 OpenAI SDK Compatibility — Chat completions, text completions, streaming, and model endpoints. Works with OpenAI SDKs and tools using that surface.
- 🔧 Extended Context — YaRN RoPE scaling via
SHIMMY_MAX_CTX. - 📦 Migrating from v1.x — llama.cpp, MLX, HuggingFace, and RustChain backends removed in v2.0+. Shimmy is now a pure Airframe product.
- 🏆 Certification — Every model passes a 3-box certification regimen (MATH + INFERENCE + DETERMINISM). See docs/CERTIFICATION.md.
- 🧠 MOE support — Mixture-of-Experts CPU offloading is on the Airframe roadmap.
Quick Start
Local loopback access needs no key. Network binds require SHIMMY_API_KEY;
browser clients must also be listed in SHIMMY_ALLOWED_ORIGINS.
See the network-access instructions.
cargo install shimmy --locked shimmy serve --model-path /absolute/path/to/model.gguf --bind 127.0.0.1:11435
Then in another terminal:
shimmy list --short curl -s http://127.0.0.1:11435/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"tinyllama-1.1b","messages":[{"role":"user","content":"Say hi in 5 words."}],"max_tokens":32}'
Full install, model acquisition, GPU, VRAM sizing, platform-specific builds: docs/quickstart.md
Documentation
| Start here | What you need |
|---|---|
| Quick Start | Install, models, GPU, VRAM |
| Supported Models | Certified models and quantization |
| API Compatibility | Endpoints, SDKs, integration |
| Configuration | Env vars and config options |
| Troubleshooting | GPU errors, model failures |
Complete documentation index
| Section | Documents |
|---|---|
| Models & Performance | TurboShimmy — INT4 KV cache compression · Extended Context — YaRN RoPE scaling, VRAM math · Performance — Tuning and token/sec · Model Expansion — Onboarding protocol |
| API & Integration | API Reference · OpenAPI / Swagger UI · Integration Guides · Examples · Cross-Compilation |
| Engine | Architecture · GPU Pipeline · Quantization · Chat Templates |
| Certification | Certification · Methodology · Regression Testing · PPT Testing · Metrics |
| FAQ | FAQ · Features · Migration · Windows GPU |
Development Testing
Shimmy maintains high code quality through comprehensive testing:
# Full test suite (default features = GPU engine) cargo test --release --locked # Quick CPU-only tests (no GPU required) cargo test --release --locked --no-default-features --features fast
See docs/ppt-invariant-testing.md for technical details.
Community & Support
- 🐛 Bug Reports: GitHub Issues
- 💬 Discussions: GitHub Discussions
- 📖 Security: Security Policy
🚀 Momentum Snapshot
🌟 
📰 As Featured On
🔥 Hacker News · Front Page Again · IPE Newsletter
Companies: Need invoicing? Email [email protected]
Performance
| Tool | Startup | Memory | API |
|---|---|---|---|
| Shimmy | <1s | ~50MB | Chat, completions, streaming, models |
| Ollama | 5-10s | 200MB+ | Partial |
Measured on RTX 3060, Shimmy v2.6.0, TinyLlama-1.1B. Your results vary by hardware.
Support
This project is independently maintained and free forever. Sponsorship helps fund ongoing development, certification, compatibility work, and releases.
Sponsor on GitHub · Buy Me a Coffee
Any amount is appreciated. Suggested tiers:
- $5/month — Thanks
- $15/month — Supporter
- $50/month — Backer
- Custom — Email for invoicing
License & Philosophy
MIT License — see LICENSE. Shimmy will be free forever.
Promise: This will never become a paid product.
Shimmy is infrastructure: it should be invisible. Reliability through comprehensive validation and property-based testing.
Maintainer: Michael A. Kuykendall · Mission: Making local model inference simple and reliable
Support
This project is a safe space. Trans rights are human rights.
If you or someone you love needs support:
- The Trevor Project — 24/7 for LGBTQ+ young people. Call 1-866-488-7386 or text START to 678-678
- Trans Lifeline — peer support run by and for trans people. US: 877-565-8860
- 988 Suicide & Crisis Lifeline — call or text 988
No comments yet.
to comment.