llama.cpp alternatives
Tools that could take the place of llama.cpp, ranked by the stacks builders put them in rather than by stars alone.
Other runtimes in the runchr registry that do the work of llama.cpp — a tool somebody pairs with it ranks above a more popular stranger. Read the llama.cpp page or browse every runtimes.
Run open models locally and serve an OpenAI-compatible API.
A high-throughput inference and serving engine for GPUs.
Single-file executables that run LLMs and speech-to-text locally.
Optimized inference runtime for LLMs and visual gen models on NVIDIA GPUs.
High-performance LLM inference engine for multi-vendor GPUs and clusters.
CLI proxy that compresses shell command output before it reaches an LLM.
Extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS.
Distributed AI compute engine for scaling ML training, tuning, and serving.
On-device inference engine for phones, wearables, and robots.
Self-host open-source LLMs as OpenAI-compatible API endpoints.
OpenVINO toolkit optimizes and deploys deep learning inference across Intel
Portable C99 inference engine running Kimi K3 on a single CPU in 8 GB RAM.