TensorRT-LLM alternatives
Tools that could take the place of TensorRT-LLM, ranked by the stacks builders put them in rather than by stars alone.
Other runtimes in the runchr registry that do the work of TensorRT-LLM — a tool somebody pairs with it ranks above a more popular stranger. Read the TensorRT-LLM page or browse every runtimes.
A high-throughput inference and serving engine for GPUs.
Datacenter-scale distributed inference serving framework for LLMs and diffusion
Run quantised models locally, with an OpenAI-compatible server.
GPU kernel library for fast LLM inference: attention, GEMM, MoE, sampling.
High-performance LLM inference engine for multi-vendor GPUs and clusters.
CLI proxy that compresses shell command output before it reaches an LLM.
Extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS.
Distributed AI compute engine for scaling ML training, tuning, and serving.
Self-host open-source LLMs as OpenAI-compatible API endpoints.
Run and manage AI workloads across clouds, Kubernetes, and Slurm.
Python framework for serving AI models and LLM apps as production APIs.
Toolkit for compressing, deploying, and serving LLMs with TurboMind and PyTorch