Self-hosted LLM inference service with LMDeploy TurboMind and vLLM comparison

What they did

They wanted a self-hosted, OpenAI-compatible inference endpoint for an internal chat and summarization feature, without sending data to a hosted API. Starting from the LMDeploy README, they installed the package via pip, converted a Hugging Face model into the TurboMind format, and launched the API server so existing OpenAI SDK client code could point at a local base URL. They then stood up vLLM on the same machine with the same model weights to compare serving behavior. The work covered: picking a model that both engines support, checking GPU memory headroom for KV cache, running the conversion step, starting each server on a distinct port, and writing a small load script that fires concurrent chat completions at both endpoints while recording time-to-first-token, tokens per second, and p95 latency. They also tested 4-bit weight-only quantization on the LMDeploy side to see whether a larger model would fit in the same VRAM, and checked that streaming responses and tool/function-call style payloads behaved the same through the OpenAI-compatible layer. Finally they wrapped the winning configuration in a systemd unit plus a reverse proxy so the endpoint survives reboots and is reachable from the app's staging environment.

How it went

Both engines served the same model behind an OpenAI-compatible API, so the application code needed only a base-URL change. LMDeploy's TurboMind path handled the quantized variant comfortably within the GPU budget and gave stable streaming output; vLLM was kept as a fallback and for models TurboMind did not yet cover. The benchmark script produced a small comparison table that made the engine choice defensible to the team, and the systemd plus proxy setup turned a notebook experiment into something staging could actually call. Polish level was solid for an internal service: reproducible launch commands, a documented model path, and a load script checked into the repo. It was not hardened for public traffic.

Worth knowing

Results are single-machine and model-specific; the README's headline speedups come from particular hardware and precision combinations, so their own numbers mattered more than the marketing ones. Model conversion adds a step that must be redone whenever weights change, and quantized variants can shift output quality in ways a latency benchmark will not catch. Neither server was configured with auth, rate limits, or TLS termination in this setup, and multi-GPU tensor parallelism plus PD disaggregation were left untested. GPU driver and CUDA version mismatches are the most common install failure.

Wiring

Order of operations: install lmdeploy (pip install lmdeploy) and confirm the CUDA/driver combination matches the wheel; run the model conversion CLI to produce a TurboMind workspace from the Hugging Face weights; launch the OpenAI-compatible API server pointing at that workspace, binding to a chosen port, with tensor-parallel size set to the GPU count and a KV-cache fraction that leaves headroom. Separately, install vLLM and start its OpenAI-compatible server on a second port with the same model identifier so client code is interchangeable. The application connects only through the OpenAI SDK, with base_url and api_key supplied by environment variables, so switching engines is a config change rather than a code change. A reverse proxy (nginx or similar) fronts the chosen port for staging access, and a systemd unit keeps the process alive across reboots. The benchmark script is the connective tissue: it takes a base URL as an argument and is run once per engine so the two result tables are directly comparable.

Try it yourself

In this repo, set up a self-hosted OpenAI-compatible LLM endpoint and benchmark it. Steps: (1) inspect the repo for any existing model-serving code, docker-compose files, or client code that calls an OpenAI-style API, and report what you find; (2) write a setup script that pip-installs lmdeploy, converts a Hugging Face model to the TurboMind format, and launches the api_server on a configurable port with a configurable model path; (3) add a second script that launches vLLM with the same weights on a different port; (4) write a Python benchmark that sends N concurrent streaming chat completions to a given base URL and reports time-to-first-token, output tokens/sec, and p95 latency as a table; (5) add a systemd unit and a short README section covering GPU memory sizing, quantization flags, and how to switch the app's base URL between the two engines. Do not hardcode model names or ports; use env vars with sane defaults.