Self-hosted OpenAI-compatible LLM API with OpenLLM and vLLM

What they did

The developer wanted to stop paying per-token API bills while prototyping an internal chat assistant, so they self-hosted an open-source model instead of calling a hosted provider. They installed OpenLLM with pip, ran `openllm hello` to confirm the CLI worked, and then started a Qwen2.5 7B server with `openllm serve qwen2.5:7b`, which fit on a single 24G GPU. Because OpenLLM does not store weights itself, they set an HF_TOKEN environment variable first so the gated model download would succeed. The server came up with an OpenAI-compatible REST API plus a built-in chat UI, which they used to sanity-check responses before wiring anything else. They pointed their existing app's OpenAI client at the local base URL by changing only the base_url and api_key values, so no application code changed. To make the setup reproducible for teammates, they wrapped the server in a Docker image and documented the GPU requirement and token setup in the repo README. They also noted vLLM as the inference backend under the hood, which is why throughput on batched requests was reasonable even on modest hardware.

How it went

The local endpoint worked end to end: the app's existing OpenAI client code ran unchanged against the self-hosted model, and the built-in chat UI made quick prompt testing easy. Docker packaging gave teammates a repeatable one-command start. Response quality on a 7B model was noticeably below hosted frontier models for nuanced tasks, but fine for the prototype's classification and drafting flows.

Worth knowing

Requires a GPU with enough VRAM (24G for the 7B model) and a Hugging Face token for gated weights; OpenLLM does not host or cache model weights for you. Larger models need multi-GPU setups. The README excerpt does not cover auth, rate limiting, or production hardening, so those must be added separately before exposing the endpoint beyond localhost.

Wiring

Order of operations: install OpenLLM via pip, export HF_TOKEN so gated model weights can download, then run `openllm serve <model>:<size>` to launch the OpenAI-compatible API and chat UI on a local port. The app's OpenAI client is configured with base_url pointing at that local port and a placeholder api_key, so existing chat completion calls route to the self-hosted model. For deployment, a Dockerfile installs openllm and starts the same serve command, with GPU access passed through at runtime. vLLM acts as the inference backend behind OpenLLM, handling batching and GPU memory. Config lives in environment variables (HF_TOKEN, base URL, model name) rather than code.

Try it yourself

Set up a self-hosted, OpenAI-compatible LLM endpoint in this repo using OpenLLM. Steps: (1) add openllm to a requirements or pyproject file and document `pip install openllm`; (2) create a script `scripts/serve_llm.sh` that exports HF_TOKEN from the environment and runs `openllm serve qwen2.5:7b`; (3) add a small Python client module that reads OPENLLM_BASE_URL and OPENLLM_API_KEY env vars and calls the OpenAI-compatible chat completions endpoint, with a fallback to the hosted OpenAI API if the env var is unset; (4) add a Dockerfile that installs openllm and exposes the server port, noting the 24G GPU requirement in a comment; (5) write a README section covering HF_TOKEN setup, GPU requirements, and how to switch models. Do not hardcode any tokens.