FastDeploy + vLLM-compatible LLM/VLM serving stack for Paddle models

What they did

They wanted a production-grade way to serve PaddlePaddle LLM and VLM checkpoints without rewriting their existing inference clients. Starting from the FastDeploy README, they picked a supported model (Qwen3-VL family or ERNIE-4.5-VL-Thinking were the documented options), installed FastDeploy on a Linux host with Python 3.10-3.12 against their target accelerator (NVIDIA GPU being the default path), and launched the OpenAI-compatible server with a single command. Because FastDeploy advertises vLLM interface compatibility, they pointed their existing vLLM-style client code at the FastDeploy endpoint instead of standing up a separate vLLM deployment, keeping request/response shapes and chat-completion semantics stable. They then exercised the endpoint with a smoke test covering text-only chat, a multimodal image prompt, and a streaming request, and compared latency and throughput against their prior baseline. Finally they documented the launch flags, quantization choice (W8A8 or FP8 depending on hardware), and the model revision pin so the deployment could be reproduced.

How it went

The stack came up cleanly on a single NVIDIA GPU node and served both text and image prompts through the OpenAI-compatible endpoint. Existing vLLM-style client code worked after only a base-URL change, which was the main win. Throughput and latency were acceptable for a staging workload; the team did not run a full production load test, so SLO behavior under sustained concurrency remains unverified. Documentation and launch flags were captured in the repo so a second engineer could reproduce the deployment.

Worth knowing

Linux only, Python 3.10-3.12, and installation differs per accelerator, so the exact wheel/index URL matters. vLLM compatibility covers the API surface, not every vLLM feature or flag, so advanced sampling or scheduling options may not map one-to-one. PD disaggregation, KV-cache transfer, and speculative decoding are documented but need multi-node or multi-GPU setup that this walkthrough did not exercise. Model support is a moving list; pin a release tag rather than tracking develop.

Wiring

Order: install FastDeploy for the detected accelerator, pin a release tag, then launch the server, then point clients at it. Config lives in scripts/serve.sh (model path or HF id, quantization, tensor-parallel size, max-model-len, port) and in a .env for the base URL. The server exposes an OpenAI-compatible HTTP API; clients use any OpenAI-style SDK with base_url set to http://localhost:<port>/v1. vLLM compatibility is at the API layer, so existing vLLM client code connects by changing only the base URL and model name. The smoke test script is the connection check between the two tools: it calls the FastDeploy endpoint using vLLM-style request shapes and asserts non-empty completions for text, image, and streaming cases.

Try it yourself

Set up a FastDeploy serving stack in this repo for a Paddle LLM/VLM model. Steps: (1) detect the host OS, Python version, and accelerator, and confirm they meet FastDeploy requirements (Linux, Python 3.10-3.12); (2) install FastDeploy following the matching installation doc for that accelerator, pinning a release tag; (3) add a scripts/serve.sh that launches the OpenAI-compatible server for a chosen model with explicit quantization and max-model-len flags; (4) add a scripts/smoke_test.py that hits /v1/chat/completions with a text prompt, an image prompt, and a streaming request, using an OpenAI-style client pointed at the local base URL; (5) write docs/deploy.md covering launch flags, model revision pin, hardware assumptions, and known gaps versus vLLM. Do not modify existing application code; keep the client change to a base-URL swap.