Ascend NPU inference stack for vLLM serving

What they did

They set up vLLM inference on Huawei Ascend hardware using the vllm-ascend plugin, which extends upstream vLLM with Ascend NPU device support. The work followed the official documentation and support matrix to pick a compatible release (e.g. v0.23.0 stable or v0.26.0rc1 release candidate), installed the plugin alongside a matching vLLM version, and validated that the Ascend device backend was detected. They then launched an OpenAI-compatible server to serve a model, and for larger deployments configured multi-device and large-scale Expert Parallelism (EP) following the dedicated EP tutorial. They used the support matrix to confirm which models and features were covered, and the DeepWiki/forum channels to resolve gaps.

How it went

A working Ascend-backed vLLM serving endpoint with OpenAI-compatible APIs, plus a path to scale out with expert parallelism. Release cadence is regular (roughly quarterly official versions plus RCs), and docs are versioned per release, so the setup is reproducible. Polish is good for supported model/hardware combos; results outside the support matrix are unverified.

Worth knowing

Requires Ascend NPU hardware and a CANN/driver stack, so it cannot be tried on a laptop or plain GPU box. Plugin and vLLM versions must match exactly; mismatches are a common failure. Model coverage is limited to the support matrix, and some features lag upstream vLLM. The README excerpt is mostly release news, so install specifics must come from the versioned docs.

Wiring

Order: (1) confirm Ascend hardware and CANN/driver prerequisites; (2) pick a release from the news list and read its versioned docs; (3) check the support matrix for model and feature coverage; (4) install vLLM at the pinned version, then the matching vllm-ascend plugin; (5) verify device detection; (6) launch the OpenAI-compatible server; (7) for scale-out, follow the large-scale EP tutorial and set tensor/expert parallel sizes across devices. Config lives in launch flags and env vars; version pinning between vLLM and the plugin is the critical coupling.

Try it yourself

Set up vLLM on this repo's Ascend NPU target. Read the versioned docs for the release pinned in this repo, check the support matrix for my model, install the matching vllm and vllm-ascend versions, verify the NPU device is detected, then start an OpenAI-compatible server and run a smoke-test completion. Report exact versions, any CANN/driver prerequisites, and anything outside the support matrix.