Chitu single-binary multi-node LLM inference deployment

What they did

The team needed to serve large MoE models (DeepSeek-R1 671B, Qwen3, GLM-4.5) in production but did not want to hand-assemble a serving stack from scratch. They picked Chitu (赤兔), a production-grade inference engine from thu-pacman, and leaned on the v0.6.0 chitu.run executable, which is designed so a single file can launch complex jobs: multi-node, multi-instance, and prefill/decode (PD) disaggregation. They downloaded chitu.run from the GitHub Releases Assets page rather than building from source, then followed the DEVELOPMENT.md manual for full install and usage instructions. They checked SUPPORTED_MODELS.md to confirm their target models were covered, and reviewed the PERFORMANCE.md numbers to set expectations for throughput on their hardware. Because Chitu targets heterogeneous compute, they planned a mixed fleet: NVIDIA GPUs for the bulk of traffic plus domestic accelerators (Ascend 910B, MetaX, Hygon, Moore Threads) where procurement required it. They also noted the CPU+GPU heterogeneous path, which since v0.2.2 allows running DeepSeek-R1 671B on a single card, as a fallback for smaller sites. For quantization they used the FP4-to-FP8/BF16 online conversion operators to run the FP4 DeepSeek-R1 variant without a separate offline conversion step. They wired the engine behind their existing gateway and load balancer, using the multi-instance mode to run several replicas per node and PD separation to scale prefill and decode pools independently as traffic shifted.

How it went

The deployment met the goal of a reproducible, single-artifact launch: one chitu.run file covered multi-node and PD-disaggregated topologies that previously required bespoke orchestration scripts. Model coverage was broad enough that DeepSeek, Qwen, GLM and Kimi workloads ran on the same engine, and the heterogeneous backend support let them reuse existing domestic accelerators instead of standardizing on one vendor. Stability was the stated design target and held up under concurrent business traffic in their testing. Polish is good for a research-lab-origin project: versioned releases, bilingual docs, a DeepWiki index, and a WeChat support group. Limits: performance figures are hardware- and load-dependent and fluctuate between runs, so their own benchmarks were needed before capacity planning.

Worth knowing

The maintainers explicitly say team bandwidth limits timely fixes for all user issues, so production users should expect to self-diagnose or buy commercial support via [email protected]. chitu.run is a prebuilt binary from Releases, so supply-chain review and checksum verification are on you. Exact CLI flags, config file schema, and PD-disaggregation topology settings are not in the README; they live in DEVELOPMENT.md and the FAQ, which you must read before launching. Multi-node setups assume working interconnect and matching driver/toolkit versions per accelerator vendor. Third-party submodules under third_party/ carry their own licenses, so audit before redistribution.

Wiring

Order of operations: (1) pick hardware and confirm the accelerator is supported (NVIDIA multi-generation, Ascend 910B, MetaX, Hygon, Moore Threads) and install the matching driver/toolkit; (2) download chitu.run from the GitHub Releases Assets page for v0.6.0 or later and verify the artifact; (3) confirm the model in docs/zh/SUPPORTED_MODELS.md and decide the weight format, using FP4-to-FP8/BF16 online conversion if running the FP4 DeepSeek-R1 build; (4) write the launch config per DEVELOPMENT.md, starting with a single-node multi-instance profile, then adding multi-node and PD-disaggregation roles (separate prefill and decode pools) as scale grows; (5) for constrained sites, enable the CPU+GPU heterogeneous path to fit 671B-class models on fewer GPUs; (6) front the instances with your gateway/load balancer and point health checks at each instance; (7) run the benchmark harness and compare against PERFORMANCE.md before promoting to production traffic. Config lives in the launch files referenced by DEVELOPMENT.md; the FAQ covers common failure modes.

Try it yourself

Read the Chitu repo (thu-pacman/chitu) and its docs/zh/DEVELOPMENT.md, SUPPORTED_MODELS.md, PERFORMANCE.md and FAQ. Then produce a deployment plan for my environment: (1) confirm my target model is in SUPPORTED_MODELS.md and note any quantization path (FP8, FP4-to-FP8/BF16 online conversion); (2) give exact steps to download chitu.run from the Releases Assets page, verify it, and launch a single-node multi-instance server; (3) show the config for scaling to multi-node with prefill/decode disaggregation, listing every flag and file I must set; (4) list the accelerator-specific prerequisites (NVIDIA vs Ascend 910B vs MetaX/Hygon/Moore Threads) including driver and toolkit versions; (5) propose a benchmark script that measures throughput and TTFT at my expected concurrency so I can validate against PERFORMANCE.md. Flag anything the docs leave ambiguous instead of guessing.