Chitu 단일 바이너리 멀티노드 LLM 추론 배포

# Chitu Deployment Plan

**Important caveat up front:** I can't actually read the repo or those docs in this session — I have no browsing/file access here, and I won't fabricate file contents, flag names, or version numbers. Everything below is a *plan structure* with explicit "verify in docs" markers where I'd otherwise be guessing. Treat every bracketed `[VERIFY]` as something you must confirm against `docs/zh/DEVELOPMENT.md`, `SUPPORTED_MODELS.md`, `PERFORMANCE.md`, and `FAQ` before executing.

---

## 1. Target model confirmation & quantization path

**What to do:**
- Open `SUPPORTED_MODELS.md` and search for your exact model ID (e.g., `DeepSeek-V3`, `Qwen2.5-72B`, `Llama-3.1-70B`). Match on the *exact* string — Chitu typically lists model families with variant suffixes.
- Note the listed **precision/quantization column** for that row. Chitu's docs generally distinguish:
  - Native FP8 checkpoints (loaded as-is)
  - FP4 checkpoints requiring **online FP4→FP8 or FP4→BF16 dequant** at load time
  - BF16/FP16 baseline

**Ambiguities to flag (do not guess):**
- `[VERIFY]` Whether your model appears under a *family* entry vs. an exact checkpoint name — the doc may list `DeepSeek-V3` but not `DeepSeek-V3-0324`.
- `[VERIFY]` Whether FP4→FP8 conversion is automatic on load or requires an explicit flag (e.g., a `--quant` / `--dtype` argument). The docs may state "supported" without specifying the trigger.
- `[VERIFY]` Whether FP4→BF16 is offered as a fallback when FP8 kernels aren't available on your accelerator (Ascend/MetaX paths often differ from NVIDIA here).
- `[VERIFY]` Minimum GPU count / memory per rank for the quantized variant — `SUPPORTED_MODELS.md` may list this, or it may only be in `PERFORMANCE.md`.

**Action:** Record the exact row (model, precision, hardware column) and cross-check it against your accelerator in step 4. If your model is absent, stop — do not assume compatibility.

---

## 2. Download, verify, and launch single-node multi-instance

**Download from Releases:**
1. Go to `https://github.com/thu-pacman/chitu/releases` → latest release → **Assets**.
2. Identify the asset matching your platform. `[VERIFY]` the exact asset naming convention (e.g., `chitu.run`, `chitu-<arch>.run`, or a tarball) — the docs should name it explicitly.
3. Download:
   ```
   curl -L -O <asset-url-from-releases-page>
   ```
4. **Verify integrity** — `[VERIFY]` which method the release provides:
   - If a `SHA256SUMS` asset exists: `sha256sum -c SHA256SUMS --ignore-missing`
   - If GPG signatures are published: `gpg --verify <sig> <asset>`
   - If neither is published, note this as a supply-chain gap and at minimum record the SHA256 you compute for your own audit.
5. Make executable and inspect: `chmod +x chitu.run && ./chitu.run --help` (or `--version`). `[VERIFY]` whether it's a self-extracting installer or a direct binary.

**Single-node multi-instance launch:**
- `[VERIFY]` the exact CLI. Chitu's docs describe launching via a Python entrypoint or the packaged binary; the flag names for **tensor parallel size**, **number of instances per node**, **port/base-port**, and **model path** must come from `DEVELOPMENT.md`.
- Typical shape (illustrative only — confirm flags):
  ```
  ./chitu.run serve \
    --model <path-or-hf-id> \
    --tp <N> \
    --instances <K> \
    --port <base-port> \
    [--quant <fp8|fp4>]
  ```
- `[VERIFY]` How multi-instance is expressed: separate processes with distinct ports, a `--num-instances` flag, or a launcher script. The docs may only show the single-instance case.
- `[VERIFY]` Whether instances share a model cache on disk or each load independently (affects startup time and disk).

---

## 3. Multi-node with prefill/decode disaggregation

**This is the section most likely to be under-specified in the docs — flag aggressively.**

**Files you will likely need to set (confirm each):**
- `[VERIFY]` A **cluster/config file** (YAML/JSON/TOML) describing nodes, roles, and endpoints — `DEVELOPMENT.md` should name it.
- `[VERIFY]` A **role flag** per process: prefill vs. decode vs. hybrid.
- `[VERIFY]` **KV-cache transfer** configuration: transport (RDMA/IB vs. TCP), addresses, and whether a separate transfer service is required.
- `[VERIFY]` **Scheduler/router** config: how requests are split between prefill and decode pools, and the endpoint the client hits.

**Flags to enumerate from docs (do not invent):**
- Node role / disaggregation mode flag
- Prefill pool endpoint(s) and decode pool endpoint(s)
- KV transfer backend + NIC/IB device selection
- Tensor-parallel and pipeline-parallel sizes per role
- Master/coordinator address and port
- Health-check / readiness endpoints

**Ambiguities to flag:**
- `[VERIFY]` Whether disaggregation requires a specific transport (e.g., NVLink-only, or RDMA) and whether it works on Ascend/MetaX at all.
- `[VERIFY]` Whether prefill and decode must run the *same* quantization — mixing FP8 prefill with BF16 decode may or may not be supported.
- `[VERIFY]` Whether the router is bundled in `chitu.run` or a separate component.
- `[VERIFY]` Minimum node count and whether a dedicated coordinator node is needed.

**Recommendation:** If `DEVELOPMENT.md` doesn't fully specify the disaggregation config, open an issue on the repo rather than inferring flag names — a wrong flag will silently fall back to non-disaggregated mode in many frameworks.

---

## 4. Accelerator prerequisites

**NVIDIA:**
- `[VERIFY]` Minimum driver version and CUDA toolkit version from `DEVELOPMENT.md` / `FAQ`.
- `[VERIFY]` Supported GPU generations (Hopper/Blackwell for FP8/FP4 paths; Ampere likely BF16-only).
- `[VERIFY]` Whether FP8 requires specific compute capability (e.g., ≥8.9).

**Ascend 910B:**
- `[VERIFY]` CANN toolkit version, driver/firmware version, and `torch_npu` version — these are almost always pinned and version-sensitive.
- `[VERIFY]` Which quantization paths are supported on 910B (FP8 support on Ascend is generation-dependent).
- `[VERIFY]` Whether disaggregation is supported on Ascend.

**MetaX / Hygon / Moore Threads:**
- `[VERIFY]` Each vendor's driver + toolkit version, and whether Chitu ships prebuilt kernels or requires building from source.
- `[VERIFY]` Whether these are listed as "supported" vs. "experimental" in the docs — the distinction matters for production.

**Cross-cutting ambiguity:** The docs may list a vendor as supported without pinning versions. If so, treat the *newest* documented version as the safe target and record the exact versions you test with.

---

## 5. Benchmark script (throughput + TTFT at your concurrency)

**Design goals:** measure TTFT and output throughput at a sweep of concurrency levels, and compare against `PERFORMANCE.md` numbers for your model/hardware/precision.

**Script outline (Python, using an async HTTP client against the OpenAI-compatible endpoint — `[VERIFY]` that Chitu exposes one):**

```python
import asyncio, time, statistics, aiohttp, argparse

async def one_request(session, url, payload, idx):
    t0 = time.perf_counter()
    ttft = None
    tokens = 0
    async with session.post(url, json=payload) as resp:
        async for line in resp.content:
            if not line.strip():
                continue
            if ttft is None:
                ttft = time.perf_counter() - t0
            tokens += 1  # adjust: count tokens from streamed chunks
    total = time.perf_counter() - t0
    return {"ttft": ttft, "total": total, "tokens": tokens}

async def run(concurrency, url, payload, n_requests):
    async with aiohttp.ClientSession() as s:
        sem = asyncio.Semaphore(concurrency)
        async def bounded(i):
            async with sem:
                return await one_request(s, url, payload, i)
        return await asyncio.gather(*[bounded(i) for i in range(n_requests)])

def summarize(results):
    ttfts = [r["ttft"] for r in results if r["ttft"]]
    toks  = sum(r["tokens"] for r in results)
    wall  = max(r["total"] for r in results)
    return {
        "ttft_p50": statistics.median(ttfts),
        "ttft_p99": sorted(ttfts)[int(len(ttfts)*0.99)-1],
        "throughput_tok_s": toks / wall,
        "n": len(results),
    }

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--url", required=True)
    ap.add_argument("--concurrency", type=int, nargs="+", default=[1,4,8,16,32])
    ap.add_argument("--requests", type=int, default=200)
    ap.add_argument("--prompt-tokens", type=int, default=512)
    ap.add_argument("--max-tokens", type=int, default=256)
    args = ap.parse_args()
    payload = {
        "model": "<your-model>",
        "messages": [{"role":"user","content":"x "*args.prompt_tokens}],
        "max_tokens": args.max_tokens,
        "stream": True,
    }
    for c in args.concurrency:
        res = asyncio.run(run(c, args.url, payload, args.requests))
        print(c, summarize(res))
```

**Validation against `PERFORMANCE.md`:**
- `[VERIFY]` The exact benchmark conditions in `PERFORMANCE.md`: input/output token lengths, batch/concurrency, precision, and hardware. Your numbers are only comparable if these match.
- `[VERIFY]` Whether `PERFORMANCE.md` reports TTFT, TPOT, or end-to-end latency — the metric definitions must align.
- `[VERIFY]` Whether the reported throughput is per-instance or aggregate across instances.

**Ambiguities to flag:**
- `[VERIFY]` Token counting: the script above counts streamed chunks, which may not equal tokens. Use the `usage` field if the API returns it, or a tokenizer.
- `[VERIFY]` Whether the server supports a `/metrics` endpoint (Prometheus) — if so, prefer server-side metrics over client-side timing for throughput.
- `[VERIFY]` Warmup: `PERFORMANCE.md` may or may not include warmup; run a warmup pass before measuring.

---

## Summary of what to confirm before executing

| Item | Source to check | Status |
|---|---|---|
| Exact model ID + quant path | `SUPPORTED_MODELS.md` | `[VERIFY]` |
| Release asset name + checksum method | Releases page | `[VERIFY]` |
| Single-node multi-instance flags | `DEVELOPMENT.md` | `[VERIFY]` |
| Disaggregation config file + flags | `DEVELOPMENT.md` | `[VERIFY]` — likely incomplete |
| Driver/toolkit versions per accelerator | `DEVELOPMENT.md` / `FAQ` | `[VERIFY]` |
| Benchmark conditions in perf doc | `PERFORMANCE.md` | `[VERIFY]` |

If any `[VERIFY]` item is absent from the docs, that's a documentation gap worth filing as an issue — don't fill it in by inference, especially for disaggregation flags and version pins.