LMDeploy TurboMind와 vLLM 비교를 포함한 자체 호스팅 LLM 추론 서비스

# Self-Hosted OpenAI-Compatible LLM Endpoint: Setup & Benchmark

## 1. Repo Inspection Findings

I don't have direct access to your repository contents in this conversation, so I can't enumerate actual files. Below is what to look for and how to report it — run these checks and paste the output back if you want me to tailor the scripts to what's actually there.

**What to search for:**

```bash
# Existing model-serving code
grep -rIl --include='*.py' -E 'vllm|lmdeploy|text-generation-inference|tgi|ollama|llama.cpp|sglang' .

# Docker compose / container definitions
find . -maxdepth 3 \( -name 'docker-compose*.yml' -o -name 'docker-compose*.yaml' -o -name 'Dockerfile*' -o -name 'compose*.yml' \)

# Client code hitting an OpenAI-style API
grep -rIl --include='*.py' --include='*.ts' --include='*.js' -E 'openai|chat/completions|base_url|OPENAI_BASE_URL|OPENAI_API_BASE' .

# Env / config that may already define a base URL
grep -rIn -E 'OPENAI_(BASE_URL|API_BASE|API_KEY)|LLM_(BASE_URL|ENDPOINT)' . --include='*.env*' --include='*.yml' --include='*.yaml' --include='*.toml' --include='*.json'
```

**Typical findings to report back:**
- **Model-serving code:** none / `vllm` / `lmdeploy` / `ollama` / `llama.cpp` / `sglang` (note file paths).
- **Compose files:** list paths; note whether any service already exposes an OpenAI-compatible port (8000, 23333, 11434, etc.).
- **Client code:** list files that construct an OpenAI client; note whether `base_url` is hardcoded or read from env. This determines how much of step 5's "switch base URL" work is already done.
- **Existing env vars:** any `OPENAI_BASE_URL` / `LLM_ENDPOINT` already in use — reuse those names in the scripts below rather than inventing new ones.

The scripts below assume **no existing serving code** and a **Python client using the `openai` SDK with a configurable `base_url`**. Adjust if the inspection shows otherwise.

---

## 2. `scripts/serve_lmdeploy.sh`

```bash
#!/usr/bin/env bash
# Convert a HF model to TurboMind format and launch lmdeploy api_server.
set -euo pipefail

# ---- Configurable via env (sane defaults) ----
MODEL_PATH="${MODEL_PATH:-Qwen/Qwen2.5-7B-Instruct}"   # HF repo id or local path
WORK_DIR="${WORK_DIR:-./.lmdeploy}"                    # converted weights live here
PORT="${LMDEPLOY_PORT:-23333}"
HOST="${LMDEPLOY_HOST:-0.0.0.0}"
TP="${TP:-1}"                                          # tensor parallel size
SESSION_LEN="${SESSION_LEN:-8192}"
CACHE_MAX_ENTRY_COUNT="${CACHE_MAX_ENTRY_COUNT:-64}"
QUANT_POLICY="${QUANT_POLICY:-}"                       # e.g. awq, w4a16, or empty
MODEL_NAME="${MODEL_NAME:-local-llm}"                  # name clients see in /v1/models
VENV_DIR="${VENV_DIR:-./.venv-lmdeploy}"

# ---- Install ----
if [[ ! -d "$VENV_DIR" ]]; then
  python3 -m venv "$VENV_DIR"
fi
# shellcheck disable=SC1091
source "$VENV_DIR/bin/activate"
pip install --upgrade pip
pip install "lmdeploy>=0.6"

# ---- Convert HF -> TurboMind ----
CONVERT_ARGS=( "$MODEL_PATH" "$WORK_DIR" --model-name "$MODEL_NAME" )
if [[ -n "$QUANT_POLICY" ]]; then
  CONVERT_ARGS+=( --model-format "$QUANT_POLICY" )
fi
if [[ ! -f "$WORK_DIR/model_weights" && ! -d "$WORK_DIR/turbomind" ]]; then
  echo "[lmdeploy] converting $MODEL_PATH -> $WORK_DIR"
  lmdeploy convert "${CONVERT_ARGS[@]}"
else
  echo "[lmdeploy] converted weights already present at $WORK_DIR, skipping convert"
fi

# ---- Launch api_server ----
echo "[lmdeploy] serving on http://$HOST:$PORT/v1 (model=$MODEL_NAME)"
exec lmdeploy serve api_server "$WORK_DIR" \
  --model-name "$MODEL_NAME" \
  --server-name "$HOST" \
  --server-port "$PORT" \
  --tp "$TP" \
  --session-len "$SESSION_LEN" \
  --cache-max-entry-count "$CACHE_MAX_ENTRY_COUNT"
```

Notes:
- `lmdeploy convert` writes a TurboMind workspace; `api_server` then serves it. If you prefer to skip conversion and let lmdeploy auto-convert on first launch, drop the `convert` block and pass `$MODEL_PATH` directly to `api_server`.
- `QUANT_POLICY` maps to lmdeploy's `--model-format` (e.g. `awq`, `w4a16`). Leave empty for FP16/BF16.

---

## 3. `scripts/serve_vllm.sh`

```bash
#!/usr/bin/env bash
# Launch vLLM's OpenAI-compatible server on the same weights, different port.
set -euo pipefail

MODEL_PATH="${MODEL_PATH:-Qwen/Qwen2.5-7B-Instruct}"
PORT="${VLLM_PORT:-8000}"
HOST="${VLLM_HOST:-0.0.0.0}"
TP="${TP:-1}"
MAX_MODEL_LEN="${MAX_MODEL_LEN:-8192}"
GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.90}"
QUANTIZATION="${QUANTIZATION:-}"                       # e.g. awq, gptq, fp8
DTYPE="${DTYPE:-auto}"
SERVED_NAME="${SERVED_NAME:-local-llm}"
VENV_DIR="${VENV_DIR:-./.venv-vllm}"

if [[ ! -d "$VENV_DIR" ]]; then
  python3 -m venv "$VENV_DIR"
fi
# shellcheck disable=SC1091
source "$VENV_DIR/bin/activate"
pip install --upgrade pip
pip install "vllm>=0.6"

ARGS=( "$MODEL_PATH"
  --host "$HOST"
  --port "$PORT"
  --tensor-parallel-size "$TP"
  --max-model-len "$MAX_MODEL_LEN"
  --gpu-memory-utilization "$GPU_MEM_UTIL"
  --dtype "$DTYPE"
  --served-model-name "$SERVED_NAME"
)
if [[ -n "$QUANTIZATION" ]]; then
  ARGS+=( --quantization "$QUANTIZATION" )
fi

echo "[vllm] serving on http://$HOST:$PORT/v1 (model=$SERVED_NAME)"
exec vllm serve "${ARGS[@]}"
```

---

## 4. `scripts/bench_openai.py`

```python
#!/usr/bin/env python3
"""Benchmark an OpenAI-compatible streaming chat endpoint.

Reports TTFT, output tokens/sec, and p95 end-to-end latency as a table.
"""
from __future__ import annotations

import argparse
import asyncio
import os
import statistics
import time
from dataclasses import dataclass, field

from openai import AsyncOpenAI


@dataclass
class Result:
    ttft: float | None = None
    total: float | None = None
    out_tokens: int = 0
    error: str | None = None

    @property
    def tps(self) -> float | None:
        if self.ttft is None or self.total is None or self.out_tokens == 0:
            return None
        gen_time = self.total - self.ttft
        return self.out_tokens / gen_time if gen_time > 0 else None


async def one_request(client: AsyncOpenAI, model: str, prompt: str, max_tokens: int) -> Result:
    r = Result()
    start = time.perf_counter()
    try:
        stream = await client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            max_tokens=max_tokens,
            stream=True,
            temperature=0.0,
        )
        async for chunk in stream:
            if not chunk.choices:
                continue
            delta = chunk.choices[0].delta
            if delta and delta.content:
                if r.ttft is None:
                    r.ttft = time.perf_counter() - start
                r.out_tokens += 1  # approximation: one chunk ~ one token
        r.total = time.perf_counter() - start
    except Exception as e:  # noqa: BLE001
        r.error = f"{type(e).__name__}: {e}"
    return r


def pct(values: list[float], p: float) -> float:
    if not values:
        return float("nan")
    s = sorted(values)
    k = max(0, min(len(s) - 1, int(round((p / 100) * (len(s) - 1)))))
    return s[k]


def fmt(x: float | None, unit: str = "") -> str:
    return "—" if x is None or x != x else f"{x:.3f}{unit}"


async def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--base-url", default=os.getenv("OPENAI_BASE_URL", "http://localhost:8000/v1"))
    ap.add_argument("--api-key", default=os.getenv("OPENAI_API_KEY", "EMPTY"))
    ap.add_argument("--model", default=os.getenv("MODEL_NAME", "local-llm"))
    ap.add_argument("--concurrency", type=int, default=int(os.getenv("BENCH_CONCURRENCY", "8")))
    ap.add_argument("--requests", type=int, default=int(os.getenv("BENCH_REQUESTS", "32")))
    ap.add_argument("--max-tokens", type=int, default=int(os.getenv("BENCH_MAX_TOKENS", "256")))
    ap.add_argument("--prompt", default=os.getenv("BENCH_PROMPT", "Write a short paragraph about distributed systems."))
    args = ap.parse_args()

    client = AsyncOpenAI(base_url=args.base_url, api_key=args.api_key)
    sem = asyncio.Semaphore(args.concurrency)

    async def guarded() -> Result:
        async with sem:
            return await one_request(client, args.model, args.prompt, args.max_tokens)

    t0 = time.perf_counter()
    results = await asyncio.gather(*(guarded() for _ in range(args.requests)))
    wall = time.perf_counter() - t0

    ok = [r for r in results if r.error is None]
    errs = [r for r in results if r.error is not None]

    ttfts = [r.ttft for r in ok if r.ttft is not None]
    tpss = [r.tps for r in ok if r.tps is not None]
    totals = [r.total for r in ok if r.total is not None]

    print(f"\nEndpoint : {args.base_url}")
    print(f"Model    : {args.model}")
    print(f"Requests : {args.requests}  Concurrency: {args.concurrency}  max_tokens: {args.max_tokens}")
    print(f"Wall time: {wall:.2f}s   OK: {len(ok)}   Errors: {len(errs)}\n")

    rows = [
        ("TTFT (s)",        fmt(statistics.mean(ttfts) if ttfts else None),
                            fmt(pct(ttfts, 95))),
        ("Output tok/s",    fmt(statistics.mean(tpss) if tpss else None),
                            fmt(pct(tpss, 95))),
        ("Latency (s)",     fmt(statistics.mean(totals) if totals else None),
                            fmt(pct(totals, 95))),
    ]
    w = max(len(r[0]) for r in rows) + 2
    print(f"{'Metric'.ljust(w)}{'mean'.rjust(12)}{'p95'.rjust(12)}")
    print("-" * (w + 24))
    for name, mean, p95 in rows:
        print(f"{name.ljust(w)}{mean.rjust(12)}{p95.rjust(12)}")

    if errs:
        print("\nSample errors:")
        for e in errs[:3]:
            print(f"  - {e.error}")


if __name__ == "__main__":
    asyncio.run(main())
```

Usage:

```bash
pip install openai
python scripts/bench_openai.py --base-url http://localhost:23333/v1 --concurrency 16 --requests 64
python scripts/bench_openai.py --base-url http://localhost:8000/v1  --concurrency 16 --requests 64
```

---

## 5. systemd unit + README section

### `deploy/[email protected]`

```ini
[Unit]
Description=Self-hosted OpenAI-compatible LLM endpoint (%i)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=llm
WorkingDirectory=/opt/llm-endpoint
EnvironmentFile=/etc/llm-endpoint/%i.env
ExecStart=/opt/llm-endpoint/scripts/serve_%i.sh
Restart=on-failure
RestartSec=5
# GPU access
Environment=CUDA_VISIBLE_DEVICES=0
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
```

`/etc/llm-endpoint/lmdeploy.env`:

```ini
MODEL_PATH=Qwen/Qwen2.5-7B-Instruct
LMDEPLOY_PORT=23333
TP=1
SESSION_LEN=8192
QUANT_POLICY=
MODEL_NAME=local-llm
```

`/etc/llm-endpoint/vllm.env`:

```ini
MODEL_PATH=Qwen/Qwen2.5-7B-Instruct
VLLM_PORT=8000
TP=1
MAX_MODEL_LEN=8192
GPU_MEM_UTIL=0.90
QUANTIZATION=
SERVED_NAME=local-llm
```

Enable:

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now llm-endpoint@lmdeploy
# or
sudo systemctl enable --now llm-endpoint@vllm
```

### README section

```markdown
## Self-hosted LLM endpoint

Two interchangeable OpenAI-compatible servers are provided. Both read the
same `MODEL_PATH` and expose `/v1/chat/completions`.

| Engine    | Script                     | Default port | Env prefix   |
|-----------|----------------------------|--------------|--------------|
| lmdeploy  | `scripts/serve_lmdeploy.sh`| 23333        | `LMDEPLOY_*` |
| vLLM      | `scripts/serve_vllm.sh`    | 8000         | `VLLM_*`     |

### GPU memory sizing

Rough VRAM budget for weights + KV cache:

- **Weights:** `params × bytes_per_param`. FP16 ≈ 2 B/param, INT8 ≈ 1 B, INT4 ≈ 0.5 B.
  A 7B model is ~14 GB FP16, ~7 GB INT8, ~3.5 GB INT4.
- **KV cache:** `2 × layers × kv_heads × head_dim × seq_len × batch × bytes`.
  For a 7B model at 8k context and batch 8, budget ~4–8 GB.
- **Rule of thumb:** `VRAM ≈ weights × 1.2 + KV cache + 1–2 GB overhead`.
  Set `GPU_MEM_UTIL` (vLLM) or `CACHE_MAX_ENTRY_COUNT` (lmdeploy) to leave headroom.

### Quantization flags

- **lmdeploy:** `QUANT_POLICY=awq` or `w4a16` (passed to `lmdeploy convert --model-format`).
- **vLLM:** `QUANTIZATION=awq|gptq|fp8` (passed to `vllm serve --quantization`).
  Leave empty for BF16/FP16. Quantized weights must match the flag — don't
  point `QUANTIZATION=awq` at an FP16 checkpoint.

### Switching the app's base URL

The client should read `OPENAI_BASE_URL` (and `OPENAI_API_KEY=EMPTY`) from env:

```bash
# lmdeploy
export OPENAI_BASE_URL=http://localhost:23333/v1
# vLLM
export OPENAI_BASE_URL=http://localhost:8000/v1
```

If the app hardcodes a base URL, replace it with `os.getenv("OPENAI_BASE_URL", "http://localhost:8000/v1")`
(or the TS equivalent `process.env.OPENAI_BASE_URL ?? "http://localhost:8000/v1"`).
The `MODEL_NAME` / `SERVED_NAME` values must match what the client sends in
`model=` — both scripts default to `local-llm`.

### Benchmarking

```bash
python scripts/bench_openai.py --base-url "$OPENAI_BASE_URL" \
  --concurrency 16 --requests 64 --max-tokens 256
```

Prints mean and p95 for TTFT, output tokens/sec, and end-to-end latency.
```

---

**Open items to confirm from step 1:** the actual client file(s) that need their `base_url` swapped, whether an `OPENAI_BASE_URL` env var already exists (reuse it), and whether any existing compose file should be extended rather than replaced by systemd.