OpenLLM과 vLLM을 활용한 자체 호스팅 OpenAI 호환 LLM API

# Self-Hosted OpenAI-Compatible LLM Endpoint with OpenLLM

## 1. Dependencies

**`requirements.txt`** (append):

```txt
openllm>=0.6.0
openai>=1.30.0
```

**`pyproject.toml`** (if you use one, add to `[project]` dependencies):

```toml
dependencies = [
    "openllm>=0.6.0",
    "openai>=1.30.0",
]
```

Install with:

```bash
pip install openllm openai
```

## 2. `scripts/serve_llm.sh`

```bash
#!/usr/bin/env bash
# Serve a Qwen2.5 7B model via OpenLLM's OpenAI-compatible endpoint.
# Requires: HF_TOKEN exported in the environment (never hardcode it).
# Requires: a GPU with ~24G VRAM for qwen2.5:7b.

set -euo pipefail

if [[ -z "${HF_TOKEN:-}" ]]; then
  echo "ERROR: HF_TOKEN is not set. Export it before running this script." >&2
  echo "  export HF_TOKEN=hf_xxx" >&2
  exit 1
fi

export HF_TOKEN

# Optional: override the model via MODEL env var, e.g. MODEL=qwen2.5:14b
MODEL="${MODEL:-qwen2.5:7b}"

exec openllm serve "${MODEL}"
```

Make it executable:

```bash
chmod +x scripts/serve_llm.sh
```

## 3. Python client module

**`llm_client.py`**:

```python
"""OpenAI-compatible chat client.

Uses a self-hosted OpenLLM endpoint when OPENLLM_BASE_URL is set,
otherwise falls back to the hosted OpenAI API.
"""

from __future__ import annotations

import os
from typing import Any

from openai import OpenAI


def _build_client() -> tuple[OpenAI, str]:
    base_url = os.environ.get("OPENLLM_BASE_URL")
    if base_url:
        api_key = os.environ.get("OPENLLM_API_KEY", "not-needed")
        return OpenAI(base_url=base_url, api_key=api_key), "openllm"

    # Fallback: hosted OpenAI API. Requires OPENAI_API_KEY.
    return OpenAI(), "openai"


def chat(
    messages: list[dict[str, Any]],
    model: str | None = None,
    **kwargs: Any,
) -> str:
    """Send a chat completion request and return the assistant message text."""
    client, backend = _build_client()

    if model is None:
        if backend == "openllm":
            model = os.environ.get("OPENLLM_MODEL", "qwen2.5:7b")
        else:
            model = os.environ.get("OPENAI_MODEL", "gpt-4o-mini")

    response = client.chat.completions.create(
        model=model,
        messages=messages,
        **kwargs,
    )
    return response.choices[0].message.content or ""


if __name__ == "__main__":
    print(chat([{"role": "user", "content": "Say hello in one sentence."}]))
```

Usage:

```bash
# Self-hosted
export OPENLLM_BASE_URL=http://localhost:3000/v1
export OPENLLM_API_KEY=local-dev-key   # any non-empty string
python llm_client.py

# Hosted fallback
unset OPENLLM_BASE_URL
export OPENAI_API_KEY=sk-...
python llm_client.py
```

## 4. `Dockerfile`

```dockerfile
# OpenLLM server image.
#
# GPU REQUIREMENT: qwen2.5:7b needs roughly 24G of GPU VRAM.
# Run with: docker run --gpus all -p 3000:3000 -e HF_TOKEN=$HF_TOKEN <image>
FROM python:3.11-slim

ENV PYTHONUNBUFFERED=1 \
    PIP_NO_CACHE_DIR=1

WORKDIR /app

RUN pip install --upgrade pip && \
    pip install "openllm>=0.6.0"

# HF_TOKEN is injected at runtime via -e HF_TOKEN=... (never baked into the image).
ENV MODEL=qwen2.5:7b

EXPOSE 3000

CMD ["sh", "-c", "openllm serve ${MODEL}"]
```

Build and run:

```bash
docker build -t openllm-server .
docker run --gpus all -p 3000:3000 -e HF_TOKEN="$HF_TOKEN" openllm-server
```

## 5. README section

````markdown
## Self-Hosted LLM Endpoint (OpenLLM)

This repo can serve an OpenAI-compatible LLM endpoint locally using
[OpenLLM](https://github.com/bentoml/OpenLLM). The Python client in
`llm_client.py` talks to it automatically when `OPENLLM_BASE_URL` is set,
and falls back to the hosted OpenAI API otherwise.

### Prerequisites

- Python 3.10+
- A CUDA-capable GPU. **`qwen2.5:7b` requires ~24G of GPU VRAM.**
  Smaller models (e.g. `qwen2.5:1.5b`) fit on smaller cards.
- A Hugging Face account and access token.

### 1. Install

```bash
pip install openllm openai
```

### 2. Set up `HF_TOKEN`

Create a token at <https://huggingface.co/settings/tokens> and export it
in your shell. **Never commit tokens to the repo.**

```bash
export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxx
```

The serve script and Dockerfile both read `HF_TOKEN` from the environment
at runtime; nothing is hardcoded.

### 3. Start the server

```bash
./scripts/serve_llm.sh
```

This runs `openllm serve qwen2.5:7b` and exposes an OpenAI-compatible API
at `http://localhost:3000/v1`.

Or with Docker:

```bash
docker build -t openllm-server .
docker run --gpus all -p 3000:3000 -e HF_TOKEN="$HF_TOKEN" openllm-server
```

### 4. Point the client at it

```bash
export OPENLLM_BASE_URL=http://localhost:3000/v1
export OPENLLM_API_KEY=local-dev-key   # any non-empty string
python llm_client.py
```

If `OPENLLM_BASE_URL` is unset, `llm_client.py` falls back to the hosted
OpenAI API and uses `OPENAI_API_KEY`.

### 5. Switching models

Pick any model supported by OpenLLM. Two ways:

- **Script:** set the `MODEL` env var before running the script:

  ```bash
  MODEL=qwen2.5:14b ./scripts/serve_llm.sh
  ```

- **Docker:** override the `MODEL` env var at runtime:

  ```bash
  docker run --gpus all -p 3000:3000 \
    -e HF_TOKEN="$HF_TOKEN" \
    -e MODEL=qwen2.5:14b \
    openllm-server
  ```

Then update the client's model name if needed:

```bash
export OPENLLM_MODEL=qwen2.5:14b
```

Check VRAM requirements for larger models before switching — 14B and
above typically need 40G+ of GPU memory.
````