LlamaIndex RAG 앱을 위한 LLM 관측성 및 평가 루프: Langfuse 트레이스와 promptfoo 테스트 활용

# LLM 파이프라인 관측성 및 평가 루프 설정

요청하신 작업을 진행하기 전에, 현재 저장소의 구조를 확인할 수 없어 일반적인 패턴을 기준으로 작성했습니다. 실제 파일 경로와 LLM 호출 위치는 저장소에 맞게 조정이 필요합니다.

---

## 1. Langfuse 트레이싱 추가

### `package.json` (의존성 추가)

```diff
 {
   "dependencies": {
+    "langfuse": "^3.0.0",
     ...
   }
 }
```

### `src/observability/langfuse.ts` (신규 파일)

```diff
+import { Langfuse } from "langfuse";
+
+if (!process.env.LANGFUSE_PUBLIC_KEY || !process.env.LANGFUSE_SECRET_KEY) {
+  console.warn(
+    "[langfuse] LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY not set; tracing disabled."
+  );
+}
+
+export const langfuse = new Langfuse({
+  publicKey: process.env.LANGFUSE_PUBLIC_KEY,
+  secretKey: process.env.LANGFUSE_SECRET_KEY,
+  baseUrl: process.env.LANGFUSE_HOST ?? "https://cloud.langfuse.com",
+  flushAt: 1,
+});
+
+export async function flushLangfuse() {
+  await langfuse.flushAsync();
+}
```

### `src/pipeline/llm.ts` (메인 LLM 호출 + 검색 단계 래핑)

```diff
+import { langfuse } from "../observability/langfuse";
+
 export async function runPipeline(userInput: string) {
+  const trace = langfuse.trace({
+    name: "llm-pipeline",
+    input: userInput,
+  });
+
+  // --- Retrieval span ---
+  const retrievalSpan = trace.span({
+    name: "retrieval",
+    input: { query: userInput },
+  });
+  const retrievedDocs = await retrieveDocuments(userInput);
+  retrievalSpan.end({
+    output: retrievedDocs.map((d) => ({ id: d.id, score: d.score })),
+  });
+
+  // --- LLM generation span ---
+  const generation = trace.generation({
+    name: "llm-call",
+    model: process.env.LLM_MODEL ?? "gpt-4o-mini",
+    input: { userInput, context: retrievedDocs.map((d) => d.text) },
+  });
+
+  const start = Date.now();
+  const result = await callLLM(userInput, retrievedDocs);
+  const latencyMs = Date.now() - start;
+
+  generation.end({
+    output: result.text,
+    usage: {
+      input: result.usage?.promptTokens,
+      output: result.usage?.completionTokens,
+      total: result.usage?.totalTokens,
+    },
+    metadata: { latencyMs },
+  });
+
+  trace.update({ output: result.text });
+  return result;
 }
```

### LlamaIndex를 사용하는 경우

수동 span 대신 공식 통합을 사용하세요.

```diff
+import { Langfuse } from "langfuse";
+import { LlamaIndexInstrumentation } from "langfuse-llamaindex";
+
+const langfuse = new Langfuse({
+  publicKey: process.env.LANGFUSE_PUBLIC_KEY,
+  secretKey: process.env.LANGFUSE_SECRET_KEY,
+  baseUrl: process.env.LANGFUSE_HOST,
+});
+
+const instrumentation = new LlamaIndexInstrumentation({ langfuse });
+instrumentation.observe();
```

이렇게 하면 LlamaIndex의 retrieval, query engine, LLM 호출이 자동으로 trace/span으로 기록됩니다.

---

## 2. promptfoo 설정

### `promptfooconfig.yaml` (신규 파일)

```diff
+description: "LLM pipeline regression tests"
+
+prompts:
+  - file://prompts/main.txt
+
+providers:
+  - id: openai:gpt-4o-mini
+    config:
+      temperature: 0
+
+tests:
+  - vars:
+      question: "How do I install runchr on macOS?"
+    assert:
+      - type: llm-rubric
+        value: "The answer explains the macOS installation steps clearly."
+      - type: contains-any
+        value: ["npm install", "brew", "install"]
+      - type: javascript
+        value: "output.length < 1200"
+
+  - vars:
+      question: "What is the difference between MCP and RAG?"
+    assert:
+      - type: llm-rubric
+        value: "The answer correctly distinguishes MCP (tool/context protocol) from RAG (retrieval-augmented generation)."
+      - type: contains-any
+        value: ["MCP", "RAG"]
+      - type: javascript
+        value: "output.length < 1500"
+
+  - vars:
+      question: "Show me how to configure LANGFUSE_HOST."
+    assert:
+      - type: contains
+        value: "LANGFUSE_HOST"
+      - type: llm-rubric
+        value: "The answer includes a concrete example of setting LANGFUSE_HOST."
+      - type: javascript
+        value: "output.length < 1000"
+
+  - vars:
+      question: "Which LLM providers does runchr support?"
+    assert:
+      - type: llm-rubric
+        value: "The answer lists at least one supported provider and does not hallucinate unsupported ones."
+      - type: javascript
+        value: "output.length < 1200"
+
+  - vars:
+      question: "How do I run the eval suite in CI?"
+    assert:
+      - type: contains-any
+        value: ["promptfoo", "npm run eval"]
+      - type: llm-rubric
+        value: "The answer references promptfoo or the eval npm script."
+      - type: javascript
+        value: "output.length < 1000"
+
+  - vars:
+      question: "What env vars are required for Langfuse?"
+    assert:
+      - type: contains
+        value: "LANGFUSE_PUBLIC_KEY"
+      - type: contains
+        value: "LANGFUSE_SECRET_KEY"
+      - type: javascript
+        value: "output.length < 800"
+
+  - vars:
+      question: "How do I add a new test case to promptfoo?"
+    assert:
+      - type: contains
+        value: "promptfooconfig.yaml"
+      - type: llm-rubric
+        value: "The answer explains adding a test entry with vars and assert."
+      - type: javascript
+        value: "output.length < 1200"
+
+  - vars:
+      question: "Does runchr support TypeScript and Python?"
+    assert:
+      - type: contains
+        value: "TypeScript"
+      - type: contains
+        value: "Python"
+      - type: javascript
+        value: "output.length < 600"
+
+  - vars:
+      question: "What happens if LANGFUSE keys are missing?"
+    assert:
+      - type: llm-rubric
+        value: "The answer states that tracing is disabled or a warning is logged, not that the app crashes."
+      - type: javascript
+        value: "output.length < 800"
+
+  - vars:
+      question: "How do I view traces in Langfuse?"
+    assert:
+      - type: contains-any
+        value: ["Langfuse UI", "dashboard", "trace"]
+      - type: llm-rubric
+        value: "The answer points to the Langfuse dashboard or trace view."
+      - type: javascript
+        value: "output.length < 900"
+
+  - vars:
+      question: "What is the recommended temperature for eval runs?"
+    assert:
+      - type: contains
+        value: "0"
+      - type: llm-rubric
+        value: "The answer recommends temperature 0 for deterministic eval runs."
+      - type: javascript
+        value: "output.length < 600"
+
+  - vars:
+      question: "How do I fail CI when pass rate drops?"
+    assert:
+      - type: contains-any
+        value: ["threshold", "pass rate", "fail"]
+      - type: llm-rubric
+        value: "The answer mentions a pass-rate threshold that fails the build."
+      - type: javascript
+        value: "output.length < 1000"
```

> 참고: `llm-rubric`는 답변 관련성(relevance)을, `contains`/`contains-any`는 인용·출처 존재 여부를, `javascript`는 최대 길이를 검증합니다. 실제 파이프라인 프롬프트 파일 경로(`prompts/main.txt`)는 저장소에 맞게 조정하세요.

---

## 3. CI용 npm 스크립트

### `package.json` (스크립트 추가)

```diff
 {
   "scripts": {
+    "eval": "promptfoo eval --config promptfooconfig.yaml --output results.json",
+    "eval:ci": "npm run eval && node scripts/check-pass-rate.js",
     ...
   },
   "devDependencies": {
+    "promptfoo": "^0.95.0",
     ...
   }
 }
```

### `scripts/check-pass-rate.js` (신규 파일)

```diff
+const fs = require("fs");
+
+const BASELINE = Number(process.env.EVAL_BASELINE_PASS_RATE ?? 0.9);
+const results = JSON.parse(fs.readFileSync("results.json", "utf8"));
+
+const stats = results.results?.stats ?? results.stats;
+const passRate = stats?.successes / (stats?.successes + stats?.failures);
+
+if (!Number.isFinite(passRate)) {
+  console.error("Could not compute pass rate from results.json");
+  process.exit(1);
+}
+
+console.log(`Pass rate: ${(passRate * 100).toFixed(1)}% (baseline ${(BASELINE * 100).toFixed(1)}%)`);
+
+if (passRate < BASELINE) {
+  console.error("Pass rate below baseline — failing CI.");
+  process.exit(1);
+}
```

### `.github/workflows/eval.yml` (신규 파일)

```diff
+name: LLM Eval
+
+on:
+  pull_request:
+    branches: [main]
+
+jobs:
+  eval:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+      - run: npm ci
+      - run: npm run eval:ci
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          EVAL_BASELINE_PASS_RATE: "0.9"
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: promptfoo-results
+          path: results.json
```

---

## 필요한 환경 변수 및 서비스

**환경 변수**
- `LANGFUSE_PUBLIC_KEY` — Langfuse 프로젝트 공개 키
- `LANGFUSE_SECRET_KEY` — Langfuse 시크릿 키
- `LANGFUSE_HOST` — Langfuse 호스트 (예: `https://cloud.langfuse.com` 또는 self-hosted URL)
- `LLM_MODEL` — 사용 중인 모델 식별자 (선택)
- `OPENAI_API_KEY` — promptfoo provider용 (또는 사용 중인 provider 키)
- `EVAL_BASELINE_PASS_RATE` — CI 통과 기준선 (기본 0.9)

**프로비저닝할 서비스**
- **Langfuse**: 클라우드 계정 또는 self-hosted 인스턴스 (Docker Compose로 Postgres + ClickHouse + Redis 필요)
- **LLM provider**: OpenAI 등 실제 파이프라인에서 쓰는 provider의 API 키
- **GitHub Actions secrets**: `OPENAI_API_KEY` 등 CI에서 필요한 키 등록

**확인 필요 사항**
- 파이프라인이 LlamaIndex 기반인지 여부 (기반이면 수동 span 대신 `langfuse-llamaindex` 통합 사용)
- 실제 LLM 호출 함수명(`callLLM`)과 검색 함수명(`retrieveDocuments`) — 저장소 구조에 맞게 조정
- 프롬프트 파일 경로(`prompts/main.txt`)