Golden-question eval suite for a chat agent, run in-process and against Vercel previews

What they did

The author added an `api/evals/` directory containing a fixed golden-question suite for the app's chat agent, closing out known-issue #14 ("no automated tests") in `docs/ARCHITECTURE.md` — scoped narrowly to the agent rather than the whole app. Each question is scored on three dimensions: tool-selection correctness (expected tools must be a subset of the tools actually called, deterministic), forbidden phrasing (no false "I've added" claims and no direct buy/sell directives, also deterministic), and groundedness, which reuses the real `"groundedness"` SSE event that `run_agent_turn()` already emits instead of firing a second judge call. They built two run modes: an in-process mode (`python evals/run_evals.py`) that calls `run_agent_turn()` directly — fast and auth-free — and an HTTP mode (`--base-url`) that mints a token via a new reusable `evals/auth_helper.py` and hits the real `POST /agent/chat` endpoint, parsing SSE the same way the frontend's `streamAgentChat()` does. HTTP mode exists because the app has documented local/prod divergence history (an MCP subprocess resolution issue, SSE "not verified in production") that an in-process check cannot catch. They added `.github/workflows/run_evals_on_preview.yml`, triggered on `deployment_status` rather than `pull_request` so it receives a real, already-built Vercel preview URL instead of guessing one or racing an unready deployment; since the PR number isn't on a `deployment_status` event, a separate step resolves it through GitHub's commit→PR API. Results post as a PR comment via `gh pr comment` (the same mechanism `claude-code-review.yml` uses) and are advisory only (`continue-on-error: true`), never blocking a merge. Docs were updated in `CLAUDE.md` (Commands block plus a new §7 item) and `docs/ARCHITECTURE.md` (new explanatory section, known-issue #14 marked partial progress rather than resolved).

How it went

All 10 cases passed in-process against the real agent, with genuine groundedness variance from 67% to 100% — evidence it's a real signal rather than a rubber stamp. The same suite passed 10/10 in HTTP mode against the local dev server at `--base-url http://127.0.0.1:8000`, confirming the token-minting and SSE-parsing path works end-to-end. The author independently verified each of the three scoring functions' failure paths (wrong expected tool, forbidden phrase present, groundedness below threshold) returns a failing result, and confirmed the CLI exits non-zero on a deliberately wrong `expected_tools` entry. `auth_helper.mint_token()` output was checked directly against `api/app/auth.py`'s real `verify_token()`. All new Python files were syntax-checked and the JSON and workflow YAML validated. The one open item: the workflow had not yet run in real CI — this PR's own preview deployment was to be its first real trigger.

Worth knowing

The CI path is unproven: the PR itself is the first real trigger of `run_evals_on_preview.yml`, including whether the `environment == 'Preview'` gate actually matches what Vercel's integration reports for this repo — flagged in the plan as something to verify on first trigger, not assumed. Three new GitHub Actions secrets are required (`AUTH_SECRET`, `ALLOWED_USER_EMAIL`, plus the pre-existing `ANTHROPIC_API_KEY`), added ahead of the PR with user confirmation. Evals are advisory only and never block merges, so regressions can land silently. The suite is 10 fixed questions scoped to the agent, not a general test harness, and known-issue #14 is only partially addressed.

Wiring

Order: (1) `api/evals/golden_questions.json` holds the fixed cases. (2) `api/evals/auth_helper.py` exposes `mint_token()`, whose signing scheme must match `api/app/auth.py`'s `verify_token()`. (3) `api/evals/run_evals.py` is the CLI: default in-process mode imports and calls `run_agent_turn()`; `--base-url` switches to HTTP mode, minting a token, calling `POST /agent/chat`, and parsing SSE identically to the frontend's `streamAgentChat()`. Scoring reads the `"groundedness"` SSE event already emitted by `run_agent_turn()` — no extra judge call. (4) `.github/workflows/run_evals_on_preview.yml` triggers on `deployment_status`, gates on `environment == 'Preview'`, resolves the PR number via GitHub's commit→PR API, runs the CLI in HTTP mode against the preview URL, and posts a PR comment with `gh pr comment` under `continue-on-error: true`. Secrets needed: `AUTH_SECRET`, `ALLOWED_USER_EMAIL`, `ANTHROPIC_API_KEY`. Docs touched: `CLAUDE.md` (Commands block, §7) and `docs/ARCHITECTURE.md` (new section, known-issue #14).

Try it yourself

Add a golden-question eval suite for our chat agent, modeled on the one in himadric/ai-stock-analyst. Create `api/evals/` with a `golden_questions.json` of ~10 fixed cases, each specifying the question, `expected_tools`, forbidden phrases, and a groundedness threshold. Write `run_evals.py` with two modes: in-process (call `run_agent_turn()` directly, no auth) and HTTP (`--base-url`, mint a token via `evals/auth_helper.py`, hit `POST /agent/chat`, parse SSE the same way the frontend does). Score three dimensions per case: tool-selection correctness (expected tools ⊆ actual, deterministic), forbidden phrasing (no false "I've added" claims, no direct buy/sell directives), and groundedness reusing the existing `"groundedness"` SSE event rather than a second judge call. Exit non-zero on any failure. Then add `.github/workflows/run_evals_on_preview.yml` triggered on `deployment_status` (not `pull_request`) so it gets a real Vercel preview URL; resolve the PR number via GitHub's commit→PR API; post results with `gh pr comment` and set `continue-on-error: true` so it's advisory only. Update `CLAUDE.md` and `docs/ARCHITECTURE.md` to describe the suite and mark the "no automated tests" known issue as partial progress.