LLM observability and eval loop for a LlamaIndex RAG app using Langfuse traces and promptfoo tests

What they did

They started from the Langfuse repo to get a self-hosted observability backend, then wired their existing LlamaIndex retrieval pipeline into it so every query produced a trace: retrieval spans, the assembled prompt, the LLM call, token usage, and latency. Once traces were flowing, they reviewed low-scoring and slow sessions in the Langfuse UI, tagged the recurring failure modes (bad retrieval, hallucinated citations, over-long answers), and exported representative examples. Those examples became a promptfoo test file with assertions for groundedness, citation presence, and answer length. They ran promptfoo locally against candidate prompt variants, compared pass rates, and promoted the winning prompt. Finally they added a small CI job that runs the promptfoo suite on pull requests and posts the score delta, while Langfuse keeps collecting live traffic for the next round of failure mining.

How it went

They ended up with a working trace-to-test loop: production traffic feeds Langfuse, curated traces feed promptfoo, and prompt changes are gated by a scored suite instead of vibes. The setup is reasonably polished for a small team, though the eval assertions are hand-written and only as good as the curated examples. Latency and cost dashboards came almost for free from the traces.

Worth knowing

Self-hosting Langfuse needs Postgres, ClickHouse, Redis, and object storage, which is more infrastructure than a weekend project suggests. Trace sampling and PII scrubbing must be configured before sending real user data. promptfoo assertions for groundedness are heuristic unless you add an LLM-graded rubric, which costs tokens and adds flakiness. Nothing here covers fine-tuning or online A/B traffic splitting.

Wiring

Order: (1) stand up Langfuse (Docker Compose self-host or Langfuse Cloud) and create a project to get API keys; (2) install the langfuse Python or JS SDK and initialize the client from env vars; (3) instrument the pipeline — if LlamaIndex, use its Langfuse callback/integration so retrieval and synthesis become nested spans, otherwise wrap calls manually with trace/span/generation; (4) verify traces appear in the Langfuse UI with correct input/output and token counts; (5) export 10-15 representative traces as JSON and convert them into promptfooconfig.yaml test cases with assertions; (6) run promptfoo eval locally, tune assertions until the baseline is stable; (7) add a CI step that runs promptfoo and compares pass rate to a stored baseline. Langfuse and promptfoo stay decoupled — Langfuse is the data source, promptfoo is the test runner — so either can be swapped without touching the app code.

Try it yourself

Set up an observability and eval loop for this repo's LLM pipeline. First, add Langfuse tracing: install the langfuse SDK, configure LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_HOST from env, and wrap the main LLM call plus any retrieval steps in spans so each request produces one trace with input, output, token usage, and latency. If the pipeline uses LlamaIndex, use the Langfuse LlamaIndex integration instead of manual spans. Second, add a promptfoo config (promptfooconfig.yaml) with 10-15 test cases drawn from realistic inputs, each with assertions for answer relevance, citation or source presence, and max length. Third, add an npm script to run promptfoo in CI on pull requests and fail if the pass rate drops below the current baseline. Show me the diff for each file and list any env vars or services I need to provision.