AgentOps observability for a CrewAI research crew

What they did

They had a small CrewAI crew that answered research questions by pulling documents through a LlamaIndex retrieval pipeline and then writing a synthesized brief. The problem was visibility: when an answer was wrong or slow, they could not tell whether the retriever returned bad chunks, the agent picked the wrong tool, or the model just hallucinated. They added AgentOps as the observability layer. They installed the agentops package, grabbed an API key from the dashboard, and initialized a session at the top of the entrypoint so every run got a session id. They then let AgentOps auto-instrument the CrewAI agents and tasks, which produced a trace tree of agent steps, tool calls, and LLM completions. They tagged each session with metadata (crew name, question id, retriever index version) so runs could be filtered later. They also wrapped the LlamaIndex query engine call so retrieval latency and returned node counts showed up alongside the agent spans. Finally they added a small evaluation pass that recorded a score per session, so they could compare prompt variants in the dashboard instead of eyeballing outputs.

How it went

They ended up with a working trace view: each research run shows the crew's task sequence, which tools fired, token counts and estimated cost per LLM call, and the retrieval step. Debugging moved from re-running with print statements to opening a session replay. The evaluation tagging let them compare two prompt versions side by side. Polish level is prototype-to-internal-tool: it runs reliably on their machine and in CI smoke tests, but it is not hardened for multi-tenant production traffic.

Worth knowing

Auto-instrumentation covers the common CrewAI paths but custom tool wrappers may need manual spans. Session metadata and evaluation scores are only as useful as the tags you remember to add. The API key must be kept out of source control, and the free tier has event limits that a chatty crew can hit quickly. LlamaIndex retrieval spans required an explicit wrapper rather than appearing automatically.

Wiring

Order matters: load environment variables first, then call agentops.init(api_key=...) before importing or constructing the CrewAI crew so auto-instrumentation patches the right classes. Build the LlamaIndex index and query engine next, wrapping the query call in a manual span. Construct the CrewAI agents and tasks, then kickoff inside a try/finally that calls agentops.end_session with a success or fail status. Attach metadata (crew name, run id, index version) at session start so dashboard filters work. Config lives in .env: AGENTOPS_API_KEY, model provider keys, and the index path. For CI, run a single smoke question and assert a session id was returned.

Try it yourself

Add AgentOps observability to this repo's CrewAI agent. Steps: (1) install agentops and add AGENTOPS_API_KEY to .env.example, reading it via os.environ; (2) initialize agentops.init() once at the entrypoint before any crew is constructed, and end the session in a finally block; (3) tag the session with metadata for crew name, run id, and retriever index version; (4) wrap the LlamaIndex query engine call in a span so retrieval latency and node count are recorded; (5) record a per-session evaluation score from the existing output check. Keep changes minimal, do not alter agent prompts, and show me the diff plus how to verify traces appear in the dashboard.