Agentic LLM Serving Benchmark on TensorRT-LLM with Trace Replay

What they did

They wanted to know whether their agent stack's inference layer was actually the bottleneck, so they built a benchmark harness around TensorRT-LLM instead of trusting single-request latency numbers. First they captured real agent traffic — multi-turn tool-calling sessions with variable prompt lengths, KV cache reuse across turns, and bursty concurrency — and stored it as replayable traces. Then they stood up a TensorRT-LLM server, loaded a supported model, and wrote a replay driver that feeds the traces at controlled arrival rates while recording per-request timings. Rather than reporting only tokens/sec, they computed job-level metrics: end-to-end time per agent job, time-to-first-token under load, queueing delay, and how often a job's latency blew past its budget. They swept batch sizes, CUDA graph settings, and KV cache configuration, comparing each configuration against the same trace so results were apples-to-apples. Finally they used the job-level view to pick a serving configuration and documented which knobs mattered for agentic traffic versus plain chat traffic.

How it went

The trace-replay approach gave a much clearer picture than synthetic benchmarks: they could see which agent jobs were dominated by queueing versus generation, and they settled on a serving configuration tuned for their real arrival patterns. The harness is reusable for regression testing after model or config changes. Polish is moderate — it works for their workload but is not a general-purpose benchmark product, and results are specific to the traces they captured.

Worth knowing

Requires NVIDIA GPUs and a TensorRT-LLM-supported model; setup involves CUDA, PyTorch, and MPI dependencies that can be version-sensitive. Trace capture must faithfully preserve concurrency and prompt structure or the replay misleads. Job-level metrics depend on how you define a job boundary. No public dataset or turnkey script is implied — you supply your own traces and model weights.

Wiring

Order: install CUDA toolkit and PyTorch matching the TensorRT-LLM release, install TensorRT-LLM (pip wheel or build from source), download a supported model checkpoint, then launch the server (e.g. trtllm-serve) exposing an OpenAI-compatible endpoint. The replay client is a separate Python process that reads trace files (JSONL with arrival timestamps, prompt, max_tokens, job_id), uses asyncio plus an HTTP client to fire requests at the scheduled offsets, and logs per-request timings keyed by job_id. A metrics aggregator joins request logs into job-level summaries and writes CSV/JSON. A sweep runner shells out to restart the server with different config flags (batch size, KV cache fraction, CUDA graph batch sizes), waits for health checks, runs the same trace, and collects results into one comparison table. Config lives in a YAML file shared by the replay client and sweep runner so traces and settings stay reproducible.

Try it yourself

In this repo, help me build an agentic serving benchmark for TensorRT-LLM. Steps: (1) add a trace format that records per-request arrival time, prompt token ids or text, max_tokens, and a job_id grouping multi-turn agent calls; (2) write a replay client that reads the trace and issues requests to a running TensorRT-LLM OpenAI-compatible server at the recorded arrival times with configurable concurrency; (3) record per-request TTFT, total latency, and output tokens, then aggregate to job-level metrics (job end-to-end latency, p50/p99, queueing delay); (4) add a config sweep script that varies max_batch_size, CUDA graph settings, and KV cache fraction, reruns the same trace, and writes a comparison table; (5) add a small synthetic trace generator so I can test without production data. Keep dependencies minimal and document GPU/model requirements in the README.