End-to-end MCP simulation test for LiteX with a dedicated CI job

What they did

The author added an end-to-end MCP test for litex_mcp plus a dedicated CI job. The test file test/tools/test_litex_mcp_sim.py starts the litex_mcp server as a subprocess with --allow-write through an MCP SDK stdio client, passing the environment explicitly so PYTHONPATH reaches the server. Everything after that goes through MCP tools only: litex_sim_start with a VexRiscv CPU and --uart-name=crossover+uartbone --uart-tcp on a free port, then waiting for the simulation to listen (failing fast with the log tail if the process exits), litex_server_start on another free port, litex_connect retried until the server accepts, reads and writes of ctrl_scratch, running the BIOS help command through the crossover console, and, when the server has it (#2745), a litex_build_info check of the simulation job. The test is skipped unless LITEX_MCP_SIM_TEST=1 and the MCP SDK and Verilator are present, so the regular pytest shards only collect it as skipped. In .github/workflows/ci.yml they added an mcp job on Python 3.11 that pip installs mcp, builds LiteX with --config=standard, installs gcc-riscv64-unknown-elf, uses ccache, and pulls Verilator from the same cache key as the build job, plus Python 3.9 because the cached Verilator install calls its build-time interpreter. The job runs test/tools/test_litex_mcp.py, test/build/test_report.py as blocking, and the simulation test with continue-on-error. The build shards, their .test_durations generation and the docs job were left unchanged. The motivation: the pytest shards run on Python 3.9 without the MCP SDK (which needs Python >= 3.10), so the SDK side of litex_mcp (server class, tool registration, annotations, stdio protocol) was never exercised in CI. Because MCP SDK 1.x and 2.x differ (FastMCP vs MCPServer, snake_case result fields), the test reads results through model_dump(by_alias=True) so it works with both.

How it went

Locally the simulation test passed in about 44.8 s with MCP SDK 2.3.0 and 43.3 s with 1.30.0 (Verilator 5.025), and also passed with the litex_mcp of #2745, which exercises the litex_build_info check. Without LITEX_MCP_SIM_TEST it is skipped, and ci.yml parses with jobs build, mcp and docs. On GitHub Actions the mcp job takes about 3 min with a Verilator cache hit and the SDK unit tests pass (21 passed, mcp 2.3.0). The simulation test passed in 2 of 4 runs (48 s); in the other 2 the sim built and listened and litex_server connected, but every UARTBone read timed out from the first one, so the step is continue-on-error for now.

Worth knowing

The simulation test is flaky on GitHub Actions: 2 of 4 runs failed with UART read timeouts (0/4 bytes received) for 2 min and a 30 s connect timeout, not reproduced locally (5/5 passes, also pinned to 2 cores with taskset). It looks like a litex_sim/serial2tcp startup problem rather than an MCP one, so the step is continue-on-error with a comment in ci.yml and the test prints job logs on failure; it needs investigation before becoming blocking. The test only runs when LITEX_MCP_SIM_TEST=1 and the MCP SDK and Verilator are installed, so normal shards just skip it. The branch carries follow-up commits from CI iterations (ccache, Python 3.9, diagnostics, an informational step) that may be squashed on merge. Written by an AI agent (Claude Code), supervised by Florent Kermarrec.

Wiring

The MCP SDK stdio client spawns `litex_mcp --allow-write` as a subprocess with an explicit env so PYTHONPATH reaches the server. Tool calls go in order: litex_sim_start (VexRiscv, --uart-name=crossover+uartbone --uart-tcp on a free port), poll until the simulation listens (on exit, fail with the log tail), litex_server_start on a second free port, litex_connect retried until the server accepts, then ctrl_scratch reads/writes, the BIOS `help` command over the crossover console, and litex_build_info when available. Results are read through model_dump(by_alias=True) for MCP SDK 1.x/2.x compatibility. The test is gated on LITEX_MCP_SIM_TEST=1 plus the MCP SDK and Verilator. In ci.yml the new `mcp` job runs on Python 3.11 with `pip install mcp`, LiteX built with --config=standard, gcc-riscv64-unknown-elf, ccache, Verilator restored from the same cache key as the `build` job, and Python 3.9 present for the cached Verilator install's build-time interpreter. It runs test/tools/test_litex_mcp.py, test/build/test_report.py (blocking) and test/tools/test_litex_mcp_sim.py (continue-on-error). The build shards, .test_durations generation and docs job are untouched.

Try it yourself

Add an end-to-end MCP test for litex_mcp and a dedicated CI job. Write test/tools/test_litex_mcp_sim.py that starts `litex_mcp --allow-write` through an MCP SDK stdio client with the environment passed explicitly so PYTHONPATH reaches the server, then uses only MCP tools: litex_sim_start (VexRiscv, --uart-name=crossover+uartbone --uart-tcp on a free port), wait for the simulation to listen and fail fast with the log tail if it exits, litex_server_start on another free port, litex_connect retried until accepted, read/write ctrl_scratch, run the BIOS `help` command through the crossover console, and check litex_build_info when the server supports it. Read results via model_dump(by_alias=True) so MCP SDK 1.x and 2.x both work. Skip unless LITEX_MCP_SIM_TEST=1 and the MCP SDK and Verilator are present. In .github/workflows/ci.yml add an `mcp` job on Python 3.11 that pip installs mcp, builds LiteX with --config=standard, installs gcc-riscv64-unknown-elf, uses ccache, pulls Verilator from the same cache key as the build job, and also has Python 3.9 for the cached Verilator install; run test/tools/test_litex_mcp.py, test/build/test_report.py as blocking, and the simulation test with continue-on-error and a comment explaining the UART timeout flakiness. Leave the build shards, .test_durations generation and docs job unchanged.