Private MCP execution proof and behavior signals for NVIDIA SkillEvaluator

What they did

The author implemented split-plan items 20–21 on main in the NVIDIA SkillEvaluator repo, after the runtime prerequisite in #264 merged. The work makes private runs able to distinguish declared MCP servers, successful host initialization and tool listing, confirmed agent loading, and successful correlated agent calls. They added opt-in HTTP/SSE probes with endpoint policy checks, pinned addresses, scoped environment grants, shared deadlines and bounded identity-encoded responses; default runs make no host probe and host stdio commands are never executed. They normalized logical trials once and reused rewards, traces, native census and startup evidence, preserving child ownership, authoritative multi-step topology, physical attempt identity, recorded usage fallback, unknown load states and explicit missing-evidence outcomes. Dataset and server-authored patterns are checked before compilation: counted bounds capped at 100, nested counted expansion rejected, unsupported repetition grammars conservatively refused. Malformed schema values and harness logs can no longer discard otherwise valid evidence, and shared diagnostic sanitization strips controls before credential redaction. They preserved main's source guards, native MCP policy, baseline-only behavior and setup-only retries, resetting native, observer and sidecar state before retries including attempts that fail before creating a jobs directory. Obsolete merge-era plumbing and a redundant commit-bound Gitleaks exception were removed, a bounded redacted evidence sidecar is persisted, and the already-resolved regex dependency is declared explicitly in the base package.

How it went

475 tests passed in the affected evidence, transport, lifecycle, usage and child-graph scope; final pattern changes separately passed 43 targeted regressions and 260 signal tests. Regressions reproduce review findings before their fixes, rejected expansion patterns use a compiler trap so no hostile pattern is compiled, and independent re-review passed. Ruff, formatting and diff checks passed, and full reachable-history Gitleaks scanning passed with the exact synthetic-fixture exception inherited from main. Merge compatibility checks passed 258 with 1 skipped. Wheel and source archive built successfully, and an offline base-only installation verified 230 packaged files byte-for-byte across checkout, wheel, source archive and installation, with isolated imports, compile guards, redaction and three installed CLI help commands passing. All four new review threads have pushed fixes, technical replies and resolved status.

Worth knowing

Tests use saved artifacts and local/fake transports; no live provider, native agent binary or external MCP service was exercised. This PR only provides private evidence collection: runtime report presentation and assembled-workflow acceptance remain item 24, and public CLI/configuration activation remains item 25, so private activation flags are still absent. #322 builds comparisons and static report views on this branch. CI status remains visible in PR checks rather than being asserted here.

Wiring

Work sits on main after the #264 runtime prerequisite. Order: (1) opt-in transport layer for HTTP/SSE probes with endpoint policy, pinned addresses, scoped env grants, shared deadlines and bounded identity-encoded responses, leaving default runs probe-free and stdio unexecuted; (2) trial normalization feeding rewards, traces, native census and startup evidence from one pass; (3) pre-compilation pattern checks with a compiler trap for rejected expansions; (4) diagnostic sanitization before credential redaction; (5) retry reset of native, observer and sidecar state, including pre-jobs-directory failures; (6) bounded redacted evidence sidecar persistence; (7) explicit regex dependency in the base package. Validation runs through the repo's test suite, Ruff, formatting, diff checks and Gitleaks over reachable history, plus wheel/source-archive builds and an offline base-only install check. Public CLI/configuration activation is deferred to item 25, and report presentation to item 24.

Try it yourself

In this repo, implement private MCP execution proof and behavior signals on top of the existing private runtime. Add opt-in HTTP/SSE probes with endpoint policy checks, pinned addresses, scoped environment grants, shared deadlines and bounded identity-encoded responses; default runs must make no host probe and must never execute host stdio commands. Normalize logical trials once and reuse rewards, traces, native census and startup evidence, preserving child ownership, multi-step topology, physical attempt identity, usage fallback, unknown load states and explicit missing-evidence outcomes. Validate dataset and server-authored patterns before compilation: cap counted bounds at 100, reject nested counted expansion, and conservatively refuse unsupported repetition grammars. Ensure malformed schema values and harness logs cannot discard valid evidence, and sanitize diagnostics by stripping controls before credential redaction. Preserve existing source guards, native MCP policy, baseline-only behavior and setup-only retries; reset native, observer and sidecar state before retries including attempts that fail before creating a jobs directory. Persist a bounded redacted evidence sidecar, declare the regex dependency explicitly in the base package, and add regression tests that reproduce each review finding before its fix.