Cloud latency evaluation harness with opt-in Staging diagnostics for an agent runtime

What they did

The author added a Cloud latency evaluation lane to the opentag repo. They established common Cloud latency endpoints and source/evidence coverage, then built a repeatable offline orchestration baseline. On top of that they added the first human-approved increment of opt-in Staging diagnostics: the diagnostic lane requires Staging, an explicit flag, and up to 16 selected Session UUIDs. Existing Server/Runner log streams were reused to observe preparation/allocation, workspace readiness, parent worker execution, checkpoint and matching report acknowledgement, and existing Pi phase durations were reused rather than re-instrumented. Measurement records explicitly carry INFO severity, and the reader tolerates Cloud Logging moving that transport field out of jsonPayload. Server retries share a single 32-record / 64 KiB delivery quota with a reserved final summary; duplicate attempts are suppressed while distinct retries stay observable within budget. Non-dispatch returns get incomplete summaries, and unavailable allocation results are treated as failure events excluded from successful allocation samples. Runner observations keep the same bounded stream quota, and bookkeeping never changes work admission. A manual stdin-only reader caps input at 16 MiB / 10,000 rows and 24 hours, aggregates in memory, and outputs counts and quantiles without resource IDs or raw files. It surfaces missing or conflicting evidence, keeps failed/cancelled/unknown outcomes distinct, and refuses to infer timeout or ingress/user end-to-end latency from partial records. No database, application cache, custody/retry, workspace-persistence, credential, protocol, exporter, retention or scheduler behavior changed. The retained Linux/arm64 fixture holds 402 attempts: 100 successes in each of four scenarios plus exact rejection/timeout injections, under one CPU / 1 GiB and no network. Pi/MCP/model/IM replacements and single-file gzip measure local fixture orchestration only. The author also resolved a conflict with main, preserving main's accepted-inbound-receipt fence and retry handling alongside the new diagnostics and keeping both observability sections.

How it went

Runner/diagnostic observer tests passed 53/53, and the Agent Runtime coverage suite passed 821/821 with 100% statements, branches, functions and lines. Shared/Client/Server builds and typechecks, pnpm check, lint/format and full pre-push typecheck passed. The preceding conflict-resolution check passed 8 Shared, 71 Server and 9 offline harness cases. The 402-attempt baseline remains synthetic component evidence, and the author explicitly states that the complete current-head CI matrix and a commit-bound non-author review are still required for integration.

Worth knowing

Native Cloud/amd64, real ingress/queue, GCS, model/tool TTFT, provider send receipts, resource pressure, deployed log arrival and genuine end-to-end acceptance remain unverified. Source integration is separate from live operational acceptance; no deployment, real model call, IM send or task replay was performed or authorized. The fixture is synthetic and does not benchmark production Cloud or logging overhead. The referenced startup PR #1112 is only cited at its existing evidence head; its implementation and acceptance remain with its owner. Issue #1108 stays open for authorized live collection, a joined Cloud baseline and repeated candidate/rollout acceptance.

Wiring

Work proceeds in layers. First, the measurement contract and endpoint definitions land in testing/performance/cloud-latency/README.md, with BASELINE.md, LIVE.md and REPORT.md documenting baseline, Staging boundaries and evidence gaps. Second, the offline harness fixture is generated and retained as synthetic data under a Linux/arm64, one-CPU, 1 GiB, no-network profile, with Pi/MCP/model/IM replaced by local stand-ins and single-file gzip used for orchestration measurement. Third, diagnostic instrumentation is added to Server and Runner log streams behind a Staging-only flag plus explicit opt-in and a bounded set of Session UUIDs; records carry INFO severity and the reader tolerates Cloud Logging relocating that field out of jsonPayload. Fourth, a shared delivery quota of 32 records / 64 KiB with a reserved final summary is applied to both Server retries and Runner observations, with duplicate suppression and incomplete-summary marking for non-dispatch returns. Fifth, the stdin-only reader is wired as a manual, offline aggregation step that consumes capped input and emits counts and quantiles only. Finally, the branch is rebased onto main c0d12aef, resolving conflicts so main's accepted-inbound-receipt fence and retry handling coexist with the new diagnostics, and the public-export snapshot is updated for the additive Shared diagnostic exports.

Try it yourself

Add a Cloud latency evaluation lane to this repo, modeled on testing/performance/cloud-latency. Steps: (1) define common Cloud latency endpoints and document source/evidence coverage in a README; (2) build a repeatable offline orchestration baseline fixture (Linux/arm64, one CPU, 1 GiB, no network) with 100 successes per scenario across four scenarios plus exact rejection/timeout injections, and record it in BASELINE.md; (3) add opt-in Staging diagnostics gated on Staging, an explicit flag, and up to 16 selected Session UUIDs, reusing existing Server/Runner log streams for preparation/allocation, workspace readiness, parent worker execution, checkpoint and report acknowledgement, and reusing existing Pi phase durations; (4) enforce a shared 32-record / 64 KiB delivery quota with a reserved final summary, suppress duplicate attempts, mark non-dispatch returns as incomplete, and exclude unavailable allocation results from successful samples; (5) write a manual stdin-only reader capped at 16 MiB / 10,000 rows / 24 hours that aggregates in memory and prints counts and quantiles without resource IDs or raw files, flags missing/conflicting evidence, keeps failed/cancelled/unknown distinct, and never infers timeout or end-to-end latency from partial records; (6) keep all changes additive with no database, cache, credential, protocol, exporter, retention or scheduler changes, and document unverified live boundaries in REPORT.md and LIVE.md.