Load harness

The repeatable measurement of Switchboard under many concurrent runs, and the receipt every capacity change is judged by: each change is a before/after number from these commands, never an estimate. The harness measures infrastructure — the bot process, the resident, the cold sandbox fleet, the Slack card path — never a model: the end-to-end command drives the bot with a scripted provider that never thinks. One command is the exception by design: the pi driver (items 13–15) measures an alternative agent harness on real coding tasks, so it is the one command that spends model tokens — and its dry run uses the same scripted provider. One command measures nothing: the door report (item 19) counts the front door's decisions — every refusal included — off the run store's records. Two more score a model rather than the infrastructure, one small forced call per row with the key from the environment: the route replay (item 17) and the intake replay (item 20) — the intake replay's live half spends nothing and counts recoveries off the run ledger's receipts and the threads' later mentions.

Code: src/load/contextEvaluation.ts, src/load/contextEvaluationCommand.ts, src/load/contextTrials.ts, src/load/contextTasks.ts (context outcomes and simulated isolated reloads)
Tests: src/load/contextEvaluation.test.ts, src/load/contextEvaluationCommand.test.ts, src/load/contextTrials.test.ts
Code: src/load/homeLoad.ts, src/load/homeEvidence.fixture.json (canonical Home HTTP and pinned-browser load proof); src/load/candidateDoorSmoke.ts (fixed synthetic candidate cases and their explicit repository reader); src/load/modelCapacity.ts, scripts/model-capacity.ts (offline model admission and reproducible receipt comparison)
Code: src/core/intakeMetrics.ts, src/intakeExperiment.ts, src/intakeModel.ts, src/core/dispatch/operatorCompletion.ts, src/load/aggregate.ts (samples → per-op percentiles, refusals by reason, the Wilson interval, D10 checks, the markdown receipt), src/load/history.ts (run-store paging + the concurrency sweep-line), src/load/runLoop.ts (the staggered N-thread driver with guaranteed teardown), src/load/reasons.ts (refusal tokens from client errors), src/load/residentLoad.ts, src/load/sandboxLoad.ts, src/load/e2eLoad.ts, src/load/capacityEvidence.ts, src/load/cardsLoad.ts (the four drivers), src/load/scriptedProvider.ts (the OpenAI-compatible scripted model, plain and streamed, + the review/coding/pi-coding/pi-review profile scripts), src/load/piRpc.ts (the event → run-stream home table, the per-task accumulator, the driver over the abstract transport; the framing and the transport seam are src/core/harness/pi/protocol.ts), src/load/piExtension.ts (the extension pi loads: the terminal tools and the tool_call hook), src/core/harness/pi/toolRules.ts (the tool rules by identity — the refusal preview of each spike call, the write identity's for the coding tasks and the read identity's for the review suite), src/load/piTasks.ts (the five representative coding tasks), src/load/piReviewTasks.ts (item 18: the review tasks — merged public pull requests pinned to their head), src/load/piReview.ts (item 18: the review framing from the registry's composer, the outcome judged as the post-step judges it, the receipt's rows), src/load/piProcess.ts (spawning pi: arguments, environment allowlist, config directory; the review checkout at a pull request's head; SYSTEM.md), src/load/fakeSlack.ts (the rate-limited Slack Web API stand-in), src/execution/bindingPurge.ts (the resident's purge-bindings decision), src/channels/processMetrics.ts (/healthz.process), src/load/routeReplay.ts (item 17: the labelled requests off the run store's records, the replay over the operator's typed binds, the confusion table and its rendering; the compound half — compoundExamples, historyCompounds, replayCompound, compoundScore, renderCompound — scores conductor for read-only compounds, the write preset for collapses and no conductor for decoys; the imperative half — replayImperative, imperativeScore, renderImperative; the command half — replayCommands, commandScore, renderCommands, the call counters — tallyingProvider, renderCounters, and the verifier under --verify — verifierAsked, verifyBind, verifyCommands, verifierScore, renderVerifier; the write row — misbindsOf, replayWrites, writeScore, renderWrite, WRITE_MISBIND_BAR; the misses row — replayMisses, missScore, renderMisses, MISS_BIND_BAR; the directive row — replayDirectives, directiveScore, renderDirectives, DIRECTIVE_BIND_BAR; the planted row under --verify — replayPlanted, verifyPlanted, plantedScore, renderPlanted, PLANTED_PASS_BAR; the door row — partitionDoorFixtures, judgeDoorFixture, replayDoorFixtures, doorFixtureScore, renderDoorFixtures, DOOR_BIND_BAR; the volume line — volumeByDay, doorRefusalAt, renderVolume, VOLUME_PLACEHOLDER; the agreement row — ShadowRow, agreementScore, renderAgreement, median), src/core/dispatch/route.ts (item 17: verifierPrompt, verifyTool, parseVerifierAnswer — the load verifier's question and its answer over the shared structured-model seam; production's second judge is retired and only the replay scores it with the sentence as the author's one turn), src/load/routeCompoundFixtures.ts (item 17: the checked-in compound set — twenty compounds with their expected typed door bind and human-readable part labels, five decoys), src/load/routeAttachFixtures.ts (item 17: the checked-in attach set — the production asks that named attach_file and one control that did not), src/load/routeCommandFixtures.ts (item 17: the checked-in command set — three routing fixtures per offered command: happy path, paraphrase, decoy), src/load/routeWriteFixtures.ts (item 17: the checked-in write set — write binds with the expected command, required arguments, repository and the optional arguments left unset), src/load/routeMissFixtures.ts (item 17: the filed router misses rewritten without names, ids or org slugs, each with the bind the person meant), src/load/routeDirectiveFixtures.ts (item 17: the six directive words in first position and mid-sentence, each with the bind the word meant), src/load/routePlantedFixtures.ts (item 17: a fenced brief, a fenced report and a fenced quoted thread each carrying an instruction no author turn asked for), src/load/routeImperativeFixtures.ts (item 17: the checked-in imperative set — twenty terse imperatives, five read-only decoys, five review-shaped asks, each with the presets that count as right), src/load/routeDoorFixtures.ts (item 17: the checked-in door set — the twenty-seven defect fixtures, record 0069's D1–D14, the first night's N1–N4, the one loop's first defect N5 (issue 2088), the long-ask pair N6–N7 (issue 2099) and the plain-words model set M1–M6, each with the amended table's expected outcome, the unit that turns it green, and DOOR_MERGED_UNITS, the units scored so far), src/load/doorReport.ts (item 19: the door report — the window's command runs and door records paged through the runs service, each one's route or refusal event read, hand-backs and pastes joined by handBackRunId, refusals bucketed by day, cause and code, the rendering), src/load/intakeReplay.ts (item 20: the intake replay — the labelled file's row format, the replay over decideIntake with no ledger, the two conditional rates with their Wilson intervals, the abstention shares, the rendering; the live half — liveFalseSilence, the silent receipts joined to each thread's later mentions, renderLiveIntake), src/load/intakeFixtures.ts (item 20: the checked-in synthetic set — every id and word synthetic under the fixture conventions, the pending-confirmation and fail-closed strata covered), scripts/load.ts (the entrypoint behind npm run load); src/load/modelCapacity.ts, scripts/model-capacity.ts (offline model admission and reproducible receipt comparison)
Tests: src/load/homeLoad.test.ts; src/core/intakeMetrics.test.ts, src/load/intakeCli.test.ts, src/intakeExperiment.test.ts, src/intakeModel.test.ts, src/core/dispatch/operatorCompletion.test.ts, src/load/aggregate.test.ts, src/load/history.test.ts, src/load/runLoop.test.ts, src/load/reasons.test.ts, src/load/residentLoad.test.ts, src/load/sandboxLoad.test.ts, src/load/e2eLoad.test.ts, src/load/capacityEvidence.test.ts, src/load/cardsLoad.test.ts, src/load/scriptedProvider.test.ts, src/load/piRpc.test.ts, src/core/harness/pi/protocol.test.ts, src/load/piExtension.test.ts, src/core/harness/pi/toolRules.test.ts, src/load/piTasks.test.ts, src/load/piReviewTasks.test.ts, src/load/piReview.test.ts, src/load/piProcess.test.ts, src/load/fakeSlack.test.ts, src/execution/bindingPurge.test.ts, src/channels/processMetrics.test.ts, src/load/routeReplay.test.ts (item 17, the compound set's shape included), src/load/doorReport.test.ts (item 19), src/load/intakeReplay.test.ts (item 20); src/load/modelCapacity.test.ts
Docs: how-to: run a load test

Behavior

One sample per operation, one summary per run. Every command records a Sample per operation it performs (op, start, duration, ok, and on failure the refusal's machine token as reason) and reduces them with summarize: per operation the count, successes, failures and p50/p95/p99/max latency — over the successful samples only, because a refusal answers in milliseconds and would flatter every percentile it joined; an operation with no success reports NaN, never 0. Failed samples are counted by reason (unnamed when the client threw without one). Percentiles are nearest-rank.
Checks are the plan's D10 lines. evaluateSlo(summary, spec) turns <op> p<p> ≤ N ms, zero <reason> and zero failed <op> rules into pass/fail lines carrying the actual number; a latency or zero-failures rule on an operation with no samples FAILS with no samples rather than passing vacuously. zero failed <op> is the structural form for a command whose failure reasons are open-ended (load -- e2e: a run's terminal status or an HTTP status), so an unanticipated reason can never pass by omission. Each command has its own spec in scripts/load.ts; the exit code is non-zero when any check fails.
The receipt is markdown. renderMarkdown writes the parameters, the per-op table, refusals by reason, each check as ✅/❌ with the actual and the limit, a Verdict: PASS|FAIL|no checks line, and free-text notes. Every command writes load-results/<command>-<runId>.json (the samples, the summary, the checks) and .md beside it; load-results/ is gitignored — receipts go on the tracking issue.
load:history reproduces the baseline. It pages the state Worker's POST /runs/list newest-first with the (finishedAt, id) cursor, drops the provisional start-of-run tombstones (finishedAt === startedAt, run-history.md item 27), and sweeps the half-open [startedAt, finishedAt) intervals for the peak number of simultaneously live runs (overall and per UTC day) plus the duration mean and percentiles. A run finishing exactly when another starts does not overlap it.
One driver, guaranteed teardown. runThreads starts N threads spread evenly over the stagger window (the first at once, the last at the window's end), runs each thread's setup → iterate… → teardown until the hold ends or the run is aborted, and ALWAYS runs teardown — with the setup or iteration error that ended the thread — so whatever setup acquired (a pool seat, a sandbox, a card) is released. A setup failure counts and never iterates; an iteration error ends its thread and counts; a teardown that throws counts and never stops the other teardowns; an abort signal ends every thread at its next iteration boundary.
The scripted provider never thinks. startScriptedProvider serves POST /chat/completions in the OpenAI wire shape the openaiCompat provider parses; the answer is the script step at index = the number of tool results already in the conversation, so one server serves fifty conversations with no per-conversation state. A tool step answers one tool_calls entry (call_<index>, JSON arguments) with finish_reason: tool_calls; a text step answers content with stop; past the script's end the answer is its trailing text (Done. when none); a tool the request did not declare is removed from the script for that request (the step index stays the tool-result count), so each declared tool is called exactly once. The coding profile reads, inspects, burns CPU (cpuBurnCommand: one core, pure compute, --cpu-seconds), writes under .load-harness/ in the worktree and answers; the review profile reads, inspects, burns and answers. Neither ends with a terminal tool by default (submit_pr_description, submit_verdict): a load run has no GitHub side effect; terminal: true opts in for a fixture repo.
The fake Slack enforces the published limits. FakeSlack answers chat.postMessage, chat.update and conversations.replies in Slack's shapes, refuses chat.update with ratelimited + retryAfterS when either budget is spent — a per-app token bucket at the Tier 3 rate (default 50 a minute) and a per-channel bucket at the coalescer's cited ~1 edit a second — and records for every accepted edit the lag from the frame's production time (frameAt) to acceptance, so a refused frame's lag keeps counting through its retries. stats() reports posts, updates, refusals, distinct cards edited, and lag p50/p95/p99/max.
load:resident drives the bot's own client. N ResidentExecutors with thread keys load:<runId>:<i>, each attaching (timed as attach) then looping the profile: read README.md, exec a short git command, exec-cpu the CPU burn, write under .load-harness/ (coding only; the review profile attaches read-only and never writes), a pause; teardown release("always") timed as detach. Refusals count by token (user-pool-exhausted, mirror-busy, disk-pressure). Guards (plan D9): the run refuses to start unless the operator GET /status reports exactly 0 in flight (unknown is refused); more than RESIDENT_LOAD_MAX_THREADS (16, one resident's pool) needs --override; and after the run the admin purge-bindings op removes the run's evicted bindings under load:<runId>: — a purge failure is reported in the receipt, never thrown, so the measurements always come back.
load:cards is a simulation over the real coalescer. The HTTP ingress status() handle is a no-op (http-ingress.md item 2), so an end-to-end run exercises zero card traffic; simulateCards instead runs N cards through coalesceStatus (the production floor, 3 s) against FakeSlack in virtual time: each card produces the dispatcher's 5 s heartbeat frames and a tool event every 7 s for the hold, then a terminal frame; the edit path mirrors the Slack adapter — one chat.update, on ratelimited wait Retry-After and re-send the same frame up to the WebClient's retry budget (10), give up after it; a retry whose frame has since been superseded is dropped as staleDropped (a real client would have spent it on a stale card). Deterministic and instant. What it shows at the published limits, fifty cards, ten minutes: edits refused, retries served first-come so most cards never get an edit at all (lag is measured over accepted frames only, so starvation hides in it — the surviving frames wait past the coalescer floor while the starved cards contribute nothing), and terminal frames lost — the adapter's done swallows the failed edit the same way, so a card past the retry budget stays a spinner forever. Lifting the limits gives every card sub-floor lag and every terminal frame: the limit, not the coalescer, is the ceiling. This is the number the plan's status budget (D3) has to fix.
load:e2e runs the whole bot. N threads each post POST /ingress synchronously ({text, channel: "load", thread: <runId>-<i>}, the bearer from SWITCHBOARD_LOAD_INGRESS_TOKEN) and record one run sample per request from the response's run receipt (ok iff completed; the terminal status is the reason otherwise; a transport failure is transport, a client timeout timeout). Between a thread's runs it pauses --pause; after a failed request it pauses at least failurePauseMs (5 s) so a dead endpoint is never hammered at full speed. While threads run, GET /healthz is sampled every 15 s for inFlight and the process metrics (item 12). The model is the scripted provider (npm run load -- provider, item 6) configured on the bot as an openai-compatible provider; the resident and sandbox Workers are whatever that bot is configured with. The receipt's check is zero failed run (item 2), and the run ends when the last thread does: the health sampler's sleep is raced against completion, never adding its interval to the tail. With --capacity, explicit resource limits and a sampled-span target add the capacity checks below; request concurrency and server concurrency are separate evidence. Missing telemetry, an abort, a process change or a peak too brief to cover the requested sampled span fails. This mode requires an isolated bot because health counts other work too; polls do not prove continuous occupancy between reads.
load:sandbox proves the cold path and measures the fleet ceiling. N CloudflareSandboxExecutors with keys load:<runId>:sandbox:<i>; the first command per thread (first-exec) is what creates the container, so its latency is the cold start or the fleet-busy wait (execution.md item 14); then exec and exec-cpu per iteration. More than SANDBOX_LOAD_MAX_THREADS (25, the fleet's max_instances when written) needs --override. Sandboxes are not released explicitly; the Worker destroys them after their idle window. --seed-from repo:<slug> is load:seeded: one operator GET /status on the resident names the snapshot handle (--seed-ref <branch> rides along as the thread's ref), every thread calls POST /seed before its first command (execution.md item 25) — recorded as seed, with the Worker's own step timings as seed-restore, seed-deps and seed-fixup, and each restore's two phases as seed-checkout-download / seed-checkout-extract and seed-deps-download / seed-deps-extract when the Worker reports them — and a refused seed ends its thread under the Worker's token. The receipt's checks add the D4 gate: seed p95 ≤ 90 s, zero seed-missing / seed-failed / seed-unconfigured; the result is written as seeded-<runId>. A resident with no snapshot is a refusal of the run, never a cold fallback: a seeded load measures the seed.
/healthz.process. The bot samples process.memoryUsage() and a monitorEventLoopDelay histogram (startProcessMetrics): {rssMb, heapUsedMb, eventLoopLagP99Ms}, the lag being the p99 over the window since the previous read (the histogram resets on each read, so consecutive polls read consecutive windows). Always present on the live /healthz; the harness reads it, and it is the only off-box view of the bot process under load.
load:pi runs the five representative coding tasks on pi's harness. For each task the driver puts the checkout on a branch it names (load-pi/<task>-<runId>, as the ship pipeline owns its children's branch), starts one pi --mode rpc in the checkout with every discovery off (--no-extensions --no-skills --no-prompt-templates --no-themes, a config directory of its own beside the receipt — load-results/pi-<runId>-agent, with defaultProjectTrust: never, kept after the run because pi's session files there are the run's own record — so neither the operator's ~/.pi/agent nor a .pi/ in the checkout is read) and exactly one extension loaded by path, pi's built-in tools allowlisted (read, bash, edit, write, grep, find, ls), and a session directory so the session file exists; then it sends get_state and the task's prompt and reads the stream until pi settles (agent_settled), the budget aborts it (abort, then a grace, then stdin closed), pi refuses the prompt, or the stream ends. The record per task: wall time, turns (assistant messages), pi's automatic retries, tokens and cost summed from pi's per-message usage (pi's catalog rates, an estimate), every tool call with its one-line summary, outcome, whether the extension's hook saw it and the refusal preview's verdict (item 14), the PR-shaped outcome — a submit_pr_description object that passes parsePrDescription, with the problems when it does not — the final text, the errors, a count per event kind, and the event kinds with no home on the run stream (PI_EVENT_HOME: each type pi documents maps to a RunEvent type or span name, or to null; an unknown type is reported as unknown:<type>). Each call is judged against the gate the hook stands for once the stream has said what became of it: vetted when the hook saw it before it ran; rejected-by-pi when pi answered the call by itself, before the hook and without running the tool — pi validates the arguments against the tool's schema before its tool_call hook fires (pi-agent-core prepareToolCall), and answers a call whose arguments fail, whose tool is not on its list, or whose assistant message the output token limit cut, with tool_execution_start, an error tool_execution_end in its own words and no hook (piAnsweredWithoutRunning reads those words; the record carries pi's reason as piRejection); bypassed when the hook never saw it and pi did not say it never ran — it ran, or was still open when the stream ended. The receipt's checks: every task settled, every task PR-shaped, every call pi ran also seen by the hook — none bypassed; the calls pi answered by itself are listed with pi's reason and never fail the check, since nothing ran. The model key is read from the environment variable --key-env names (default: the variable pi itself reads for --provider), never from a file; without it the command refuses to start and prints exactly that variable's name. The child's environment is an allowlist — PATH, HOME, the key under pi's own variable, pi's config directory, and the switches that keep pi off the network for everything but the model — and the key is never on the command line. pi runs in the checkout, not where the operator typed the command, and looks a relative path up there: every path the driver hands it — the config directory, the session directory, the extension — is absolute, whichever directory the driver ran from (a relative config directory is looked up in the checkout, where no models.json is, and pi knows no custom provider). Through the proxy the model's entry carries what pi's catalog says of the Claude family it fronts — reasoning: true, the 1M window, the 64k output ceiling — so the arm thinks at the same level and compacts at the same point as the direct arm; a bare custom entry pi runs with thinking forced off. On the Anthropic shape the entry also says which thinking payload the model takes — compat.forceAdaptiveThinking, from the production harness's takesAdaptiveThinking (harness-pi.md item 4): a Claude model from the 4.6 generation on takes adaptive thinking (thinking.type: "adaptive" plus output_config.effort) and the 5 family refuses the legacy budget with a 400, a model before 4.6 refuses adaptive and keeps the budget — because pi's built-in catalog says that per model and a model behind the proxy is not in it: without the flag pi sends the budget and a Claude 5 model answers 400 on its first thinking turn, every task settling in one turn with no tokens. The completions shape carries no such switch. A task pi left before its first turn (exited, no turn) has no stream to explain it: the receipt's notes carry pi's stderr for that task — redacted, folded onto one line, its first 300 characters — so the reason is on the receipt and not only in the JSON. Every string the record carries passes redactSecrets. --print-prompt --task <name> prints the task's prompt (with a branch of its own) and exits without starting pi or needing a key: the same words go to today's coding agent through npm run cli -- ask for the other arm of the comparison.
The extension and the refusal preview. The extension (piExtension.ts) imports the pure finding schema through pi's own loader: it registers submit_pr_description and submit_verdict as the bot's own definitions — the native table's description and JSON Schema field for field, with the fresh finding schema shared directly and the remaining declarations held equal by test, so a verdict requires head and findings exactly as the relay's does (harness-pi.md item 11); pi validates the call against that schema without TypeBox — and a tool_call hook that observes and never blocks. Its one channel back is pi's notify UI request, which RPC mode writes to stdout as an extension_ui_request prefixed switchboard-pi , so a notice rides the same stream as pi's events and pairs with them by toolCallId; a dialog request (select, confirm, input, editor) is answered cancelled so no extension can hang an unattended run. The refusal preview (the coding preset's tool rules, src/core/harness/pi/toolRules.ts) stands in for the tool-level gate the authorization spec still lists as a gap and is not the policy table: each pi tool maps to a capability-profile bundle (bash → shell; read/grep/find/ls → files; write/edit → write-files; submit_pr_description → pr; submit_verdict → verdict); a bundle outside the coding preset's reach (verdict, or an unknown tool) is outside-profile; a bash push must name the checked-out task branch explicitly as both source and destination (git push origin HEAD is refused even when HEAD points at that branch); a push to a remote other than origin or a branch outside that grant is refused; write-identity shell calls with active expansion, escaped words, malformed quotes or nested shell execution refuse even without literal push words, while quoted output stays data; a bash that merges or approves a pull request by CLI or API, that reads the executor's credential store, dumps the whole environment (env/printenv alone, export -p, declare -p, a bare set, /proc/*/environ, process.env whole — never env VAR=x cmd or one variable) or expands a credential-named variable, is refused with the rule named; a file tool whose path resolves outside the checkout, or names the credential file, is refused; everything else is allowed.
The scripted provider streams, and speaks pi's tool names. A stream: true request (pi's client always streams) is answered as text/event-stream: per step, one chunk carrying the whole call or text as a delta, one carrying the finish reason, one usage-only chunk with no choices, then data: [DONE]; a request without stream is answered as before. The pi-coding profile is the coding mix in pi's vocabulary (read, bash, bash, write) and always ends by submitting a description that passes parsePrDescription before its text — with pi the terminal tool reaches the driver and never GitHub, so there is no side effect to opt out of. npm run load -- provider --profile pi-coding plus npm run load -- pi --provider scripted --model any --base-url http://127.0.0.1:8089 --key-env <any set variable> is the dry run: a real pi process, the whole protocol, no model spend.
A refusal's token is recovered from the error a client threw, never counted by message text. reasonOf (src/load/reasons.ts) scans the message for one of the named tokens the Workers fold into their errors — mirror-busy, user-pool-exhausted, disk-pressure, fleet-busy, runtime-replaced, not-onboarded, needs-ref, not-attached, rate-limited, seed-missing, seed-failed, seed-unconfigured — wherever the token sits in the message; when no token is present, message fallbacks recover fleet-busy from the sandbox capacity error, not-onboarded from the resident's 404 answer, and rate-limited from an HTTP 429 or a "rate limit" phrase, checked before the timeout rule so a throttled request that also mentions a timeout counts as rate limiting. A timeout-shaped error (TimeoutError, "timed out", "timeout") is timeout; anything else is unknown — never a message fragment.
load:route replays the one door, not a readers’ classifier (routing-and-config.md items 21 and 29). It still pages labelled historical requests and scores the checked-in compound, imperative, attach, command, write, miss, directive, planted and door fixture sets, but every live decision is one runOperator call over the full preset and command projection. The replay converts the loop’s typed bind into the existing score shape; it never calls route(), builds a router prompt, or spends a second model turn. The door row’s unit proof runs every checked-in preset fixture through bind_preset: grounded fixtures bind, including read-only compounds through conductor and write compounds through their write preset, while a review with no requester-authored PR target ends as non_decision without a reader fallback. Its no-call fixture runs two text-only turns and requires a recorded non_decision with no run or reader fallback. The provider and model flags select the door’s own model for this offline measurement. With --smoke --profile-model, the deployment config selects the resolved operator model and effort through the same completion builder as production, including its card-based reasoning allowance and configured preset projection; the General profile cannot override this selection. --smoke uses that model and the candidate source on an ordinary read, a read containing review and severity words, an existing-PR Ship review, a standalone Review request with adjacent head context, a review with requested renewals and an ordinary request with explicit run settings and a same-thread plain Fix it after a requester-authored issue and historical foreign PR citation before deployment, with the full command projection but no run store or agent execution; any wrong bind fails the command. Runtime terminal-command fulfillment uses the same configured model and shared read allowance; --verify remains a separate replay measurement. Usage counters include those calls, so command costs are not presented as one unchanged model turn.
load:pi --suite review runs the review tasks on pi's harness under the read identity, and posts nothing. The tasks (piReviewTasks.ts) are merged pull requests of the repository the checkout clones — three to five shapes a review is routinely handed — each carrying its number, its head at merge, its head branch and its title, and never the repository's name: the checkout's origin names it (originSlug), the receipt renders each pull request's URL from that, and a checkout whose origin does not hold a pinned head refuses to start. The suite is repeatable because GitHub keeps refs/pull/<n>/head after a merge. For each task the driver fetches that ref beside the base branch into a ref of its own, detaches the checkout at it and refuses to start when HEAD is not the pinned head — the resident attaches a review's worktree at the PR head the same way, and a review of the wrong head is the one thing the post-step never posts; then it writes pi's SYSTEM.md — the registry's resident review prompt composed by the run loop's own makeSystemComposer with the REVIEW TARGET block naming the task's pull request, head and base and the checkout as the worktree, the read identity's harness note (the five read tools, no edit, no write, submit_verdict the one relay), and a driver note naming the relays this driver has no bot to serve (diff_digest, update_status, web_fetch, the skill tools) — and starts pi with the read identity's allowlist (PI_REVIEW_TOOLS: pi's tools less edit and write, plus submit_verdict; no submit_pr_description), the same extension, the same budget rules with 25 minutes as the default, and the prompt a person would type (the pull request by URL). The extension's submit_verdict — the native definition, so the model is held to naming head — records the verdict on the stream and the driver keeps it in the receipt in place of the post: nothing reaches GitHub; the driver note says the head is held against the reviewed one as the post-step's guard holds it. Each call is previewed against the tool rules under identity: read (item 14's rules with the read reach: edit and write outside the profile, any push refused). The score is the post-step's judgement without the post: the verdict parsed by the same parser (parseVerdictInput), its head held against the pinned one by the same guard (checkReviewedHead), the body the post-step would have posted rendered by the same buildReviewPostBody — its first line on the receipt — and the read-only facts: no write tool ran (a write pi refused before the hook is listed and does not fail the row; one that ran does), the checkout's tree clean and its HEAD still the pinned head after the task. The receipt's checks: every task settled, every task's verdict in the house shape naming the reviewed head, every call pi ran seen by the hook, no write tool ran, the checkout untouched, no credential persisted; the notes carry one row per task with the verdict, its finding count, the head check and the write calls. Both arms run as for coding — a built-in provider with its own key, or --through-proxy with a run bearer. The dry run is npm run load -- provider --profile pi-review plus --suite review --provider scripted …: the pi-review profile reads the head and the diff with pi's bash and submits an approve whose head is a placeholder the scripted provider resolves off the REVIEW TARGET block in the request (REVIEW_HEAD_PLACEHOLDER), the way a model reads it; a request without the block leaves the placeholder and the parser drops it.
load:door counts the door's hand-backs (routing-and-config.md item 21; record 0044). npm run load -- door [--since <date>] pages the run store's finished command runs — listRuns with agent command, visibleTo: all, the list's own cursor, at most 50 pages of 200 (run-history.md item 20) — through the one runs service over a bare registry (this process drives no runs), reads each run's events, and counts a run whose route event carries outcome: hand_back under the UTC day it finished and the command it bound; a routed read (a route with no outcome) and a typed inline run (no route) are read and left out, and a legacy record carrying outcome: pasted is read and left out too — the paste counters retired with the chat hand-back (record 0069's plan, the retirement unit), so no paste counter prints on the volume line. It prints the totals, then each day with its commands under it, and a last line when the store could not be read, the counts being the live registry's alone and the exit code non-zero. It invokes nothing and spends no model tokens; it writes no receipt file: a count, not a load test.
load:intake scores the intake verdict — offline against labels, live against recoveries (routing-and-config.md item 27; record 0058). npm run load -- intake --fixtures <path> reads a labelled file — one JSON reply per line: the reply, the thread's newest turns, the facts computed by code, the ground-truth label (addressed or silent) and the stratum; a malformed row is refused naming its line, since a dropped row would move the rates — and replays every row through decideIntake itself over the router's RouteModel seam on --provider/--model (the provider built exactly as load -- route builds it; the key from the environment, refused by name without it; --concurrency at a time, no ledger, so every row is decided fresh and fail-closed outcomes are verdicts like any other). The receipt (load-results/intake-<runId>.md and .json) prints the false-silence rate over the addressed replies and the false-answer rate over the silent ones, each as k/n with its 95% Wilson interval (wilsonInterval in src/load/aggregate.ts — the sets are small, so a bare rate would overclaim; zero addressed or zero silent replies prints a stated no-denominator line, never NaN), the abstention shares over all replies (unsure — read off the reason's unsure: prefix, the one place the abstention stays visible — plus the timeouts and the errors, each failed closed to silent), the model ref line, every miss with its fixture's coordinates, and the call's p50/p95/p99 as the intake op's row (latency over the answered calls alone; a timeout or provider error counts by name). No check is configured: the labelled set cannot prove a rate at its size, so the receipt is a measurement and the live ratio is the gate. The labelled file lives OUTSIDE the repository under load-results/ (real people's words are forbidden in the tree by public hygiene; the first real run's counts are posted on the tracker, never the rows); the checked-in set (src/load/intakeFixtures.ts) is synthetic under the fixture conventions, covers the pending-confirmation and fail-closed strata, round-trips through the file format, and runs the whole scoring in CI through src/load/intakeReplay.test.ts over a scripted RouteModel. The live half: npm run load -- intake --live [--since <date>] reads the run ledger's receipts (listIntake({ since }) on the state Worker) and joins each silent one to its thread's later messages (Slack's conversations.replies over fetch, paged; the bot's own id from auth.test): the silenced reply is the thread's newest non-bot message at or before the receipt's decision — the receipt carries the thread and the instant, not the message — and the receipt counts recovered when the same person's message mentioning the bot lands within the recovery window of it (INTAKE_RECOVERY_WINDOW_MS, ten minutes: the mention is the ignored person's recovery move). It prints recovered/silent with the window named, one line per UTC week of the decisions, zero over zero stated as such, and the receipts it could not join (a non-Slack thread key, a failed thread read, a reply not found) named and kept in the denominator — the ratio is over ALL silent receipts, and an unjoinable one is never counted recovered. Each thread is read once however many receipts it holds; an addressed receipt's thread is never read. Like the door report it is printed and not judged: no model spend, no receipt file.

Historical review comparisons can supply --fixtures <JSON> (a nonempty array of named tasks with exact head, optional baseHead, and frozen context), --review-system <file> and --extension <file>. A historical base pins origin/<baseRef> as well as the checkout head. Comparison arms use the same model, effort, budget, source heads and prior context. Labels and later findings remain outside the inputs. Receipt checks establish execution and verdict shape; defect detection and closure correctness need separate source-based assessment, not a shape PASS.

load:model-capacity is a versioned offline model-proxy benchmark. The fixed team profile uses20 callers; burst uses64; each makes two sequential calls, with768KiB requests,128KiB response frames, a1s synthetic provider wait and960MiB held baseline. The runner builds the current checkout, uses a cached pinned Node image under explicit Docker CPU/memory/no-swap limits and disables external network access. Receipts record actual loaded settings, source file hashes, runtime/image/hardware/quota, workload, raw call samples, latency/queue/health percentiles, occupancy, memory, failures and final credits. JSON and Markdown files are append-only. Comparisons identify settings changes and refuse timing deltas when workload, runtime or quota differ. Missing cgroup telemetry cannot qualify a capacity claim. The default CPU/memory quota follows the bot template; a saved baseline selects its exact cached runtime image when no image override is provided. No real provider, production request, credential or deployment is used.
load:context scores retained context outcomes. npm run load -- context --fixtures <JSONL> [--baseline <receipt>] scores local synthetic or recorded answers with independent labels; fixture mode makes no provider calls. Explicit provider/model/key flags select bounded isolated synthetic trials under --context-arm raw|managed; compaction and restart are simulations, not production harness or process acceptance. A wrong or unsupported fact, missing source or retained note, repeated failed command, unknown producer answer, unconfirmed boundary or partial trace fails separately. Missing usage and prices remain unknown. Fresh JSON and Markdown receipts bind the current evaluator revision and loaded module hashes, including the load entry. A baseline must contain source provenance and its PASS/FAIL label must agree with its own outcome fields; changed workload, context, limits, model, environment, effort or qualification yields no improvement deltas.

Validation criteria

Criterion	Evidence
22: the actual load command scores local fixtures, emits bound metric receipts, returns nonzero for failed tasks or incomparable baselines, and accepts the documented context flag; inconsistent baseline PASS/FAIL labels are refused	`[unit]` `src/load/contextEvaluationCommand.test.ts::actual context load CLI::`, `::context evaluation baseline integrity::`; `src/load/contextEvaluation.test.ts::context outcome evaluation::`, `::context outcome comparison::`; `src/load/contextTrials.test.ts::local context task trials::*`
Candidate command smoke checks both `status show` and a clear typo against the real selection and fulfillment model; a substituted `plane show` fails acceptance. These calls select and check commands without executing them.	`[unit]` `src/load/candidateDoorSmoke.test.ts::fixed candidate smoke named command::*`; `[live]` Run `npm run load -- route --smoke --profile-model` with the deployment profile and trusted model origin; both named-command rows must pass.
Fixed candidate smoke supplies an explicit synthetic repository catalogue through the existing brief-reader seam. Reads return metadata only, with no source files, live heads, owners, grants or execution receipts; unknown repositories and invalid inputs remain unavailable and reader errors preserve the existing fallback. Catalogue/read copies cannot change other cases. Existing Review target and renewal cases remain intact.	`[unit]` `src/load/candidateDoorSmoke.test.ts::fixed candidate smoke repository dependency::*`
21: actual offline HTTP calls preserve bytes and return every credit; comparisons name settings changes and reject different quotas; help requires no Docker or credentials	`[unit]` `src/load/modelCapacity.test.ts::offline model capacity::*`
20: the telemetry CLI parses the documented options and reads the Analytics Engine intake report without querying the ledger, Slack or a model	`[unit]` `src/load/intakeCli.test.ts::intake A/B CLI::reads the telemetry report without touching the ledger, Slack or a model`
17: every checked-in routed preset fixture passes through the operator; a second no-call stops with `non_decision`	`[unit]` `src/load/routeReplay.test.ts::the door row owns every checked-in routed fixture::*`
1: nearest-rank percentiles; per-op stats over successful samples only; NaN with no success; refusals by reason with `unnamed`	`[unit]` `src/load/aggregate.test.ts::percentile — nearest-rank over a sorted array::`, `::summarize — per-operation latency and refusals by reason::`
2: latency and zero-reason checks pass/fail with the actual number; no samples fails	`[unit]` `src/load/aggregate.test.ts::evaluateSlo — the D10 lines::*`
3: the receipt carries parameters, the op table, refusals, checks and the verdict line	`[unit]` `src/load/aggregate.test.ts::renderMarkdown — the receipt::*`
4: tombstones dropped; sweep-line peak, first instant, per-day; half-open intervals; cursor paging bounded by maxPages	`[unit]` `src/load/history.test.ts::*`
5: staggered starts; teardown always runs with the ending error; setup/iteration/teardown failures counted; abort ends threads	`[unit]` `src/load/runLoop.test.ts::runThreads::*`
6: step = count of tool results; the OpenAI wire shape; undeclared tools skipped; profiles end in text and carry no terminal tool unless asked; the server serves `/chat/completions` and 404s the rest	`[unit]` `src/load/scriptedProvider.test.ts::*`
7: token buckets; per-channel and per-app refusals with Retry-After; lag counted from the frame's production; replies oldest-first; stats	`[unit]` `src/load/fakeSlack.test.ts::*`
8: the coding mix, read-only review, named refusals past the pool, the 0-in-flight preflight, the 16-thread guard, the purge (and a failed purge reported)	`[unit]` `src/load/residentLoad.test.ts::runResidentLoad::*`
8: the purge decision — evicted bindings under a whole non-production namespace only	`[unit]` `src/execution/bindingPurge.test.ts::selectBindingsToPurge::*`
9: one card is well-behaved; fifty on the pre-budget client lose terminal frames and starve; fifty on the budgeted client see no refusal, paint every card, land every terminal frame, and send the same edits whether or not the limits are lifted; today's seven-card shape is clean; deterministic	`[unit]` `src/load/cardsLoad.test.ts::simulateCards::*`
10: one synchronous ingress request per iteration with bearer, text and thread; run status as the outcome; transport errors; the failure pause	`[unit]` `src/load/e2eLoad.test.ts::runE2eLoad::*`
10: opt-in capacity checks conservatively bound server observation times within each probe's start-to-receipt window: span runs from the first receipt to the last probe start, client overlap covers entire windows and intervening time, and the largest possible observation gap is bounded. Every requested thread completes with healthy complete telemetry within explicit limits and one unchanged non-draining process; disjoint spans, brief peaks, queued servers, missing data, aborts or restarts fail	`[unit]` `src/load/capacityEvidence.test.ts::evaluateCapacity::*`
11: cold start as `first-exec`, `fleet-busy` named past the seats, the 25-thread guard	`[unit]` `src/load/sandboxLoad.test.ts::runSandboxLoad::*`
11: seeded — the seed before the first command with the Worker's step timings as their own ops, a refused seed counted under its token, a client without `/seed` refused	`[unit]` `src/load/sandboxLoad.test.ts::runSandboxLoad — seeded::*`
16: the seed's tokens are recovered like the resident's	`[unit]` `src/load/reasons.test.ts::reasonOf — the machine token inside a client's error::finds the seed's tokens — the handle gone, a step failed, a Worker without presigned transfer`
12: `/healthz` carries `process` when sampled; the sampler reads whole MiB and a fresh lag window	`[unit]` `src/channels/health.test.ts::healthPayload — process metrics::`, `src/channels/processMetrics.test.ts::startProcessMetrics::`
13: LF-only framing; records and notices parsed; turns, usage and cost summed once per assistant message; tool calls with outcome, hook pairing and preview; the PR-shaped judgment through the validator; the answer, event counts and the unmapped list; provider errors, retries and unknown kinds; settle only on `agent_settled`; dialogs cancelled	`[unit]` `src/core/harness/pi/protocol.test.ts::splitJsonl…::`, `::parsePiLine…::`; `src/load/piRpc.test.ts::hookNotice…::`, `::PiTaskAccumulator…::`
13: the gate's judgment per call — `vetted` when the hook saw it, `rejected-by-pi` with pi's reason when pi answered the call without running it, `bypassed` when it ran (or was still open) unseen; pi's three pre-hook answers read by name and nothing else	`[unit]` `src/load/piRpc.test.ts::PiTaskAccumulator…::judges each call against the gate…`; `src/core/harness/pi/protocol.test.ts::piAnsweredWithoutRunning…::*`
13: every documented pi event type has a home or a `null`	`[unit]` `src/load/piRpc.test.ts::PI_EVENT_HOME…::*`
13: the driver asks state and sends the prompt, answers dialogs, closes stdin on settle; a refused prompt is `error`; an ended stream is `exited`; the budget aborts, waits the grace and is `budget`; timers cancelled	`[unit]` `src/load/piRpc.test.ts::drivePiTask…::*`
13: RPC mode with every discovery off and one extension by path, the tools allowlist, the model with its thinking suffix, a session directory, never `--api-key`; the environment allowlist with the key under pi's name; the provider → variable map; the config directory (never-trust settings, a models.json for a custom endpoint)	`[unit]` `src/load/piProcess.test.ts::`; the line reader `src/core/harness/pi/protocol.test.ts::jsonlLines::`
13: five tasks of five shapes; each prompt carries the branch, the proof, the terminal tool, never merge, do not push	`[unit]` `src/load/piTasks.test.ts::*`
18: historical review fixtures validate exact heads and optional bases; an explicit comparison policy receives frozen history as untrusted source, with evaluation labels kept outside model input	`[unit]` `src/load/piReviewTasks.test.ts::parsePiReviewTasks::*`; `src/load/piReview.test.ts::reviewSystemPrompt::accepts an explicit comparison policy and treats frozen history as untrusted source`
14: two tools as the native definitions field for field — the description and the JSON Schema, `head` required of a verdict — with no TypeBox symbol; the hook reports and never blocks; the submissions and the session start reach `notify` with the prefix, the verdict's head as the model passed it	`[unit]` `src/load/piExtension.test.ts::piExtension::*`
14: bundles and the coding reach; `outside-profile` for `verdict` and unknown tools; bash pushes, merge/approve, credentials, environment dumps; paths outside the checkout; malformed inputs refused	`[unit]` `src/core/harness/pi/toolRules.test.ts::*`
15: a `stream: true` request streams three chunks per step and ends in `[DONE]`; the pi-coding profile uses pi's names and submits a valid description	`[unit]` `src/load/scriptedProvider.test.ts::chatCompletionChunks…::`, `::the pi coding profile::`, `::startScriptedProvider — the server::answers a stream: true request…`
16: the named tokens found anywhere in the message; the `fleet-busy`, `not-onboarded` and `rate-limited` fallbacks, rate limiting before the timeout rule; `timeout`; `unknown`, never a fragment	`[unit]` `src/load/reasons.test.ts::reasonOf — the machine token inside a client's error::*`
Live: `load:pi` without the key refuses to start naming the variable	`[agent]` `env -u ANTHROPIC_API_KEY npm run load -- pi --checkout . --task test-gap --model x` → exit 1, stderr `the model key must be in the environment variable ANTHROPIC_API_KEY`.
13: pi is handed its config directory, session directory and extension as absolute paths whatever the driver knew them as — it runs in the checkout, where a relative path would be looked up	`[unit]` `src/load/piProcess.test.ts::piEnv::hands pi an absolute config directory…`, `src/load/piProcess.test.ts::piArgs::hands pi an absolute session directory and extension path…`
13: the proxy's model entry carries `reasoning: true`, the 1M window and the 64k output ceiling; the dry run's scripted model stays a bare entry	`[unit]` `src/load/piProcess.test.ts::writeAgentDir::the proxy's model entry says the model can think…`, `src/load/piProcess.test.ts::writeAgentDir::the dry run's scripted model stays a bare entry…`
13: on the Anthropic shape the proxy's model entry asks a Claude 5 model for adaptive thinking and a Claude model before 4.6 for the budget payload; the completions shape carries no Anthropic compat	`[unit]` `src/load/piProcess.test.ts::writeAgentDir::the proxy's model entry asks a Claude 5 model for adaptive thinking…`, `src/load/piProcess.test.ts::writeAgentDir::the proxy's model entry asks a Claude model before 4.6 for the budget payload…`, `src/load/piProcess.test.ts::writeAgentDir::the completions shape through the proxy carries no Anthropic compat…`
13: a task pi left before its first turn carries pi's stderr in the receipt's notes — redacted, one line, its first 300 characters, `(empty)` when there was none; a task that reached a turn or ended any other way adds no note	`[unit]` `src/load/piProcess.test.ts::earlyExitNote::*`
13: `--through-proxy <bot base url>` makes the bot's model proxy pi's one provider (`switchboard`, the Anthropic messages shape, or the completions shape under `<base>/v1` with `--shape openai`) and reads a run bearer from the harness's own variable; the models file names the shape asked	`[unit]` `src/load/piProcess.test.ts::writeAgentDir::names the Anthropic messages shape for the bot's model proxy when asked…`
Live: `load:pi` through the bot's model proxy — the five tasks with a run bearer as the only key, every call metered on the borrowed run's page (harness-pi.md)	`[agent]` Start a long read-only run to borrow a bearer from (`agent:explore` on any repository: a 120-minute wall clock and a 720-turn guard), read its id from the card's Live run link; `POST $SWITCHBOARD_BASE_URL/admin/model-proxy/bearer` with `authorization: Bearer $SWITCHBOARD_INGRESS_TOKEN` and `{"runId":"<id>"}` → `201` with `bearer` and `model`; `export SWITCHBOARD_PI_MODEL_KEY=<bearer>`; in a scratch clone of this repository `npm run load -- pi --checkout <clone> --task all --through-proxy "$SWITCHBOARD_BASE_URL" --model <the bearer's bare model id>` → `load-results/pi-*.md` with five `task` rows `settled` and `PR-shaped: yes`, the borrowed run's page showing one `model.turn` per pi call with token counts, the bot's log carrying `[model-proxy] run=<id> turn=…` lines and no body; compare tokens and wall time against the direct-key arm's receipt.
Live: `load:pi` dry run — a real pi over the whole protocol with no model spend	`[agent]` `npm run load -- provider --profile pi-coding --cpu-seconds 2` in one shell; in another, in a scratch clone, `SWITCHBOARD_PI_MODEL_KEY=dry npm run load -- pi --checkout <clone> --task test-gap --provider scripted --model any --base-url http://127.0.0.1:8089 --pi <pi bin>` → `load-results/pi-*.md` with one `task` row `settled`, `PR-shaped: yes`, tool calls `read, bash, bash, write, submit_pr_description` all hook-paired, no refusal, verdict PASS.
Live: `load:resident` at N = 16 against a quiet resident produces a receipt, leaves no bindings behind, and refuses to start while the resident has work in flight	`[agent]` In a quiet window: `npm run load -- resident --resource repo:<slug> --threads 16 --hold 120 --cpu-seconds 20` → `load-results/resident-*.md` with attach/read/exec/exec-cpu/write/detach rows and `purge: 16 binding(s) removed`; `/debug threads` shows no `load:` keys afterwards. Start a real run on the same repo and re-run → `refusing to start: … op(s) in flight`.
Live: `load:sandbox` at N = 1 proves the current cold path	`[agent]` `npm run load -- sandbox --threads 1 --hold 30 --cpu-seconds 5` → `first-exec` ok, no refusals.
Live: `load:e2e` against a local bot on the scripted provider completes runs with no model spend	`[agent]` `npm run load -- provider --profile coding --cpu-seconds 10` in one shell; a bot started with a config naming `providers.scripted: {type: openai-compatible, baseUrl: http://127.0.0.1:8089}` and `defaults.models.coding: scripted/any` in another; `npm run load -- e2e --ingress-url http://127.0.0.1:8080/ingress --healthz-url http://127.0.0.1:8080/healthz --text "agent:coding in <slug>: load harness" --threads 8 --hold 120` → every `run` sample `completed`, `healthz samples` with rss and lag in the notes.
Live: `load:route` over the production run store on the one door's model — the singles table, the compound set and the imperative set in one receipt	`[agent]` `SWITCHBOARD_STATE_WORKER_URL=… MEMORY_TOKEN=… ANTHROPIC_API_KEY=… npm run load -- route --since <date> --limit 200 --provider anthropic --model <the deployment's resolved operator model id>` → the receipt's compound lines: `compound: bound conductor N/13 (…), collapsed to its write preset 7/7, decoys bound conductor 0/5` and the history line with the conductor count, the three compound checks passing; its imperative line: `imperatives: N/20 to ship (…); look-alikes to a write preset 0/11 (…)`, the two imperative checks passing; posted on the unit's board item. Human-gated.
Live: `load:route` over the production run store on the one door's model	`[agent]` `SWITCHBOARD_STATE_WORKER_URL=… MEMORY_TOKEN=… ANTHROPIC_API_KEY=… npm run load -- route --since <date> --limit 200 --provider anthropic --model <the deployment's resolved operator model id>` → the receipt's table and door misses; every call is `runOperator`, with no classifier completion.
Live: `load:route --verify` over the checked-in command set on the one door's model — the verifier's two counters, the numbers record 0044's production decision is taken off	`[agent]` `SWITCHBOARD_STATE_WORKER_URL=… MEMORY_TOKEN=… ANTHROPIC_API_KEY=… npm run load -- route --verify --limit 0 --provider anthropic --model <the deployment's resolved operator model id>` → the receipt's `verifier: removed <n> of <m> write misbinds · rejected <k> of <j> correct write binds (…)` line and every rejected bind with its reason, the command rows and the counters line beside it (the verifier's calls counted); the bar — at least half removed under one in twenty rejected — is read by the maintainer, never by the replay; posted on the record's receipts. Human-gated.
18: the review tasks are three to five pull requests, distinct by name and number, each pinned to a full head sha on `main` with its head branch and carrying no repository name; a task is found by name; the URL is rendered in whichever repository the checkout's origin is	`[unit]` `src/load/piReviewTasks.test.ts::PI_REVIEW_TASKS::*`
18: the repository is read off the checkout's origin — the https and ssh forms with or without `.git`; anything else is refused	`[unit]` `src/load/piProcess.test.ts::parseGithubSlug::*`
19: hand-backs bucketed by day and command, a legacy paste record left out, a routed read and a typed run left out; the rendering with no paste counter on the volume line and an empty store printing zeros; `sinceMs` bounds the window; the store paged to its end through the cursor; an unreachable store named on the report	`[unit]` `src/load/doorReport.test.ts::doorReport — hand-backs per day and per command::*`
20: the Wilson interval — the 95% bounds at a known k/n, clamped to [0, 1], NaN bounds over an empty denominator	`[unit]` `src/load/aggregate.test.ts::wilsonInterval — the interval a small-n rate is quoted with (load-harness item 20)::*`
20: the replay through decideIntake over a scripted model — the false-silence rate over the addressed replies and the false-answer rate over the silent ones, each with its Wilson interval at the set's n, fixture order kept, every miss listed with its coordinates, each call timed	`[unit]` `src/load/intakeReplay.test.ts::replayIntake + intakeScore — the two conditional rates on a known set::*`
20: the abstention shares (unsure, timeout, error — each failed closed to silent and counted as a silence), the model ref line, the rates printed as percentages with the interval, and the stated no-denominator line over zero addressed replies — never NaN	`[unit]` `src/load/intakeReplay.test.ts::intakeScore + renderIntake — the abstention shares and the model ref line::*`
20: the labelled file's format — one JSON reply per line, blank lines skipped, a malformed row refused naming its line — and the synthetic set: both labels, the pending-confirmation and fail-closed strata, unique synthetic ids under the fixture conventions, a round-trip through the format	`[unit]` `src/load/intakeReplay.test.ts::the fixture file's rows and the synthetic set::*`
20: the live ratio — a silent receipt recovered by the same person's mention of the bot within the window (one of two prints one half), an addressed receipt never read, each thread read once, zero silent receipts printed as 0/0, per-week lines, an unjoinable receipt kept in the denominator and named	`[unit]` `src/load/intakeReplay.test.ts::liveFalseSilence — the live ratio over the receipts and the threads' later mentions::*`
Live: `load:intake` over the labelled file on the deployment's fast model — the counts on the tracker, never the rows	`[agent]` `ANTHROPIC_API_KEY=… npm run load -- intake --fixtures <the labelled file> --provider anthropic --model <the deployment's defaults.models.general id>` → the receipt's two rates with intervals, the abstention shares and the model ref line; the counts posted on the tracker. Human-gated.
Live: `load:intake --live` over the production receipts — the weekly false-silence ratio the gate reads	`[agent]` `SWITCHBOARD_STATE_WORKER_URL=… MEMORY_TOKEN=… SLACK_BOT_TOKEN=… npm run load -- intake --live --since <date>` → `live false-silence ratio: k/n (…)` with one line per week, posted on the tracker weekly. Human-gated.
19: refused route records counted per day, per cause and per code (record 0054) — an empty store printing zero refusal lines	`[unit]` `src/load/doorReport.test.ts::doorReport — hand-backs per day and per command::counts refused records per day, per cause and per code (record 0054) — two days, three codes, two causes — and an empty store prints zero refusal lines`
19: the report counts every refusal from the run store alone (record 0054, as amended): the dispatcher's `door` records and the refused route outcomes together, per day, cause and code — a gate refusal before any bind included, a code the table does not know bucketed as `unknown`, and no footer pointing at a telemetry query	`[unit]` `src/load/doorReport.test.ts::doorReport — hand-backs per day and per command::counts \`door` records and refused route outcomes together — a gate refusal before any bind counts by its day, cause and code, a code the table does not know counts as `unknown`, and no footer names a telemetry query`
18: the system prompt is the registry's resident review prompt composed as a review run composes it — the REVIEW TARGET block with the task's coordinates in the checkout's repository, the read identity's harness note naming the relayed verdict tool, the driver's note on the absent relays; the prompt is the pull request by URL	`[unit]` `src/load/piReview.test.ts::reviewSystemPrompt::*`
18: a task is judged as the post-step would judge it — a house-shaped verdict naming the reviewed head renders the `LGTM:` body with its findings, `request_changes` the `Changes requested:` line, no verdict the no-verdict line and a problem; another head or none fails the head check the guard would fail; a rejected verdict is not the house shape; every write tool asked for is listed by call with the gate's judgement	`[unit]` `src/load/piReview.test.ts::reviewOutcome — one task judged as the post-step would judge it::*`
18: the receipt's rows — every row passes for a settled task with a house-shaped verdict on the reviewed head, every call vetted, nothing written and the checkout untouched; an unsettled task, a verdict off the shape or the head, a bypassed call, a write that ran, a dirtied or moved checkout each fail their row naming the task; a write pi refused before the hook does not fail the write row	`[unit]` `src/load/piReview.test.ts::reviewChecks — the receipt's verdict rows::*`
18: the review child's allowlist is the read identity's plus the verdict tool alone; the checkout fetches the pull request's head into a ref of the driver's own beside the base branch and detaches at it	`[unit]` `src/load/piProcess.test.ts::piArgs::the review child's allowlist is the read identity's — pi's tools less edit and write — plus the verdict tool alone`, `src/load/piProcess.test.ts::the review suite's checkout::*`
18: the `pi-review` profile reads the head and the diff with pi's bash, submits a house-shaped verdict whose head is the placeholder, and ends with text; the placeholder is resolved off the REVIEW TARGET block in the request, plain and streamed, and left — then dropped by the parser — without one	`[unit]` `src/load/scriptedProvider.test.ts::the pi review profile::*`
Live: `load:pi --suite review` dry run — a real pi over the whole protocol under the read allowlist with no model spend	`[agent]` `npm run load -- provider --profile pi-review --cpu-seconds 1 --port 8091` in one shell; in another, in a scratch full clone, `SWITCHBOARD_PI_MODEL_KEY=dry npm run load -- pi --suite review --checkout <clone> --task docs-line --provider scripted --model any --base-url http://127.0.0.1:8091` → `load-results/pi-review-*.md` with one `task` row `settled`, verdict `approve, 1 finding(s)`, head `reviewed`, write calls `none`, checkout `untouched`, tool calls `bash, bash, submit_verdict` all vetted, verdict PASS.
Live: `load:pi --suite review` through the bot's model proxy — the review tasks with a run bearer as the only key, verdicts recorded and posted nowhere (harness-pi.md item 10)	`[agent]` Borrow a bearer as the through-proxy row above does; `SWITCHBOARD_PI_MODEL_KEY=<bearer> npm run load -- pi --suite review --checkout <clone> --task all --provider anthropic --model <id> --through-proxy <bot base url>` → every task `settled`, every verdict in the house shape naming the reviewed head, every call vetted, `no write tool ran`, the checkout untouched, no credential persisted, verdict PASS; the pull requests carry no new comment; the borrowed run's page carries the `model.turn` spans. The direct-key arm is the same command without `--through-proxy` and with `--key-env`.
17: the configured smoke and production operator share model and effort resolution, including an independent light profile, the card-mapped wire effort and reasoning-aware output budget; legacy config remains supported	`[unit]` `src/core/dispatch/operatorCompletion.test.ts::operatorCompletion::*`
20: `load intake --live --experiment ID` reports persisted A/B events, distinct threads, decisions, failures, pooled P50/P90/P95 latency, known token cost and missing-price/usage calls by arm and model; the actual CLI accepts the experiment ID and sample flags, and optional samples retain message coordinates without message text; it spends no model tokens and reads no Slack	`[unit]` `src/intakeExperiment.test.ts::intake A/B::reports all measured outcomes, unknown costs and pooled latency by arm`; `src/load/intakeCli.test.ts::intake A/B CLI::accepts the documented experiment and samples flags and reads only the ledger`

Short production A/B run

The experiment compares the current agents.intake model with Jev on unmentioned thread replies. Direct mentions, pending questions and existing receipts keep their normal paths. Each thread keeps its arm across restarts; only the selected model runs. Authorization and run execution stay downstream. The experiment applies only where the effective thread-reply mode is classify.

Provision the optional TYPESAFE_API_KEY on the bot through the normal secret deployment. Add this block to the installation config and restart the bot:

yaml

intake:
  experiment:
    id: intake-jev
    model: typesafe/jev-1.13.0
    percent: 50

Record the start time. After roughly two hours of ordinary traffic, read the results with the existing state URL and state credential:

npm run load -- intake --live --experiment intake-jev --since <start-time>

For a longer observation, intake measurements also go to the deployment's existing Analytics Engine dataset. Both addressed and silent receipts emit their own intake-1 point after insertion; a duplicate receipt emits nothing. The upgraded client opts in only after its run-metrics reader isolates the run schema. This keeps the memory-first rollout compatible with older bots. Telemetry failures never change the routing decision. Collection runs in production and needs no open chat, local export job or model call.

npm run load -- intake --telemetry --experiment intake-jev \
  --account <account-id> --dataset <metrics-dataset> --since <start-time> \
  --until <end-time> --samples --token-env CF_ANALYTICS_TOKEN

The token needs Analytics Engine read access. This report reads only the lake; it never silently falls back to the short-lived receipt store. Counts, token sums and percentiles use sampling weights. threadsObserved counts distinct threads in the returned observations and is a lower bound if sampled is true. Probe rows are excluded by default; --include-probes is for operator proof. Use --exclude-threads <key,key> to exclude explicitly generated Slack tests. Samples are limited to 200 rows by default (--limit permits 1–1000); they contain coordinates and scores, never message text. Missing confidence or probabilities are stored as -1. Existing Analytics Engine retention applies, independently of receipt cleanup; data is not backfilled from older receipts.

The JSON reports event and thread counts, addressed/silent decisions, errors, timeouts, P50/P90/P95 decision latency, known estimated token cost and unpriced calls by arm and model. Latency includes failed calls and repairs; it measures classification, excluding Slack context retrieval and agent execution. Cost per 1,000 events is null if any call is unpriced. Configure the Jev rate in the existing costs.prices table to estimate dollars:

yaml

typesafe/jev-1.13.0: { input: 0.042, output: 0, cacheRead: 0, cacheWrite: 0 }

Those are input/output/cache USD per million tokens, not an invoice receipt. Check the current TypeSafe price before running. --samples adds message coordinates and probabilities without message text, for a quick review of addressed and silent decisions in Slack. Counts and confidence alone do not measure correctness. Review messages from both arms, including silent ones; use the existing live recovery report as an additional signal.

Set percent: 0 and restart to stop Jev calls while retaining the control measurements. Remove the experiment block and restart to finish measurement; the stored results remain readable. Change id for another run rather than combining differently configured trials. A short, low-volume run may establish speed or cost differences without enough evidence to decide routing quality.

A single deployed shadow probe

Before enabling a live split, an operator with deploy:write may send one POST /admin/reply-probe request to the bot. Supply model and message as JSON. The request uses the serving bot's key and the same intake code, writes a synthetic receipt under probe:<operator>:<id>, and returns its experiment ID for the existing report command. It never dispatches the verdict or changes live configuration. This proves the deployed provider/store path, not Slack event delivery or live classification accuracy.

json

{ "model": "typesafe/jev-1.13.0", "message": "Please explain this deployment failure." }

The response includes apiCalls, persisted, decision and liveRoutingChanged: false. Check that the expected verdict came from the model, the receipt was inserted and persisted, and only one API call occurred. Use the returned experiment ID with load intake --live --experiment ID to verify reporting. Unknown or failed model outcomes are data, not successful functional proof. No key appears in the request or response.

| 20: synthetic probe: receipts remain available to the experiment report but are excluded from the live false-silence denominator and weekly recovery buckets | [unit] src/load/intakeReplay.test.ts::liveFalseSilence — the live ratio over the receipts and the threads' later mentions::excludes synthetic probe silences from the live denominator and weekly buckets |

Canonical Home scale

npm run load -- home --units 1000 --pipelines 1000 --iterations 25 measures the real local Home HTTP view over synthetic canonical work. It checks complete discovery and p95 first-attention 500 ms / complete-list 2 s budgets. --browser also measures the built replacement frontend on the screenshot registry's desktop and phone profiles in both themes, with a 1 s p95 first-attention budget and last-row reachability in the virtual list. Browser mode runs the existing web build from this frozen checkout before timing; unrelated bundles are refused. Browser contexts admit local fixture HTTP requests only. First and last rows must intersect both the browser viewport and scroll list and pass hit testing, rather than merely have CSS visibility.

The checked-in evidence fixture sets a minimum of 1000 units and can raise it to twice the recorded largest complete authorized fleet census. Its current census is explicitly unverified; minimum-scale local results do not qualify that separate requirement or Cloudflare network latency. The measured path is local canonical inventory/render; private native authentication, source-epoch and answer-enrichment costs are explicitly unmeasured. Receipts bind source HEAD, dirty diff and untracked-content fingerprints plus the actual served asset hashes, rechecked after measurement; a changed source or bundle refuses the receipt. Fixtures create no live work and use no provider credentials. The existing JSON/Markdown receipts carry measured counts, latency, failures and qualification notes.

Criterion	Proof
One thousand owned unresolved units remain complete and addressed across one thousand pipelines through the actual Home HTTP endpoint	`[unit]` `src/load/homeLoad.test.ts::canonical Home load::enumerates one thousand owned units through real HTTP without losing cross-pipeline attention`
Under-sized fixtures and insufficient iterations are refused before measurement	`[unit]` `src/load/homeLoad.test.ts::canonical Home load::refuses under-sized or insufficient measurements before starting a server`
Local canonical inventory/render latency budgets and pinned-browser virtualized last-row reachability hold on the fingerprinted candidate	`[live]` Run both Home load modes above at the tested head; require a successful receipt for each server and browser operation. This is synthetic local proof.
The fixture is at least twice the largest complete authorized fleet census	`[gap]` Record an authorized canonical inventory census before release; legacy Plane active/recent windows are insufficient.

| The receipt names the exact source and excludes unmeasured private authentication costs; unrelated browser bundles are refused | [unit] src/load/homeLoad.test.ts::canonical Home load::enumerates one thousand owned units through real HTTP without losing cross-pipeline attention, src/load/homeLoad.test.ts::canonical Home load::refuses a browser bundle outside the current checkout instead of qualifying unrelated assets | | Clipped, offscreen and covered attention cannot pass browser reachability checks | [unit] src/load/homeLoad.test.ts::canonical Home load::requires attention inside both the browser and scroll list without an overlay covering it |

Load harness ​

Behavior ​

Validation criteria ​

Short production A/B run ​

A single deployed shadow probe ​

Canonical Home scale ​