Switchboard

Appearance

Run-friction analyzer ​

Switchboard can diagnose what cost a run time or made it stumble from its own run-event stream — slow model turns, slow or failed tool calls, retries, setup/install time, wrap-up, budget hits, and exec-infrastructure failures — as a structured, deterministic diagnosis. This is the observe → diagnose half of the self-improvement loop. Everything in this feature is read-only analysis; the propose half — clustering these diagnoses across recent runs and filing deduped issue proposals — is self-improvement.md.

Pure and deterministic (locked design). analyzeRunFriction(events, opts?) is a function of the event stream alone — no clock, no I/O, no mutation of its input. The same events always yield a deep-equal diagnosis, so it runs identically over a live backlog, a saved capture, or a test fixture. That property is what will later let a fix-proposer be tested against recorded runs.

Signals ride the existing stream (locked design). Rather than a second telemetry channel, the run-visibility RunEvent stream (run-visibility.md) was extended additively: every event may carry at (epoch ms, from the runner's injectable clock); a tool_result may carry infra: true when the failure was an ExecInfraError (the sandbox/transport, not the command); and a new run_note variant carries the runner's lifecycle notices as typed kinds — wrap_up, time_budget_exhausted, turn_budget_exhausted, sandbox_dead — alongside the unchanged free-text onProgress. Consumers that only know tool_call/tool_result keep working; the Slack card and live page render notes as a ⏱ line. Redaction is unchanged: findings derive only from already-redacted summaries.

Behavior ​

A system refusal is an infrastructure failure even when setup ended before any tool or model turn. Policy and request refusals remain separate from infrastructure findings. Refusals remain side facts for step counts.

  1. Categories. Each finding has exactly one FrictionCategory: slow_tool (a tool.* span taking ≥ slowToolMs, default 30 s; high at ≥ 2×), slow_model_turn (a model.turn span taking ≥ slowModelTurnMs, default 60 s; high at ≥ 2×; anchored to the turn's own span_end and naming what it produced, e.g. model turn took 3m 42s before: $ grep …, (narration), (answer); a 31-min run with 50 s of tool time and 35 one-line greps must not diagnose as no friction detected), failed_tool (ok:false, incl. an unknown tool; medium, or high when the failure also took ≥ slowToolMs), retry (the same tool + call summary re-issued after that exact call previously failed; counted per re-issue), setup_install (a bash command that is a package-manager install / git clone / similar — incl. python -m pip install and corepack <pm> wrappers — matched at the start of a shell segment with the subcommand word ending there and quoted strings blanked first, so cd x && npm install counts while echo npm install, echo "cd x && npm install", npm ci-lockfile-report, and npm test do not), wrap_up (the runner's wrap-up note; its extent is from the note to the end of the window), budget_hit (time- or turn-budget exhaustion note, high; no extent), infra_failure (an infra:true result — the finding names the command that was running — the sandbox_dead and fleet_busy notes (no extent), or — in a finished stream — a tool_call with no result, i.e. the run died mid-tool, whose extent is the open tool span's; all high but fleet_busy), drain_wait (the run was admitted onto a drained resident fleet and waited at its attach for the deploy to finish — the dispatcher's drain_wait note off the binding's drainWaitMs, resident-repos.md item 69, issue 2044; medium, no extent — a half-hour wait behind a verdict of no friction detected was the incident's shape), unkept_promise (the run's answer says a file is attached — "attached below/here/above", "see (the) attached", "I've/I have attached", "is/are attached", "attachment(s) below/here", one case-insensitive expression, ATTACHMENT_PROMISE — while the run recorded no outbound artifact event; low, once per run, anchored to the answer, no extent, quoting the line that made the promise; a path such as ./attachments/1-clip.mp4 is a place, not a promise, and prose in an assistant turn is not the reply — a preset without attach_file makes no call that could fail, so the answer's words are the only witness). Every duration comes from ONE span set (tracing.md item 14): the stream normalized by normalizeSpans — a tool pair whose twin span the record budget dropped gets it back from the pair's own stamps — so a finding carries the duration of the span it is about, and a stream with no model.turn spans has no model time and no slow_model_turn, whatever the gaps between its content events. Results pair with pending calls by callId, not completion order. A replayed start with the same ID after reattach replaces the pending start for that logical call. An unmatched keyed result never consumes a different call; unkeyed legacy pairs use FIFO among unkeyed calls of that tool. Failure paths, retries and missing-result findings use the matched call.

  2. Precedence — one finding per event. infra beats everything (a dead sandbox is not a failing command); setup_install beats failed_tool/slow_tool (a slow or failed install is still setup cost, so the category total is the true install bill; a failed install is high); then failed_tool; then slow_tool. retry is anchored to the re-issued call and is additional to whatever its result yields.

  3. Timings are optional, and there is one code path. A stream is timed when a content event carried at or a window was given: runMs — the run's window (receivedAt → finishedAt, or now while live) when the caller passes one, else first→last content stamp (a stdin capture); toolTimeMs (the sum of the tool.* spans); modelTimeMs (the union of the model.turn intervals, including recorded routing model phases, present only when the stream had a turn); per-finding durationMs; per-category durationMs — the sum of the findings' durations for the tool-denominated categories, the union of the findings' intervals for every run-denominated category, including slow_model_turn, wrap_up, budget_hit and infra_failure (three calls of one batch dying together are one interval, not three). DENOMINATOR_OF names what each category's time is a share of — tool time for slow_tool/failed_tool/retry/setup_install, run time for the rest — so no share can exceed 100 %. For an agent run thinking ≤ model time and tools ≤ tool time hold by construction (a union is at most a sum over the same span set); a command run's tools bucket is the command's own work; absent tool or model activity produces no corresponding totals line. Without timings — a hand-written capture, a record from before spans — nothing is special-cased: the same classification, and every timed field is simply absent (a pair without a stamp gets no span, so no duration; runMs, toolTimeMs, modelTimeMs are absent; the report prints - in the time column and no run/tool time/model time totals). The shape (tracing.md item 5): with a window and finished, the diagnosis carries shape — the window's seven terms from partition over the same span set (windowMs, gettingReadyMs, thinkingMs, toolsMs, finishingUpMs, overheadMs, notRecordedMs, notLoadedMs), owner deciding whether run.command is tools (a command run) or getting ready; the finish-site diagnosis is what the run record stores and the closed Slack card's shape line reads, so the card, the record and the report agree. Live, or without a window, there is no shape. Every duration surface prints through formatDuration (report style, hours included).

  4. Verdict. One line naming the dominant cause: the category with the most attributed time (ties → most findings → category order), with its share of its denominator (DENOMINATOR_OF: … (60% of run time) for the run-denominated categories, of tool time for the tool ones); without timings, the category with the most findings; no friction detected when there are no findings. The verdict names the dominant friction category and the shape names the dominant bucket — two lenses on one span set, printed one under the other. Labels are one vocabulary: slow tool calls, slow model turns, failed tool calls, retries, the repo's setup/install, agent wind-down, budget hits, infra failures, the fleet drain, unkept promises — the verdict, the report's category table (column sized to the longest), its finding lines and the friction proposals' titles all print them, never the category ids. byCategory always contains every category (zeroed), so consumers need no undefined checks. Findings are in stream order, each anchored to an eventIndex. The text report prints events, then tool calls and tool time only when a tool ran, run when timed, model time only when the model turned, (input truncated — some records were dropped before analysis) when the input lost records, and shape: <line> beneath the totals when the diagnosis has one (<total> (one bucket) when fewer than two buckets are informative).

  5. Mid-run vs finished. opts.finished (default true) controls whether a trailing unpaired tool_call is friction: in a finished stream it is an infra_failure ("no result … the run ended mid-tool"); in a live one it is simply still running.

  6. Read-only surfaces. GET /runs/:id/friction?t=<token> returns { id, finished, diagnosis } as JSON (no-store, GET-only → 405) for any run still in the registry — live or finished within the TTL — behind the same constant-time capability-token gate as the page and stream (wrong/missing token or unknown run → 404, existence never revealed). npx tsx src/cli.ts friction analyze [source] [--slow-ms n] [--in-progress] [--json] (CLI only; stdin for -) analyzes a saved stream: JSON lines of RunEvents, or a raw SSE capture of /runs/:id/events (curl output — data: frames; retry:/event:/comment lines and the end payload are skipped). Input is external, so each line is shape-checked (string tool/summary, boolean ok, known kind, numeric at when present) — a recognized type alone is not enough; malformed lines are skipped and counted, never fatal (the analyzer itself also tolerates a non-string summary); zero events → exit 1. --in-progress marks a mid-run capture (finished:false) so a trailing unpaired call is not misread as a dead run; without the flag, a stream that ends on an unpaired call gets a stderr hint pointing at --in-progress. Neither surface mutates anything.

Validation criteria ​

An interrupted tool result (cut:true) records an unconfirmed outcome. It takes infrastructure precedence over command failure, setup and slow-call classifications. It does not prove that the command failed and does not make a later matching call a retry. Its interval uses the same union as other infrastructure findings. Storage failure notes remain separate evidence of the storage error.

CriterionEvidence
Empty stream → zeroed diagnosis, no findings, no friction detected[unit] src/core/runFriction.test.ts::analyzeRunFriction — empty / untimed input::an empty stream yields a clean, zeroed diagnosis…
Cut tool results retain their unconfirmed outcome and concurrent interval union beside storage failures, for live and finished runs; actual failures and retries remain visible[unit] src/core/runFriction.test.ts::keeps storage failures and cut tool outcomes separate from command failures, finished=%s; src/core/runFriction.test.ts::does not treat a cut result as a command failure or retry, but retains a real failure and retry
Clean fast run → no findings; runMs/toolTimeMs computed[unit] ::a clean, fast run has no findings and a no-friction verdict
Untimed stream (no stamps) still classifies, with no durations and no timed fields[unit] ::untimed events (a hand-written capture) still classify failures/retries, with no durations
Deterministic; input not mutated[unit] ::is deterministic: the same stream yields a deep-equal diagnosis, ::does not mutate its input
slow_tool at threshold, high at 2×, not flagged under[unit] ::slow_tool: a tool call whose result arrives past the slow threshold, ::slow_tool: ≥2× the threshold is high severity; under the threshold is not flagged
slow_model_turn at threshold (default 60 s, configurable), high at 2×, not flagged under; anchored to its model.turn span end and naming the produced event; modelTimeMs uses the interval union[unit] analyzeRunFriction — slow_model_turn…::flags a model turn at or past the threshold…, ::≥2× the threshold is high severity; the threshold is configurable
A narrating completion is one turn naming (narration); the answer is the last turn's product; a stream without model.turn spans has no model time and no slow turn whatever its gaps[unit] ::a narrating completion is one turn…, ::the final answer is the last model turn's product…
Untimed streams: no slow_model_turn, no modelTimeMs[unit] ::untimed streams never produce slow_model_turn and carry no modelTimeMs
Verdict share for dominant model time is of run time, never tool time[unit] ::the verdict for dominant model time is a share of RUN time (tool time would exceed 100%)
failed_tool incl. unknown tool; high when the failure was also slow[unit] ::failed_tool: an ok:false result (including an unknown tool), ::failed_tool severity scales with how long the failure took
Concurrent reads finishing out of order name the failed path and retry; resumed starts do not leave a completed call pending; orphan and legacy results never consume unrelated keyed calls[unit] src/core/runFriction.test.ts::analyzeRunFriction — per-category classification::pairs concurrent reads by call ID when results finish out of order, ::a resumed start with the same call ID completes the original logical call, ::an orphan keyed result never consumes another call of the same tool, ::unkeyed legacy results pair only with unkeyed calls
Analyzer tolerates a non-string summary from an external capture[unit] ::tolerates malformed field types from an external capture (non-string summary/tool) without throwing
infra_failure finding names the command that was running[unit] ::infra_failure: an infra-flagged tool result is an infra failure (asserts the call summary is in the finding)
retry only after a failure of the same call; repeats without failure are not retries[unit] ::retry: the same call re-issued after it failed…, ::retry: a repeated call that never failed is NOT a retry (red-verified: disabling the failed-call check fails these)
setup_install classification, totals, precedence over slow/failed, package-manager coverage, segment-start matching[unit] ::setup_install: install/clone commands are classified with their total time…, ::setup_install: a FAILED install is classified as setup (high)…, ::setup_install: recognizes the common package managers… (red-verified: disabling the install branch fails these)
wrap_up with wind-down duration; budget_hit for both budget kinds; infra_failure for infra:true and sandbox_dead[unit] ::wrap_up: …, ::budget_hit: …, ::infra_failure: an infra-flagged tool result…, ::infra_failure: the sandbox_dead note…
drain_wait: a drain_wait note is its own category at medium, its summary quoted, so a run that sat at the fleet drain never diagnoses no friction detected; byCategory lists it zeroed[unit] ::analyzeRunFriction — per-category classification::drain_wait: a run admitted onto a drained fleet has its wait named as the drain's own category, never a verdict of none (issue 2044)
unkept_promise: a reply that promises a file with no outbound artifact event is one low finding anchored to the answer and quoting the line; every promising phrase is recognized once; an outbound artifact, a reply without a promise, an attachments/ path and an assistant turn's prose do not fire it[unit] ::analyzeRunFriction — unkept_promise (the reply names an attachment the run never produced)::*
Finished stream: unpaired trailing call → infra_failure; unfinished: not flagged[unit] ::an unpaired trailing tool_call (run died mid-tool) is reported as an infra failure, ::an unpaired trailing tool_call in an UNFINISHED stream (finished:false) is not flagged
Verdict = most time (with share) / most findings without timings; byCategory complete[unit] ::the verdict names the category that cost the most time…, ::with no timings the verdict falls back…, ::byCategory always lists every category
Text report renders verdict, totals (incl. model time), category table, findings[unit] formatFrictionReport::*
The bridge stamps at from the harness's clock on every event[unit] src/core/harness/pi/bridge.test.ts::the bridge speaks the loop's vocabulary — the same turn, the same events::tool_call, tool_result and the model's narration are the events the native loop emitted for the same turn
Runner marks ExecInfraError results infra:true and nothing else[unit] ::marks an ExecInfraError result with infra:true; an ordinary tool error is NOT marked
Runner emits run_note for wrap-up, turn/time budget exhaustion, sandbox death[unit] ::emits a run_note for the wrap-up warning and for turn-budget exhaustion, ::emits time_budget_exhausted…, ::emits sandbox_dead…
Existing consumers unaffected: card + registry receive notes; page renders them[unit] src/core/dispatcher.test.ts::live run-view wiring…::registers the run, publishes its events, and finishes it (asserts the note reaches the registry)
snapshot is token-gated, returns a copy + finished flag[unit] src/core/runRegistry.test.ts::snapshot — token-gated read of a run's backlog …::*
/runs/:id/friction: route parsed; 404 on bad/missing token; JSON diagnosis (no-store, 405 non-GET); mid-run finished:false[unit] src/channels/liveView.test.ts::GET /runs/:id/friction…::*
parseRunEventLines parses JSONL and raw SSE captures, skips garbage AND wrong-shaped events; friction analyze [source] [--slow-ms n] [--in-progress] reads a file or stdin, renders the report + skipped count, --slow-ms/--in-progress reach the analyzer, a missing file is not_found, a stream without events invalid_input, and the command is CLI-only[unit] src/core/runEventLines.test.ts::*, src/core/commands/friction.test.ts::friction.analyze (CLI only) — the former frictionCli::*
The --in-progress hint appears only when a default analysis blames a trailing unpaired call[unit] src/core/commands/friction.test.ts::friction.analyze (CLI only)…::--slow-ms and --in-progress reach the analyzer…
CLI end-to-end over a saved SSE capture prints a report[agent] npx tsx src/cli.ts friction analyze <capture.sse> on a curl-saved /runs/:id/events stream → a verdict: line, totals, category table, findings.
Friction JSON of a real deployed run via the live link[gap] Open <live link> with /events replaced by /friction on a finished run within the TTL → 200 JSON with a plausible verdict.
3: with a finished window runMs is the window, model/tool time are the spans' sums, shape partitions the window and its identity holds; the slow turn anchors to its span end and names the call it produced[unit] src/core/runFriction.test.ts::analyzeRunFriction — the window, the shape and the span set (docs/reference/specs/tracing.md)::with a finished window…
3: no window (a capture) → no shape, runMs first→last; live with a window → no shape; an open turn on a live read counts to the window's end[unit] ::without a window (a stdin capture)…, ::a live read: an open model turn runs to the window's end…
3: a command run's run.command is its tools; the same stream as an agent run is getting ready[unit] ::a command run's window: \run.command` is its tools`
3–4: run-denominated categories take the union of their findings' intervals (a batch dying together is one interval), DENOMINATOR_OF covers every category, the share never exceeds 100 %; wrap-up runs to the window's end, a budget hit prints -[unit] ::run-denominated categories take the UNION…, ::wrap-up runs from the warning to the window's end…, ::every category names its denominator
4: the report's shape line, the gated totals lines, the labels[unit] ::the report: the shape line under the totals…, formatFrictionReport::renders the verdict, totals, a per-category table, and each finding on one line
1, 3: differential — a record whose tool.* twins the budget dropped is timed like the full stream (the pairs' own stamps); a record with no span records at all is a content stream: the window, the pairs' tool time, no model time; a model.turn span is not a step and never moves the clock[unit] ::differential: a record whose tool.* twins the budget dropped…, src/core/runFriction.test.ts::analyzeRunFriction — narrative events are not steps…::a \model.turn` span is not a step…`
3: the record's diagnosis is the finish-site one over the run's window and carries the shape; /runs/:id/friction and the service pass the window and the live schema[unit] src/core/dispatcher.test.ts::friction diagnosis reads the registry backlog …::the run record's diagnosis — what the friction ledger reads…, src/core/runsService.test.ts::RunsService.getRunFriction::passes the injected analyzer the live events and finished flag

| Routing deadlines produce budget friction, with the active phase when recorded. Recorded routing model and context intervals contribute to thinking and preparation without inventing historical model time. | [unit] src/core/runFriction.test.ts::attributes recorded routing phases and diagnoses a routing deadline | | Concurrent routing and agent slow-model intervals count each wall-clock instant once in run-denominated totals, while preserving each individual finding; context, helper and catalogue phases are preparation. | [unit] src/core/runFriction.test.ts::unions overlapping routing and agent slow-model intervals in … | | System setup refusals produce an infrastructure finding; policy and request refusals do not. | [unit] src/core/runFriction.test.ts::analyzeRunFriction…::reports system setup refusals as infrastructure failures while keeping policy and request refusals separate |

CriterionProof
A call-linked tool_input update is a side fact. It adds no tool execution or friction event and does not change the original call/result interval. The event reader retains complete updates and skips malformed identity or input fields.[unit] src/core/runFriction.test.ts::treats late argument evidence as a side fact without another tool execution; src/core/runEventLines.test.ts::parseRunEventLines::keeps call-linked argument updates and rejects incomplete update frames