Switchboard

Appearance

Resident repo environments ​

A registered repository has a primary resident and can request additional warm residents: a dedicated resident Worker (switchboard-resident) fronts per-resource Durable Objects on Cloudflare Sandbox 1.0 (@cloudflare/sandbox@next, exact-pinned). Residency is a generic resource-typed primitive — every route carries a resource id of the form <type>:<id> (repo:<owner>/<name> is the first and only type; the GitHub clone URL derives from the id), and each resident member has its own Durable Object address. The primary address is the resource id. This file covers the service surface (auth scopes, the atomic cap, onboard/offboard/reconfigure/status/residents, x-env hardening), the lifecycle engine (schedule-driven provisioning — the one timer a resident has — wake-path rehydration, the refresh cycle as a Workflow instance, the watchdog, stamped snapshots, repo-scoped GitHub token minting), the operator data plane (/attach /exec /read /write: per-thread worktrees off the bare mirror, ref binding, dep materialization, OS-user isolation, the mirror mutex, inactivity eviction), the bot-side wiring (ResidentExecutor, warm-gated selection, the per-repo gate, named fallback with a probe circuit breaker), the dispatch integration (pre-model repo/ref resolution, the needs-ref ask-once flow, resident prompt variants), the management surface (repo onboard/offboard/reconfigure/rebuild/list chat commands behind the fail-closed canManageRepos gate, --dry-run itemized plans, the /rebuild route, the onboard-time installation-membership check, and the watchdog's auto-rebuild), and the deterministic-ops surface (the /op route running readonly command-table entries in disposable per-op checkouts, per-entry effects profiles, the bot-side Operations seam with resident-backed and local implementations, and the dispatcher's modelless fast-path with repo test/build commands and conservative natural-language recognition), and residency garbage collection (items 45–46: finished-ref reclamation polled from the refresh cycle, and opt-in resident-level LRU eviction past the cap).

  • Code: web/green/pages/Runtime.vue, web/green/pages/RuntimeDetail.vue; src/execution/residentMembers.ts (pool membership and capacity), deploy/cloudflare-resident/replicaMigration.ts (maintenance import), src/execution/residentRoute.ts (saved replica address validation), src/execution/residentPlacement.ts (immutable thread placement), src/execution/repoRegistration.ts (metadata registration and resident capacity), deploy/cloudflare-resident/workspacePreservation.ts (the shared automatic removal decision), deploy/cloudflare-resident/ (worker.ts — the entry: the routes, both Durable Objects and the watchdog; refresh.ts — the refresh cycle as a Workflow instance, item 7; shared.ts — the tunables, stub factories and tracer both read; wrangler.jsonc, Dockerfile); bot side: src/execution/resident.ts (client + ResidentOperations), src/residentArchiveCli.ts (operator capture and offline restore), src/execution/residentArchive.ts (sealed archive and verifier), src/execution/residentArchiveRemote.py (fixed old-route probe), src/execution/residentWake.ts (the wake wait's pure decisions, item 65; saysContainerGone and containerGoneMessage, the answers that say the container under the thread is gone, items 43 and 27), src/execution/residentReuse.ts (the reusing attach's pure decision, item 66), src/execution/residentRebind.ts (the move onto the thread's own pull request branch and the move back to the default when that branch is gone, pure, item 16), src/execution/residentCleanliness.ts (the one-spawn tree measurement and what a release leaves behind, items 16a and 16b), src/execution/executor.ts (LocalOperations), src/execution/factory.ts (selection + circuit breaker; reattachWorkspace, item 66), src/config.ts (canUseRepo, canManageRepos), src/core/repoContext.ts (repo/ref resolver), src/core/residentAdmin.ts (the resident admin client + parseSlug/validRef), src/core/commands/repo.ts (the repo.* registry commands: management verbs + the repo test/build op verbs), src/channels/residentsView.ts (residents dash: Access-gated GET /residents + /residents?stream=1 + /residents/<owner>/<name>), src/channels/residentsFeed.ts (the index feed: the registry's repo runs and listing reads on attach end, seal and the periodic status tick), src/channels/residentsModel.ts (the shared view model: slug/live/tone, the run → worktree join, the disk headroom, the feed reducer), web/green/pages/Runtime.vue, web/green/pages/RuntimeDetail.vue, web/src/components/residents/ResidentRow.vue (the fold), src/core/operations.ts (the Operations seam + op recognizer), src/core/dispatch/resolve.ts (resolveTarget: the repo/ref resolution started before the ack), src/core/dispatch/authorize.ts (authorizeRepo: the not-onboarded, unverified and access gates, item 29), src/core/dispatch/fastPath.ts (the ops fast-path), src/core/dispatcher.ts (ask-once + prompt selection + note surfacing), src/agents/registry.ts (CODING_SYSTEM_RESIDENT, REVIEW_SYSTEM_RESIDENT), src/cli.ts (--thread); the resident engine's pure decisions, imported by the Worker: src/execution/residentDisk.ts (the disk-full signature, the df probe and the recycle plan, item 54), src/execution/residentIncarnation.ts (the incarnation id and the leases, item 22), src/execution/residentStepPlan.ts (the engine steps' read-then-act plans, item 22), src/execution/residentInstanceId.ts (the refresh instance id, the lifecycle row and the cron's instance decision, item 7), src/execution/snapshotRetention.ts (the two-generation rotation and the recorded ids, items 7/11/34), src/execution/residentAutoRebuild.ts (the down transition's rebuild decision and its budget, item 36), src/execution/residentInfraStreak.ts (which steps are the repository's and which the resident's, and the ladder over consecutive resident-step failures, item 67) + deploy/cloudflare-resident/drain.ts (item 69) and deploy/cloudflare-resident/memoryGuard.ts (the memory guard's pure reader parse and two-threshold gate, item 70; its memory-pressure token in src/execution/sandboxErrors.ts) + src/core/runSetup.ts (the setup words an attach refusal is named by, item 76); native containment qualification: scripts/native-scope-qualify.mjs
  • Docs: AGENTS.md; the thread-sandbox counterpart is execution.md
  • Tests: web/green/pages/personalReports.test.ts (selected-work UI completeness); src/execution/residentMembers.test.ts, deploy/cloudflare-resident/replicaMigration.test.ts, deploy/cloudflare-resident/replicaFleet.test.ts, src/execution/residentRoute.test.ts, src/execution/residentPlacement.test.ts, src/execution/repoRegistration.test.ts, deploy/cloudflare-resident/repoRegistration.test.ts, src/execution/resident.test.ts (incl. ::ResidentOperations.run), src/channels/residentsView.test.ts, src/channels/residentsFeed.test.ts, src/channels/residentsModel.test.ts and web/src/pages/residents.test.ts (the residents dash, item 42), src/execution/residentWake.test.ts (item 65), src/execution/residentReuse.test.ts and deploy/cloudflare-resident/reuseAttach.test.ts (item 66), src/execution/residentRebind.test.ts and deploy/cloudflare-resident/rebindAttach.test.ts (item 16's one exception and its second movement), deploy/cloudflare-resident/execReplacedWord.test.ts and src/execution/residentRefresh.test.ts (the /exec answer's gate and the two wording lists, item 43), deploy/cloudflare-resident/runtimeBusy.test.ts and deploy/cloudflare-resident/threadErr.test.ts::a refused connect (runtime-busy) (the refused connect's word at the spawn alone, its 503 and the refresh cycle's yield, item 68), deploy/cloudflare-resident/releaseAtRunEnd.test.ts (items 16a, 16b and 17: a run's end releases the tree whatever it holds, the sweep releases on idleness alone), deploy/cloudflare-resident/dirtNeverKeeps.test.ts (item 17's rule over items 16b, 45, 54 and 55: the idle gate, the reclamation, the disk-pressure eviction and the disk-full recycle never keep a tree — or the container — for its dirt, and every eviction and the recycle record what the tree held), src/execution/residentHead.test.ts (item 51), src/execution/executor.test.ts::LocalOperations, src/execution/factory.test.ts::makeExecutor resident selection, src/config.test.ts::per-repo access (canUseRepo) + ::repo management gate (canManageRepos), src/core/repoContext.test.ts, src/core/commands/repo.test.ts (incl. ::repo test / repo build …), src/core/dispatch/authorize.test.ts (the repository gates, one outcome each), src/core/dispatcher.test.ts::resident repo dispatch + ::repo/ref resolution + resident prompt selection … + ::repo management commands … + ::deterministic ops fast-path …, src/agents/registry.test.ts::resident prompt variants, src/cli.test.ts; the resident Worker's request/engine code has none (deploy/* convention: verified by npm run typecheck in deploy/cloudflare-resident plus the live checks below), but its PURE decision logic is unit-tested under plain Node: deploy/cloudflare-resident/preflight.test.mjs (deploy preflight, item 44), deploy/cloudflare-resident/runRegistration.test.ts (item 44: the run registration held from attach to release, so the preflight sees a harness run between operator calls), deploy/cloudflare-resident/gc.test.ts (residency GC, items 45–46, incl. parseRefListing) and deploy/cloudflare-resident/refresh.test.ts (a scan over the sources for the module boundary the Workflows binding depends on, item 7) and deploy/cloudflare-resident/instanceStep.test.ts (a scan over the entry for the order of the instance step's gates and its in-flight count, item 16c), src/execution/residentAutoRebuild.test.ts and deploy/cloudflare-resident/autoRebuild.test.ts (item 36: the transition's rebuild and its budget, the watchdog as backstop, the killed-extract wiring of item 61), src/execution/residentInfraStreak.test.ts and deploy/cloudflare-resident/infraStreak.test.ts (item 67: the step split, the ladder, the Worker wiring) and deploy/cloudflare-resident/lifecycle.test.ts (a scan over the sources for the absence of any lifecycle timer — no alarm slot armed, no refresh alarm handler, no alarm-missed, a schedule only for provisioning — items 7 and 9) and deploy/cloudflare-resident/runtimeUnreachable.test.ts (a scan over the entry for the runtime-unreachable counter, its ladder, the admin op, the escape hatches and the wake budget — item 64); the refresh-cycle perf helpers are unit-tested from src/: src/execution/residentCleanliness.test.ts (the one-spawn clean check, item 16b), src/execution/residentDepCache.test.ts (depCacheScript/parseDepCacheScriptOutput/mutableCacheSwapScript, item 18), src/execution/residentDepsStore.test.ts (the deps store: layout, hit/join/install plan, semaphore size, scratch clone, commit script, listing, eviction — item 57), src/execution/residentRefresh.test.ts (the refresh plan and checkout update, the failure classifier, the restore disposition, the R2 transfer budget — items 5/7/44/48/54/61), src/execution/residentIncarnation.test.ts (the incarnation id, the lease predicate, take/release, the in-flight row — item 22), src/execution/residentStepPlan.test.ts (each step plan called twice, the snapshot's compare-and-swap — item 22), src/execution/residentInstanceId.test.ts (the instance id, the lifecycle row, the cron's decision and the refresh-now gate, the retry window — item 7); the disk budget (item 55) is src/execution/residentDiskBudget.test.ts (measurement, reserve, projection, admission, eviction order, the refusal text) plus deploy/cloudflare-resident/instanceSizing.test.ts (the instance arithmetic over the same reserve constants); the purge decision (item 60) is src/execution/bindingPurge.test.ts; the snapshot rotation (items 7/11/34) is src/execution/snapshotRetention.test.ts (the decision, plus a scan over the entry for its wiring) + deploy/cloudflare-resident/drain.test.ts (item 69) + deploy/cloudflare-resident/memoryGuard.test.ts and deploy/cloudflare-resident/memoryGate.test.ts (item 70: the pure gate with a fake reader, the Worker wiring by scan); native gate and Linux helper: deploy/cloudflare-resident/nativeScope.test.ts, deploy/cloudflare-resident/testing/nativeScopeLinux.py

Base URL for all [agent] checks: the resident Worker's own hostname, written https://<resident hostname> (or ...) below. $ADMIN / $OPERATOR are the RESIDENT_ADMIN_TOKEN / RESIDENT_OPERATOR_TOKEN secrets (npm run secrets in deploy/cloudflare-resident). A convenient fixture for the [agent] checks is a tiny public Node repo such as repo:jshttp/vary (default branch master, no committed lockfile); Go repos (e.g. boldsoftware/meat) are unsuitable — the container image has no Go toolchain. With the GitHub App configured, onboarding is limited to repos in the installation (item 35).

Behavior ​

  1. Three bearer scopes, fail closed: the admin token guards POST /onboard, POST /offboard, POST /reconfigure, POST /rebuild, all of POST /debug, GET /residents; the optional read token (RESIDENT_READ_TOKEN) opens ONLY GET /residents and the pure-read /debug ops info, schedules, threads — any other op with it is 403 (authenticated but under-scoped; 401 stays "no valid bearer") even though the route is reachable (checked after auth, so an unauthenticated caller still learns nothing); it never reaches an operator or admin route, and is for dashboards and humans who need to look without being able to change anything; the operator token guards POST /attach /run-deadline /exec /read /write /op and GET /status. Admin is a strict superset (valid on operator routes); the operator token never opens an admin route. Comparison is constant-time (byte-XOR accumulate), never a plain !==. Unset/empty secrets grant nothing; unknown paths answer 401 before revealing 404. The only unauthenticated route is GET /healthz (deploy wake ping; touches no DO, returns {ok:true, build:{commit, builtAt?}} — the commit the deploy injected, which makes a deploy's edge propagation provable from outside without auth; execution.md item 13 replaced the hand-bumped u marker with it).

  2. Atomic cap: the singleton ResidentRegistryDO holds repository settings and member lists. It counts active, pending and retained member addresses and enforces RESIDENT_CAP (5) in one input-gated read-count-insert — an over-cap onboard is refused with 429 by the registry, never by the platform (max_instances is 10, deliberately above the cap). Lowering the cap leaves existing residents registered; it only refuses onboarding another resident.

  3. Onboard returns immediately; provisioning is schedule-driven — the one timer a resident has: POST /onboard {resource, commands:{test,build,...}, defaultRef, diskBudgetMb?, provisioningTimeoutMs?} validates, inserts into the registry, persists state onboarding, arms two schedules on the resident DO (the provisioning run at +1s and the fail-closed deadline at +provisioningTimeoutMs — via the Container schedule() API, never the raw DO alarm slot, which the Sandbox base class owns; these two are the only schedules lifecycle code ever arms — every other cycle is a Workflow instance, item 7, and deploy/cloudflare-resident/lifecycle.test.ts holds that line over the sources), and answers 202 without starting a container. If arming the resident fails, the registry slot is freed (no half-onboarded residents). repo ids must be lowercase GitHub <owner>/<name> slugs (the whole resource ≤ 63 chars — it doubles as the sandbox id).

  4. Provisioning pipeline: git clone --mirror https://github.com/<slug>.git into /workspace/mirror (root-owned) → resolve the default branch (the configured defaultRef when it exists in the mirror, else the mirror's HEAD) → local clone into /workspace/checkout, chowned to the unprivileged build user worker1 → full install (when the command table has one) + build via su -s /bin/bash worker1 -c 'cd /workspace/checkout && <cmd>', token-free — then two stamped snapshots (mirror + checkout, createBackup({localBucket:true}) into the BACKUP_BUCKET R2 binding, 1-year TTL as a leak backstop), facts {defaultRef, sha, lockfileHash} recorded in DO storage, state warm, refresh chain armed. Every step runs under the per-resource provisioningTimeoutMs as its exec budget; a failed step → down with reason provision-failed at <step>: … and the registry slot is KEPT (the admin sees the named reason on /status and offboards/re-onboards); a STUCK onboarding (deadline schedule or watchdog finds onboarding past its budget) → down(provision-timeout…) and a legacy single-resident cap slot is released. Pool members remain registered so their addresses and cleanup obligations are retained.

  5. Warm means rehydration, not persistent disk: DO SQLite is the source of truth (config, lifecycle state+reason, last-fetched sha, lockfile cache key, snapshot handles); container disk is a cache. On every wake (a refresh instance's fetch step or an attach, finding the runtime dead or the disk not matching the marker), the resident persists restoring (reason rehydrating) BEFORE any restore work, restores mirror + checkout from the stamped snapshots — the two restores run SEQUENTIALLY (mirror, then checkout — the SDK serializes backup operations on one queue anyway, so a concurrent pair only let the second one's clock run while it waited) and each is judged by its BYTES, not by a clock (item 61, restoreWithProgress → the pure judgeRestoreProgress in src/execution/residentRefresh.ts; the SDK's restore call takes no timeout, progress callback or AbortSignal): while du of the SDK backup work directory plus the target keeps growing the wait continues, a stall (no growth for RESTORE_STALL_MS) or the RESTORE_MAX_MS cap fails visibly as down(r2-restore-failed: <mirror|checkout> restore stalled|capped: …) with the bytes and timing named, instead of stranding restoring until the 30-min watchdog — and a previous attempt's restore still running is awaited before the pre-restore clean, never raced by it — re-verifies the stamp against the restored disk (mirror rev-parse must equal the stamped sha; the ls-tree lockfile key at that sha must equal the stamped key), then flips warm and records lastRestore {at, ms}. A stalled or capped restore → down(r2-restore-failed: …) (the SDK's restore cannot be cancelled and may still be streaming into the disk, so only a rebuild on an empty container follows); a restore the runtime replacement interrupts (a resident Worker deploy rolled the container under it; RUNTIME_REPLACEMENT_WORDING on the failure, restoreFailureDisposition in src/execution/residentRefresh.ts) → degraded(restore-interrupted: …), never down: the disk it streamed into is gone with the container, nothing can land on a rebuild, the failure is not evidence about the repo (it never counts toward the park streak), the error returns to the instance step whose classifier throws it to the engine, and the retry restores again; a stamp that does not verify → down(snapshot-stamp-mismatch: …). Restore is provably not a re-clone: lastRestore.ms is a fraction of provisioning time, and snapshot ids/provisionedAt stay untouched. The first exec after a container (re)start is bounded by the SDK's physical start — which waits WAKE_PORT_READY_MS (3 min, the template's SANDBOX_PORT_TIMEOUT_MS; the SDK's default is 90 s) for the control port to accept a request — plus ONE connect of the SDK's fixed 30 s (SDK_CONNECT_TIMEOUT_MS), so a slow boot spends its time in the port wait and is not read as an unreachable runtime (item 64).

  6. Snapshot stamps: snapshots are written ONLY by onboarding provisioning and default-branch refresh — never by attach-time or op executions — and each is stamped {ref, sha, lockfileHash}. The lockfile cache key is the sha256 of git ls-tree <sha> -- <lockfile candidates> output from the MIRROR — a pure function of the commit. It is deliberately NOT a hash of the working directory: installs generate uncommitted lockfiles (npm writes package-lock.json), which would poison a disk-derived key and fail the post-restore stamp check — which is why the key derives from the committed tree.

  7. The refresh cycle is one short Workflow instance per resident and ten-minute bucket (src/execution/residentInstanceId.ts, ResidentRefresh in refresh.ts, re-exported by worker.ts for the binding): the watchdog cron creates it (item 9) with the deterministic primary id refresh_<slug>_<bucket> or secondary id refresh__replica_<residentId>_<bucket>, so the cadence is the cron's 600 s (< the 20m sleepAfter: the steps call into the container, so a healthy resident is re-warmed before the platform can sleep it) and a second creation in one bucket meets the engine's duplicate-id refusal. Steps fetch, install (only when the plan moved the lockfile key on a repo with an install command), build (only when the branch moved), snapshot, then the housekeeping sweep (item 23) and measure (item 55) — each a step.do calling the resident's own phase method through the DO stub (refreshInstanceFetch/Install/Build/Snapshot/Sweep/Measure) under retries: {limit: 6, delay: "30 seconds", backoff: "exponential"} (the delays sum to about 15.5 minutes, past the 3 to 10 minutes a resident Worker rollover takes to settle) and a timeout equal to the method's own budget capped at the engine's 30 minutes (stepTimeoutMs); step inputs and answers are refs, shas, keys, a path and a word — never a payload, never a credential (the token is minted inside the step that needs it). One cycle: the entry gates (rehydrate if needed; the registry record; the image reconcile; the disk-full re-probe, item 54; the park streak and the idle gate, item 16b) → mint a repo-scoped token (skipped when the GitHub App is unconfigured; public repos fetch anonymously) → refreshing → git fetch --prune into the mirror → if the default-branch sha moved: plan the rebuild against the disk checkpoints (item 48) — update the checkout (fetch from the local mirror + reset --hard + clean -fdx, as worker1 — -x drops the gitignored build caches so the rebuild allocates FRESH inodes instead of writing through any hardlinked into already-attached, sha-pinned worktrees, review 1b; node_modules is excluded from the clean and install skipped when the committed lockfile key is unchanged), run build, write a new stamped snapshot pair (the mirror and checkout uploads run concurrently, each bounded by the 5-min R2_TRANSFER_TIMEOUT_MS — a hung upload degrades the cycle as snapshot-failed: … timed out instead of stranding refreshing for the watchdog), rotate the generations (src/execution/snapshotRetention.ts): the replaced pair is RETIRED (resident:snapshot:retired), and the pair retired before it has its R2 objects deleted (both prefixes swept concurrently) — two generations are kept, so the checkout handle /status published seconds before the rotation still restores for at least one more cycle; a re-fired step that replaces the retired pair itself deletes nothing → facts + snapshot updated in one storage write → warm. The phases are single methods (refreshGate, refreshFetch, refreshClearMarkers, refreshInstall, refreshSnapshot, refreshComplete, refreshFailed); the instance id is the cycle fetchMirror records, and a rebuild's markers come off in the fetch step, once per cycle, so a retried step re-enters its idempotent method (item 22) and finds its work done. A step killed from outside (the container replaced under it, refresh-interrupted) throws to the engine and is retried into the same row — the state stays refreshing, never degraded; a gate that stopped the container on purpose (a stale image, a disk-full recycle — CycleRestartError) throws the same way, and the retry, thirty seconds on, restores onto the restarted container and runs the cycle: the retry is the re-warm. Failures of the repository's own degrade with a named reason (github-unreachable: …, <step>-failed: …, and disk-full: … for a full container disk — item 54, the one failure the resident repairs itself) while the last snapshot keeps serving, and end the cycle with failed; a gate that ends the cycle (idle, an offboard) ends it with that word; the housekeeping steps still run for every verdict but an offboard, and neither wakes a slept container; the next cron firing creates the next instance from the row's state. The instance's run is the cycle's resident.refresh root carrying the engine's instanceId, each step's commands grafted under it (tracing.md item 25). No lifecycle timer exists for any of this — no alarm chain, no re-arm, no alarm-missed: the engine records that a cycle is in progress, so a dead one cannot be silent (deploy/cloudflare-resident/lifecycle.test.ts scans the sources for the absence). The per-resident row lifecycle (LIFECYCLE_KEY) reads workflow whatever it stores (lifecycleOf: a missing value, or the alarm a flagged rollout left behind); /status answers lifecycle, refresh: {instance: {id, createdAt, lastStep}, skipped} and snapshot: {ref, sha, lockfileHash, createdAt, mirrorBackupId, checkoutBackupId, depsBackupId} — the seed handle a cold sandbox restores from (execution.md item 25), depsBackupId the entry archive for the snapshot's own lockfile key when one has been taken (item 61), else null, /debug info the same; the admin POST /debug {"op":"lifecycle","resource":…,"mode":"workflow"} rewrites the row (its one accepted mode) and {"op":"refresh-now"} creates this bucket's instance on demand (item 13). The wake attach's background cycle is the next bucket's instance (at most ten minutes; the attach's own wake-fetch makes the mirror current at once, item 16b). The rows the retired alarm chain armed (onRefreshAlarm, onWorktreeSweep, onDiskMeasure) are deleted in ResidentDO's constructor, before the object handles its first event: the SDK's alarm() skips a due row whose callback method is gone WITHOUT deleting it and re-arms for that past time at once (@cloudflare/containers 0.3.7), a hot alarm loop that only the row's deletion ends — and on a resident upgraded from the chain the first event is that alarm.

  8. Token discipline: the resident Worker holds GitHub App secrets in its own wrangler secret domain for its root-owned mirror; its mirror mints repository-scoped installation tokens and uses a root-only one-shot file for mirror fetches. A writable thread worktree never receives an App token. The bot sends a run bearer and the Git door base URL at attach; the Worker points the worktree origin at that door, scrubs any .git/github-credentials left by an older image, and requires the run bearer on each writable /exec. The door enforces repository and branch bindings before forwarding with its own trusted-side App token. A read-only worktree carries no credential file or writable origin. Missing door configuration refuses writable attach and execution by name. A root-only mirror credential is never available to a thread user.

  9. Watchdog cron: every 10 minutes (strictly < SLEEP_AFTER 20m) the Worker iterates the registry and calls each resident's watchdogCheck() — storage and engine-status reads only, containers are started by the instances' steps, not the watchdog. It re-arms nothing: there is no chain. A refreshing/restoring marker older than STALE_MIDFLIGHT_MS (30 min) with no cycle or restore actually running (no live lease in the in-flight row, item 22, and the engine does not say the last recorded instance is queued, running, paused or waiting — a step between retry attempts holds no lease and writes nothing, so the engine's word overrides the marker's age) is an orphan from a cycle that died (DO evicted by a deploy, an instance out of retries mid-step): the watchdog re-reads state right before flipping it (a cycle that started during its reads owns the state), clears the dead leases and marks it degraded(stale-mid-flight: <state> since <t> with no cycle running; the last instance <id> is <engine status>|no instance recorded; the next refresh instance normalizes it) — a failed instance is visible by id on /status, and the marker is no longer refreshing, so the instance the same pass creates normalizes it. An onboarding resident past its recorded deadline (+30s grace) → down(provision-timeout…), its two schedules cancelled, cap slot released (the DO releases it; the Worker removes the registry row as backstop); a down resident gets item 36's backstop pass (the transition already rebuilt it or stamped its budget). The cron's instance-creation duty (shouldCreateRefreshInstance, refreshInstanceId, createRefreshInstance): for every resident the same pass — after the check settled the state — creates the refresh Workflow instance with the deterministic id refresh_<slug>_<bucket> (owner and name lower-cased, every character outside [A-Za-z0-9_-] mapped to -; the ten-minute bucket; cut with a short hash of the slug when the whole id would exceed the platform's 100 characters, always matching ^[a-zA-Z0-9_][a-zA-Z0-9-_]*$), only when no live cycle blocks it (refreshCycleBlocked: the row is serving — not onboarding/down; the engine is not running the last instance; no refreshing marker younger than STALE_MIDFLIGHT_MS — an older one is the orphan the stale branch above just named) and the cadence has elapsed since the last instance in whole buckets (one bucket awake; the idle interval, 6 h, for a warm row with idleSince set) — so cycles stay serialized per resident and an idle resident keeps its idle cadence. The idle cadence is for warm alone: only a cycle's gate clears idleSince, so a wake that dies before its cycle (a resident Worker deploy rolling the container mid-restore) leaves the marker on the restoring row and on the degraded(stale-mid-flight…) the watchdog stamps it with, and both are due at the next bucket like any awake row; a streak-parked degraded row (item 16b) is due every bucket too, its gate parks it again without a fetch, and the first rehydrate flips it warm. A second firing in one bucket meets the engine's duplicate-id refusal, treated as the no-op it is — confirmed by asking the engine whether an instance answers for the id (the refusal carries no code), never by the error's wording, so an unrelated failure is never mistaken for a duplicate; a skipped live cycle and a duplicate are recorded on the row (refresh.skipped) for /status. The watchdog's summary carries instance: {id, action: created|duplicate|skipped|failed, why} per resident, and the firing record counts the instances created as refreshed (live-view.md item 14).

  10. Command table is admin-writable only: commands exist solely in the registry and change solely through /onboard and /reconfigure (admin scope). commands must include test and build; extra named commands are allowed (install is honored by the engine); reconfigure.commands replaces the whole table. Command strings execute inside the resident as worker1 — never on the Worker, never as root. The record also carries per-entry effects profiles (item 39), equally admin-writable-only.

  11. Offboard is a full teardown: legacy single-resident offboard removes registration before teardown and retains failed cleanup intent on the DO. Pool offboard first closes its catalog; each member removal verifies the original pool revision under its recreate lock. Confirmed destruction precedes snapshot, dependency-backup and member-prefix cleanup. Address, retirement and cleanup evidence survive until confirmation. Only complete member receipts permit placement and catalog deletion. SQLite schedule tables remain present; cleanup deletes keys rather than deleteAll(). The response itemizes registryRemoved, schedulesCancelled, containerStopped, storageCleared, backupObjectsDeleted, r2ObjectsDeleted, errors.

  12. Lifecycle states are persisted and causal: onboarding|warm|refreshing|restoring|degraded|down, with a reason string required on every degraded/down. GET /status (operator) returns {state, reason, inFlight, lifecycle, refresh, snapshot} — lifecycle and activity (item 7), and the snapshot handle a seeded sandbox restores from (execution.md item 25: the stamp, both archive ids, the deps entry archive for its lockfile key or null; opaque ids, useless without the bucket) — operators see what a run needs, not config. GET /residents (admin) returns each registry record plus the live engine view: state/reason, defaultRef, sha, lockfileHash, provisionedAt, lastRefreshAt, lastRefreshError, lastRestore, the snapshot stamp with backup ids, pending-schedule counts, and threads — every thread binding (threadKey, ref, sha last attached at, user, deps, boundAt, lastAttachAt, evicted/evictedAt; newest attach first; worktreePath omitted as internal layout).

  13. Admin debug surface (POST /debug, admin scope; built for these live checks and kept for operability): ops info (engine view without the registry), schedules (the provisioning rows, the one timer — both empty on a settled resident), refresh-now (create this bucket's refresh instance now, due or not: created with the id; duplicate naming the instance when the cron already served the bucket; skipped with the why beside a live cycle or on a resident that is not serving), stop-container (simulate a platform sleep: a SIGTERM, the incarnation swapped — unchanged), recreate-container (item 64's rung 3 on demand: forget the SDK's runtime identity, destroy the VM, keep every snapshot, start the restore from the snapshot now — answers 202 {recreated:true, restoreStartedAt}; 409 recreate-refused: … while the engine is mid-flight or on a down resident, whose exit is /rebuild), force-onboarding (fault injection for the watchdog timeout path), mint-token (attempt a mint; returns the command-level error shape, never token material), run-watchdog (the cron's exact pass, on demand — the instance-creation duty of item 9 included), lifecycle (rewrite one resident's lifecycle row, item 7; mode is workflow, the one scheduler — the op's only effect is to replace a stale alarm value the flagged rollout left behind), force-down (fault injection for item 36 without corrupting real R2 objects: bare, persist down with a rehydration-flavored reason so the watchdog's backstop pass is the one under test; with transition: true, go through goDown itself so the rebuild or the budget stamp happens in the call), plus the thread ops threads (enumerate bindings; worktreePath omitted, like the engine view), backdate-thread (age a binding's lastAttachAt), sweep-now (run the eviction pass now), purge-bindings (item 60: delete a load run's evicted synthetic bindings), the GC ops reclaim-now (item 45) and the registry-wide set-test-overrides (item 49: lower the effective cap / LRU floor for live over-cap checks; deploy-scoped). Side effects are explicit and admin-gated; nothing here returns secrets. The existing Read-authorized threads report also accepts a paired threadKey and runId selector. It reports only that resource’s saved cancellation refusal and redacted binding/registration/fence matches. Absent or invalid evidence and successor mismatches remain unverified; a stored stopped receipt is not a fresh physical ending witness. The report body does not write or call the runtime. Cold DO activation retains the SDK schedule/alarm and Resident index initialization effects; this operation does not promise effect-free activation.

  14. x-env hardening: caller-supplied x-env-* headers are ignored on every route (nothing reads request headers besides authorization; no request header is forwarded into a resident — a deliberate deviation from the thread-sandbox Worker). The engine's ONLY injected env var is GIT_TERMINAL_PROMPT=0 (so anonymous git fails fast instead of prompting), and it passes through validateEnvNames (^[A-Z_][A-Z0-9_]*$) like all future injections must. A caller's variables for one /exec ride the request BODY's env alone, read through envFromRequest under the injected variables (harness-pi.md item 4) — the channel a write run's four commit identity variables take: the bot fills GIT_AUTHOR_NAME/GIT_AUTHOR_EMAIL (the requester's bound pair, else the bot pair) and GIT_COMMITTER_NAME/GIT_COMMITTER_EMAIL (the bot pair) into each exec's body, resolved per exec (execution.md item 5; record 0062), so the commits a run makes in the worktree carry the pairs while the image's fallback identity — switchboard-resident@switchboard.invalid, off the GitHub domain — stands only where no pair rides.

  15. Version pinning and image tools are exact: @cloudflare/sandbox is pinned to 0.13.0-next.751.1 in package.json and the Dockerfile FROM docker.io/cloudflare/sandbox: tag must match it exactly — bump both together. The image installs apt's ripgrep and fd-find, exposes the latter as fd on PATH, and fails its build unless both rg and fd resolve. Cadence pairing: refresh cadence (600s) = watchdog cron (10 min) < SLEEP_AFTER (20m). The image provisions the unprivileged build account worker1 (uid 2001); sudo is absent and root's password is locked, so su demotion works one way only. worker1 is the engine's build user; Generated thread/operation accounts are provisioned on demand (items 19 and 75).

  16. Attach: per-thread worktrees + sticky ref binding: POST /attach {resource, threadKey, refHint?, readonly?, sha?, reuse?, ownPr?, refByDefault?} (operator scope; readonly → item 50, sha → item 51, reuse → item 66, ownPr and refByDefault → the one exception below; resource selects repository configuration and immutable thread placement selects the DO — the service hosts many residents). Inputs are validated BEFORE anything derives a path or a git argument (P1): threadKey against ^[a-z]{1,32}:[A-Za-z0-9._:-]{1,128}$ (platform-namespaced, conservative charset — no /, no whitespace, no shell metacharacters), refHint against the strict branch pattern (^[A-Za-z0-9][A-Za-z0-9._/-]{0,199}$, no ../@{/.lock). The thread's ref binding is resolved from DO storage keyed by threadKey: absent + refHint → verify the ref resolves in the mirror (fetching under the mutex if needed) and bind; absent + no refHint → 409 {error:"needs-ref: …", needs:"ref", defaultRef:"<the resident's default branch>"} (the defaultRef lets the caller bind by default — item 30 — but the binding is still only ever made by an attach that names a ref); present + an explicitly named refHint → that ref replaces the sticky binding, while an absent or refByDefault hint falls back to the bound ref; the response's ref is still the resident's authoritative answer. The legacy fallback has one additional movement: the thread's own pull request (pure parseOwnPr, parseRefByDefault, boundByFor, boundByOf, rebindPlan, rebindVerdict, canReturnToDefault and returnToDefault in src/execution/residentRebind.ts, imported by the Worker like residentReuse; the Worker's rebindToOwnPr, moveOntoOwnBranch and returnBindingToDefault). The body's ownPr: {number, ref} names the pull request the thread's OWN run opened and its head branch — the reason the refHint is that branch; the bot sends it only when the ref was bound from the run record's PR (RepoContext.refFromPr with prFromRecord, ownPrOf, item 29), never for a PR a person named and never on a resume; a coordinator child's refHint is its contract's branch, always — a PR-derived ref the resolver bound off the child's own request text is dropped before the attach (item 29; issue 1860), so the binding is the unit branch the child pushes and the push guard allows it; ref is checked against the same pattern as refHint, number must be a positive integer, anything else → 400. The move is a decision about the binding alone. The tree is item 17's business afterwards: the attach provisions it at the moved ref exactly as it provisions any tree — clean at the ref's tip; a tree left dirty, or on the old branch, recreated; the binding's worktreePath kept, so a binding has one thread directory and an eviction removes all of it — so there is no checkout in place, no measurement of the tree's dirt, and no word from the rebind to the worktree step. The binding moves onto the branch only when all of these hold, decided BEFORE the ref is chosen so the rest of the attach runs on the moved ref: (1) the binding was made by default — the first message named no branch — read off boundBy (default | name, recorded when the binding is made: default iff the caller sent refByDefault: true for the resident's own default branch, item 30; a binding made before the field reads as default iff its ref IS the default branch), so a ref a person named is never moved (rebindRefused.reason: "named-ref"); (2) the thread was not rebound before, or its earlier move was returned (below) — a thread moves once per pull request ("already-rebound"); (3) the branch is the thread's own: remembered — a branch the thread's own runs pushed as the binding remembers it (ownBranches, written at each release, item 16a), the fact that decides by itself, tree or no tree — or, when nothing was remembered (a release that never came), a local branch of the thread's SURVIVING tree (git rev-parse --verify --quiet refs/heads/<ref> as the thread user, in the tree: the physical trace of this thread's run having made it; one probe, no status, no HEAD); (4) the mirror holds the branch — fetched first when it lacks it, with a token minted before the mutex like the attach's own recovery fetch, since the tree is cloned from it. A branch neither remembered nor local to a surviving tree, or one the mirror does not hold even after the fetch (deleted after a merge) is "branch-absent" — never unknown-ref: the binding stands and the attach goes on at the ref it had. The move is judged and written under the mirror mutex on the row as it stands there — re-read before anything is judged or written, so a premise that changed while waiting (the ref already moved) is re-judged, never overwritten; a mutex timeout is the attach's 503 mirror-busy. The binding records rebound: {from, to, pr, at} and keeps its boundBy; the answer carries rebound (the move) or rebindRefused: {to, pr, reason, why} (the binding stood) — never an error: a refused rebind is not a failed attach. After the move the attach runs on the new ref as any attach does — wantShaForBinding keeps the PR head (the hint now equals the bound ref), the mirror fetches a tip it lacks (item 51), and the worktree step recreates the tree unless it is already clean at the moved ref's tip or a descendant of it (item 17). A reusing attach (item 66) never rebinds: the tree is the resumed run's own. The second movement — back to the default. The thread's branch can die (its pull request merges, the branch is deleted), and a binding left on it would fail every later attach unknown-ref and fall to a cold sandbox for the rest of the thread's life. So a binding on such a ref is verified against the origin at every attach of a thread on its own or rebound branch: the attach's fetch decision (item 51, mirrorFetchReason) answers returnable-ref for a ref canReturnToDefault admits even when the mirror still holds it, the fetch --prune under the mirror mutex runs, and the ref's existence is re-read for real — so the return is decided at the first attach after the deletion, never at the next refresh cycle (a mirror trusted for still holding the ref would provision a tree at the deleted branch's stale tip until the cycle's prune caught up, and whether the thread returned would depend on where in the cycle its follow-up landed); a verification fetch that fails (the origin unreachable, no token) is not the attach's failure — one resident log line, and the attach goes on with the mirror's ref as it stands, the return only ever decided on a fetch that ran. When that fetch finds the bound ref gone from the mirror and the gone ref is the thread's own — EITHER the branch a rebind moved the binding onto (rebound recorded and not yet returned, still on that branch, bound by default) OR one of the binding's own branches, a branch this thread's runs pushed as the binding remembers it (ownBranches, item 16a), whatever boundBy says: a ship unit's coding child binds its unit branch by name, pushes it and opens the pull request, and the unit's merge deletes it (agent-ship.md item 3) — the binding goes back to the resident's default branch (canReturnToDefault), decided where the fact is established: under the mirror mutex in attachThreadCreate, on the row re-read there (returnBindingToDefault, the pure returnToDefault — one decision for the predicate and the move, so the two cannot disagree; a row it does not admit is answered as it stands, nothing written). The row's ref becomes the default and the binding reads as bound by default from here on (boundBy: default — the default was chosen for want of the branch it was on), the last move back is recorded on the row as returned: {from, to, pr, at} — the pull request the rebind's, else the one the binding remembers the gone branch heading — and a rebind's move is stamped rebound.returnedAt besides; the same worktree step provisions the tree at the default's tip, and the answer carries the same returned so the card reads · returned to <default> (the branch of this thread's PR #N is gone) — or · returned to <default> (this thread's branch <branch> is gone) from a resident whose answer names no pull request, since the card never invents a number. The thread is default-bound again: its next own pull request may move it once more — a returned move does not count as the thread's one move, and a thread that named its unit branch follows its next pull request as a default-bound thread does. A ref a person named that the thread never pushed and that vanished keeps the unknown-ref refusal — a person's branch that is gone is not this thread's finished work, and the resident never swaps it for the default unasked — and a caller that named the expected commit still gets item 51's detached checkout when the mirror holds it. A returned thread keeps the worktree path it was made with, through both movements. A nonexistent ref → 400 {error:"unknown-ref: …"}, nothing created. The worktree lives at /workspace/threads/<threadKey-slug>-<hash8>/<ref-slug> (the thread hash prevents thread-slug collisions; a provisional named replacement whose ref slug collides gets its own ref hash); the response is {workspace, ref, sha, user, container?, reconciled, recreated, deps, credentials, credentialsError?, mutexWaitMs, attachMs, rebound?, rebindRefused?, returned?} (container is the kernel's boot id of the resident's VM, item 66). The answer also carries the attach's step trace (item 63). 16a. Detach: a run releases its live binding when it ends (POST /detach {resource, threadKey, force?, pushed?}, operator scope; pushed below): detach releases the live binding and tree, while the UID stays spent for its owner until confirmed VM destruction. A later run in the same thread may reclaim that UID if its disk inspection is clean; distinct one-shot threads and disposable operations spend new UIDs. The bot calls executor.release(mode) when the run ends — AFTER the answer has been sent to the thread (or the failure card closed), checkpoint endings instead record the release acknowledgment before sealing; ordinary output never waits between "answer ready" and the reply; best-effort (a failure is a log line, never a failed run). The release calls the shared preservation decision (item 71) before removing a live binding or tree. It keeps the tree with a named blocker until that decision authorizes removal: "if-idle" (force:false) — a writable run's normal end — releases unless an op is in flight (busy, below); "always" (force:true) — read-only agents and legacy stop cleanup — requests process termination when work is in flight, then applies the same preservation decision. The fenced cancellation path below stops resident processes and retains the binding and files. Except for a failed-publication preservation hold, uncommitted or unpushed work never keeps a tree: what a run wants kept, it commits and pushes. The loss is never silent. On an if-idle release with the runtime up the resident measures the tree once AS THE THREAD USER (the one-spawn worktreeCleanliness, item 16b — git status --porcelain -uno and git rev-list --count HEAD --not --remotes via threadRun, never root git in a thread-writable tree: safe.directory would refuse it, and repo-local config is the root-execution vector it guards against; tracked files only, item 17's definition of dirt and the same flags the run loop's note counts with, so the card and the release log name one number) and answers {released: true, user, leftBehind?: {uncommittedChanges, unpushedCommits}} — leftBehind present only when something was there (pure leftBehindOf; a probe that fails names nothing, never a guess, and is a resident log line), one resident log line beside a release that discarded work; the bot's [release] line repeats the counts (leftBehindSentence). The run's own record carries the same fact as a work_left_behind run note, with the card's label reading N uncommitted change(s) and M unpushed commit(s) left behind — discarded at the run's end: read by the run loop off the tree at the run's end (observeCodingWorkspace's uncommittedChanges and unpushedCommits — git status --porcelain -uno, git rev-list --count HEAD --not --remotes over the run's own executor — workLeftBehindOf, workLeftBehindSummary, workLeftBehindLabel), after the model's last turn and BEFORE the stream finishes, because the release itself runs after the record is sealed and a note published there would land nowhere a person reads; a run that left nothing says nothing, and a hard Stop observes nothing (its card uses the neutral ⏹); destructive Kill suppresses final output. A thread with an exec/read/write in flight (DO-memory counter around those routes) is kept with reason busy, so a release can never remove a worktree under a concurrent command — unless force:true: a hard stop drops the bot's /exec fetch but the command keeps running in the container, so a busy guard alone would hold the pool user until the next sweep step. Force-detach with an op in flight first kills every process owned by the thread's pool user — the same boot-bound killThreadUserProcesses used by fenced cancellation: check the original /proc/sys/kernel/random/boot_id inside the kill command, then run kill -9 -1 as the recorded pool user. Only exit 0 or 137 without timeout or truncated output acknowledges the kill; missing identity, a foreign boot or lost response returns 503 and retains the workspace, then waits for the thread's in-flight counter to drain, polling every 250 ms up to 6 s (FORCE_DETACH_DRAIN_MS; the kill is bounded at 2 s but is a millisecond syscall, so kill + drain leaves ~2 s of the bot's 10 s client bound for the rm — a worst-case overshoot is tolerated: the bot logs [release] … failed while the DO method completes the eviction anyway). The counter drops only after admitted requests settle. A drained counter alone does not prove that detached processes or lost native operations have ended; fenced cancellation also retains durable unknown-operation holds and performs a second kill and independent UID absence check. Still busy after the bound → {released:false, reason:"busy after kill: N op(s) still in flight — kept"}. The decision is the pure planForceDetach in src/execution/residentDetach.ts (imported by the Worker): non-force keeps the plain busy guard; force refuses (refusing to kill: "<user>" is not a pool user) unless the binding's user is a generated unprivileged account — never root, empty, or the build user. The killed op itself completes normally (exit 137 through streamThreadExec to a client that has already hung up) and decrements the counter; one resident log line per kill: detach: force — killed <user>'s processes for <threadKey> (N op(s) were in flight). The bot bounds /detach at 10 s (control-plane POST, not the 5-minute exec ceiling) and only calls it once the answer is out, so a sick resident never delays the reply. Release = the sweep's eviction (evictBinding): tree removed under the mirror mutex when the runtime is up, user returned, binding KEPT marked evicted — evictedWhy: detach, and the same counts the answer carries as evictedLeftBehind (item 17's record) — so the ref stays sticky and the next attach recreates the tree. The release tells the resident what the run pushed: the body's pushed: [{ref, pr}] names the branches the run pushed and the pull requests they head — the bot reads them off the run's own pr_opened events (each carries head, the branch the PR is opened from; pushedBranchesOf, the record's pr.head — run-history.md item 2) at release time, sends the list only when non-empty (an older resident sees the body it always did), and the Worker parses it before anything runs (pure parsePushed, at most 20 entries, each ref through the one ref pattern, anything else → 400). For a live binding whose registration still belongs to this run, the resident appends them to the binding's ownBranches: [{ref, pr, at}] (pure rememberOwnBranches: one entry per branch, a re-pushed branch moving to the end with its current pull request, the newest 50 kept) before the busy or eviction decision, so the fact outlives the tree and survives the sweep's later eviction. A stale release cannot change a successor's binding; an already evicted binding remains a no-op. It is what item 16's rebind reads once the thread's tree is gone, and what its second movement reads to tell the thread's own gone branch — returned to the default — from a person's, which keeps the unknown-ref refusal. No binding → 404 no-binding; already evicted → {released:false, reason:"already-evicted"}. Lifecycle state is never touched. 16b. Clean-idle release + image reconcile + idle sleep (follow-up to 16a): (a) the sweep step (item 23) also releases a live binding that has been idle ≥ 1h (CLEAN_IDLE_RELEASE_S), has no op in flight, and has no active operation; the deadline selects a cleanup candidate, while item 71 decides whether its files can be removed. An expired registration alone is not proof that the owner finished or saved its work. This is what drains bindings whose release never came — a resident that was sick at the run's end, a run older than /detach. The binding is re-read right before removal (a re-attach since the listing keeps its fresh tree), and an inactive runtime supplies no current file proof, so a known owner stays until item 71 can establish permission to release. Every release here records what the tree held (item 17's evictedLeftBehind / evictedUnmeasured beside evictedWhy). (b) The image declares the trusted account-provisioning protocol; wrangler deploy swaps the app image but a RUNNING container keeps the old one (row "deploy while warm" — the container survives). reconcileImage probes /etc/switchboard-uid-protocol for the exact supported protocol on every refresh cycle and on attach; if it is missing or invalid and nothing is in flight — inFlightCount() (thread exec/read/write + disposable /op users + attaches past their own image check, mid clone/install) AND the run registrations beyond those ops (item 44: a harness run's process lives in the container between operator calls, so the op counters alone once restarted the container under a live run; a deferred restart re-checks on every later attach and cycle until the resident is quiet) — it stops the container so the next start uses the current image — on a refresh cycle or the deploy's /reconcile ONLY: an attach never stops a container (issue 2101 — a deploy's restart loop was exactly attaches cycling the container and probing a fresh start still restoring). It answers a word (restarted/current/inactive/deferred/stale) so the deploy's fleet pass (item 69's /reconcile) can report each resident (disk is a cache; on a refresh cycle the gate throws CycleRestartError and the engine's retry re-enters the fetch step onto the restarted container thirty seconds on, item 7). On the attach path a stale verdict is stale, never a stop: the attach is refused 503 {error:"image-stale…", state, stateReason, reason:"image-stale"} and the bot falls back cold once — while the deploy's new-image report is pending (item 69) EVERY new attach is refused, so the resident can go quiet for the refresh cycle's restart and a new run takes the seeded sandbox; with no pending marker a busy container keeps admitting (deferred), as the deferral always did — and a re-attach owned by a live registration passes whatever the verdict, as it passes the drain and the memory gate. (Without it, an older image cannot qualify dynamic account provisioning.) (c) Idle sleep: on a refresh cycle that finds the resident SETTLED (warm at entry, or degraded with the SAME refresh-produced reason (github-unreachable, <step>-failed) for DEGRADED_PARK_AFTER_CYCLES = 3 consecutive cycles — a chronically broken default branch must not bill the container 24/7 for retries that cannot succeed; any other state resets the streak — a refreshing/restoring marker at entry is an orphan from an interrupted cycle and runs the full cycle instead, and a degraded stamped by the WATCHDOG (stale-mid-flight) or by an interrupted restore (restore-interrupted) never counts toward the streak and never settles: both mean a cycle must RUN — counting those was self-fulfilling and parked residents degraded for hours at a time; accepted cost: a resident oscillating between a refresh-produced failure and watchdog stamps never parks and bills full cadence — a watchdog stamp makes the prior "same reason" observation untrustworthy, and preserving the streak across it would re-open the hole), if no live binding attached within IDLE_AFTER_S (1h), no op is in flight, and no bounded run registration remains, the fetch is skipped, facts.idleSince is set and the cycle ends idle; the cron creates the next instance at IDLE_REFRESH_INTERVAL_S (6h) instead of every bucket (item 9; for a warm row only, a parked row in any other state is due every bucket) — so SLEEP_AFTER (20m) can elapse and the container sleeps. What a live tree holds is not asked: under item 17 a dirty tree no run is using protects nothing — the next attach wipes it — so it never pins the container awake (sleep destroys the disk and worktrees are not snapshotted; the sweep's release in (a) records what the tree held). The one-spawn probe (worktreeCleanlinessScript/parseWorktreeCleanliness in src/execution/residentCleanliness.ts: the presence test and both git probes in a single tagged-output script, the git probes privilege-dropped to the thread user) serves the eviction record of items 16a, 23, 45 and 55 and the disk-full recycle's record of item 54 — never a keep. Wake path is BOUNDED: attach → refreshIfStale (when idleSince is set or the last refresh is older than the active cadence) fetches the mirror under the lock (seconds) so the bound ref is current and clears idleSince, which puts the full refresh cycle (checkout rebuild if main moved) on the cron's awake cadence — the next firing creates its instance, within one bucket; an attach never waits on a checkout rebuild/install, so a wake cannot become a cold-fallback generator. isIdle() uses inFlightCount(), recent use (recentlyUsed: a live binding attached to or used within IDLE_AFTER_S), and bounded run registrations. The disk-full recycle of item 54 reads the same protections: a sleep and a recycle destroy the same disk, so neither runs while a registered run still owns it. Engine view carries idleSince; the dash shows "idle since …" / "awake". Cost effect: an unused resident bills ~20 min per 6 h instead of 24/7. Desired warm-pool members bypass idle parking; excess retained members use the normal policy. 16c. Race fixes on the wake, idle and sweep paths: refreshIfStale bounds its wake-fetch mutex wait by ATTACH_MUTEX_WAIT_MS (a full checkout rebuild holding the lock can no longer stall a wake attach for minutes — on expiry the attach proceeds on the last mirror and the next refresh instance repays it) and RE-READS facts before clearing idleSince (never writes a stale snapshot over a concurrent cycle's sha/lastRefreshAt; the idle branch of the refresh cycle re-reads likewise). inFlightCount() now also counts a refresh cycle past its idle/reconcile gates (refreshesInFlight), so an attach-path reconcileImage cannot stop the container under an in-flight rebuild — and only past them: the instance's fetch step is handed its count by runInstanceStep and calls it once refreshGate answered go (every other step first thing), so isIdle, reconcileImage("refresh") and the gate's disk-full recycle never see the probing cycle itself as an operation in flight — a cycle that counted itself before its gates would never park, never restart a stale image and never recycle a full disk. The sweep's clean-idle release and detachThread both re-read the binding right before evictBinding and skip when it is gone, evicted, or its lastAttachAt changed (a re-attach completed inside the clean-check await) — a fresh tree is never removed from a stale snapshot; evictBinding itself re-reads once more AFTER its awaited rm (mirror lock) and gives way if the binding moved meanwhile, so user:"" is never written over a re-attach that started inside the eviction (the sweep re-reads active per binding for the same reason — a container that woke mid-sweep must get the rm). The refresh cycle's final facts put also drops idleSince (it spreads facts read at the cycle's entry; without this a wake cycle resurrected the stale idle marker, showing a stale "idle since" and forcing the wake-fetch path on every attach).

  17. Worktree mechanism + dirty/stale discipline: per-thread trees are local git clone --no-hardlinks --branch <ref> clones of the mirror, NOT git worktree checkouts and NOT hardlinked object stores — a git worktree would keep the thread's index/objects inside the root-owned mirror (commits would need write access there), and hardlinked objects chowned to the thread user would let that user chmod inodes shared with every other tree. The clone is chowned to the thread's user; origin is repointed at the GitHub door for writable runs, so fetch/push use the revocable run bearer; the mirror itself is deliberately unreadable to thread users. A new run starts from a clean tree at the bound ref only after item 71 authorizes replacing the old tree. The run must save work it wants to keep; a paused or uncertain owner retains its private tree. Sweep, finished-ref reclaim and disk-pressure eviction all use the same decision, including when the tree is dirty — and every eviction records what the tree held on the binding beside evictedWhy: evictedLeftBehind: {uncommittedChanges, unpushedCommits} when something was there, evictedUnmeasured: <the probe's first error line> when git could not read the tree (still removed — nothing readable is being kept — but never recorded as clean), and one resident log line naming them (pure evictedTreeOf and evictedTreeSentence in src/execution/residentCleanliness.ts; the Worker's measureTreeBeforeEviction, read once before each eviction's re-read guards), so nothing is discarded silently. The disk-full recycle (item 54) destroys every live tree at once, so it requires item 71 permission for every live binding in addition to the idle gate and records what each tree held on the still-live binding under recycledLeftBehind / recycledUnmeasured: the disk may go away when no run is using it, never for what the trees hold. So on every provisioning attach an existing tree is examined AS THE THREAD USER: dirty tracked files (git status --porcelain -uno — untracked scratch files are the thread's own state and survive a reuse) or a stale HEAD (neither the mirror's current ref tip nor a local descendant of it — thread commits on top of the tip are kept) → the tree is wiped and recreated (recreated:true); otherwise it is reused byte-for-byte (recreated:false). A binding a rebind has just moved (item 16, either movement) leaves a tree on the old branch that this rule recreates like any stale tree, at the binding's path; nothing is checked out in place and no word from the rebind reaches this step. The only attach that keeps a tree as it stands is a resume — the same run continuing after the bot restarted (item 66, reuse): it keeps a readable tree whatever its dirt or HEAD and refuses one it cannot keep, touching nothing. A slept container loses /workspace/threads entirely (disk is cache); the binding survives in DO storage and the next attach recreates. Staged files (execution.md item 20) land in the worktree's attachments/, pulled by the run as the thread user, with attachments/ appended once to .git/info/exclude before the first pull — untracked and now ignored, so neither the reuse check nor any git add of the run's own sees them as work; they are the thread's scratch state and go with the tree.

  18. Dep/build materialization: after (re)creating a tree, the committed-lockfile key at the thread's sha (git ls-tree from the mirror — same pure-function key as the refresh engine) is compared to the warm checkout's recorded key. Equal → no install runs (reconciled:false) and each cached dir is materialized from /workspace/checkout by the mechanism the pure depCacheMaterialization (src/execution/residentDepCache.ts, imported by the Worker) names for it: node_modules is hardlink-copied (cp -al; deps:"hardlink") with only DIRECTORIES chowned to the thread user — file inodes stay worker1-owned and stripped of group/world write, so a thread can delete/replace entries in its own tree but can never mutate the inodes shared with the warm checkout or another thread; cp -al failure (e.g. cross-device) falls back to a plain copy (deps:"copy", fresh inodes, fully chowned). Build output dirs (dist build out .next) are ALWAYS plain-copied (fresh, fully thread-owned inodes): the review agent is instructed to run the project's build and /op build runs it too, and compilers rewrite dist/** in place (open+truncate through the existing inode) — with a hardlinked read-only dist/ the build died with EACCES and a review run reported the worktree as read-only. A copy shares no inode with the warm checkout, so the isolation invariant holds without the write-protection. The same class exists one level down, INSIDE the hardlinked node_modules: tool-managed paths that builds and test runs rewrite in place — every top-level dot entry (.cache babel-loader/eslint/prettier/webpack; .vite + .vitest vite's optimizer and vitest's results; .prisma the client prisma generate regenerates; .bin shims; .package-lock.json npm's hidden lockfile) plus any .cache directory at any depth (a package's own cache, node_modules/<loader>/.cache) — except .pnpm, pnpm's virtual store: a top-level dot entry that IS the package content, immutable after install, so it stays hardlinked (a .cache nested inside it is still copied). copying it made every thread tree of a pnpm workspace a full 2.6 GB copy of its deps and a fresh attach ~190 s, and one thread filled an 8 GB disk — so after the cp -al the Worker enumerates them with the exact find from mutableCacheFindArgv (a -path "<root>/.*" glob, never -maxdepth, which GNU find applies globally even inside parentheses), mutableCachePaths dedupes descendants under a listed ancestor, and each whose counterpart exists in the source is swapped for a real copy (rm -rf + cp -R + chmod -R u+w + chown -Rh; steps deps-mutable-list/rm/copy/chmod/chown); a tree-private dot entry with no store counterpart is left in place and named on the step's output (a skipped=<path relative to node_modules> line, which the Worker logs) — the listing is a find over the tree and the copies come from the store entry, and the two can disagree (one repo's tree listed a node_modules/.eports.d.ts its store entry could not stat); such an entry is not a shared inode to swap: deleting it is a regression and failing on it took the whole refresh down. The whole materialization runs as TWO container forks, not a spawn per probe/walk/path: depCacheScript (src/execution/residentDepCache.ts) handles all five dirs in one sh -c — per-dir src/dst gating, the cp -al with its directory chown AND file-write hardening as ONE combined find walk (( -type d -exec chown … + ) -o ( -type f ( -perm -g+w -o -perm -o+w ) -exec chmod go-w … + ); step deps-perms), the plain-copy fallback, the copy dirs, and the mutable-cache listing — emitting one tagged dir:<name>=<mechanism> line per materialized dir plus the raw mutable= listing, which the DO parses (parseDepCacheScriptOutput) and feeds through the unchanged mutableCachePaths into mutableCacheSwapScript (the second fork, all swaps). The resulting ownership/permission state is identical to the per-spawn version; failures still surface as named steps via err= tags. Packages themselves stay hardlinked and tamper-proof; nested dot dirs other than .cache (a package's vendored .bin/.github) are package content and stay shared. Other locations a tool might write (.tsbuildinfo, .turbo, coverage, .eslintcache, packages/*/node_modules, packages/*/dist) are not materialized at all — a thread builds them from scratch into its own tree — so they cannot hit this. The reported deps is decided by node_modules (the dependency cache the field describes; build-dir copies only fill in when the warm checkout has no node_modules — foldDepsMechanism). Differing key → the tree is SEEDED from the warm checkout exactly like the equal case (the same hardlinked node_modules, mutable paths swapped) and THEN the repo's install command runs inside the worktree, token-free, as the thread user, reconciling only the packages whose version differs (deps:"reconcile", reconciled:true; the decision is the pure planThreadDeps, item 58) — never an install from an empty tree; a differing key with no install command in the table seeds nothing (deps:"none"): a seed nothing can reconcile would be the wrong deps presented as ready.

  19. OS-layer thread isolation: each new owner receives a generated unprivileged account after a durable, serialized UID reservation (item 75). One owner's UID is never assigned to another owner within the VM generation. Same-thread attach remains locked through commit or rollback; the candidate stays reserved across native account creation and disk inspection. There is no fixed historical pool that forces a VM reset. Every /exec /read /write runs privilege-dropped via su -s /bin/bash <user> -c … with the worktree as cwd. Thread dirs are mode 700 owned by their user (peer threads cannot traverse in); the mirror top dir is root:worker1 750 (worker1 still fetches from it for refresh; thread users are denied at traversal); /workspace/.resident/ stays 700 root. The allocation is persisted in the binding; re-attach reuses it; eviction releases the live binding but never clears its durable UID spend. The resident's seat level reports canonical active or unknown owners and durable native-operation holds against the separate work limit, using cached canonical observations matched to the current owner fence, without counting terminal historical identities or starting a VM. Response levels make no state-Worker request; admission and maintenance refresh canonical observations, and admission also checks current native processes. Unknown ownership state reports full capacity until it can be verified. A refused account or private-tree inspection never authorizes another owner to use that UID. 19a. Identity allocation and active work have separate limits (decision 0095). Fresh identities are provisioned on demand, but a resident admits at most 16 active owners. New attaches and disposable operations reserve capacity under one shared short lock before clone or command execution; successful attach stores the durable registration before releasing its provisional reservation. The exact live owner may reattach at capacity. Fenced live or unknown owners, unresolved native operations, and current processes under spent UIDs remain occupied. Duplicate records for one owner count once. Verified terminal owners with retained files consume disk rather than a workload slot. Unavailable or mismatched owner evidence occupies a slot; an unreadable UID ledger or incomplete process observation refuses new admission. Foreign, unresolved or already-allocated operator work remains protected; allocating a fresh identity does not end it. Operator work keeps a capacity marker until native echoes confirm completion; an unknown outcome retains that marker across an isolate reset. workloadLimit reports the separate limit; poolUsersSpent remains historical identity use. None of these facts authorizes cleanup or UID reuse.

  20. Thread credential removal: on every door attach, including reuse of a worktree from an older Worker, the resident deletes .git/github-credentials and unsets its local credential helper before exposing the tree. A named-ref replacement also scrubs the prior checkout before admitting the new one: both remain under the same pool user's directory until cleanup, which is best-effort. An own-PR fallback scrubs the retained checkout before its branch probe runs as the pool user. A failed prior scrub refuses the attach. The resident then sets the writable origin to the Git door; model commands receive only their run bearer through the per-exec environment. A reattach without the door is refused. The older credential refresh decision module remains as a historical pure test utility and is no longer invoked by the Worker for thread commands.

  21. Thread exec/read/write: POST /exec {resource, threadKey, command, timeoutMs?} mirrors the thread-sandbox Worker's convention — the command body is bounded at MAX_EXEC_COMMAND_LENGTH (64 000 chars; a sanity guard far above any legitimate one-liner or heredoc, raised from 8 000 which refused real agent commands the sandbox/local executors accept — real file content belongs in /write at 512 KB; the bot-side exec wrapper's ( cd <worktree> && … framing counts against the bound) → over it, a pre-validation plain 400 naming the limit (agent-fixable: the client raises a normal Error, never ExecInfraError) — then immediate headers, a whitespace heartbeat every 15s, then one JSON document over HTTP 200: {stdout, stderr, exitCode, truncated} (5-minute default; the body's timeoutMs raises it per call up to the shared 20-min ceiling, clamped server-side by clampBashTimeout — execution.md item 11; timeout → exit 124 with a stderr note naming the limit and the timeoutMs knob; output capped at 100k chars/stream with a truncation note; post-validation failures arrive in-body as {error, needs?, stdout:"", stderr:error, exitCode:127}). POST /read {resource, threadKey, path} → {content, truncated}; POST /write {resource, threadKey, path, content} → {ok:true, bytes}. Paths are confined to the thread's worktree (conservative charset, no .. segments, resolved-prefix check → 400 path-escape), and the file ops themselves run AS THE THREAD USER — the OS layer, not just the prefix check, is what bounds a planted symlink; written content travels through the per-user staging dir, never argv. An unattached/evicted/disk-recycled thread gets a named not-attached/evicted/worktree-missing error with needs:"attach" (409 on read/write; in-body on exec). Exec and attach both bump lastAttachAt. Hot-path cost: only /attach consults the singleton registry (its 404 names not-onboarded, and the record rides on to the DO so attach never re-reads it) — /exec /read /write /detach skip the registry hop because they fail closed anyway: no binding → not-attached/no-binding, and a never-onboarded resource's DO has no state → 503 not-serviceable (that wording, not a 404, is the deliberate trade for removing a fleet-wide serialization point from every tool call). Output is bounded at the source: /exec, /op runs, and /read wrap the command via capWrappedCommand (src/execution/residentExecWrap.ts) — the full streams go to container-disk temp files (disk is a cache) and only the first capBytesFor(cap) = 4×cap+4 bytes of each cross the RPC, so a verbose test/build run can no longer materialize tens of MB inside the 128 MB DO isolate before the char-cap slice (the OOM → runtime-replaced-mid-run path). Exit code, stream separation, and the DO-side truncated logic are byte-for-byte preserved (4 bytes ≥ any UTF-8 char, so the char-count flag stays exact); the engine's own probes (cleanliness, credentials, markers) stay unwrapped. A timeout keeps its debugging clue: the kill skips the wrapper's own head/cleanup lines, so /exec and /op write to FIXED unguessable file paths (execCapFiles) and, on timedOut, run one follow-up command (recoverCapturedOutput) that salvages the same capped heads the wrapper would have emitted and removes the files — a hung test run answers exit 124 WITH what it printed before dying, as it did before source-capping. Recovery is best-effort (its failure returns the bare timeout result) and a still-writing orphan sees its file unlinked, reclaimed when it dies. Named edges: mktemp failing (disk full) exits 125 before the command runs; a recovery that itself fails leaves the two files until the container recycles. The per-request preflight's hydration probe is memoized per container incarnation (60 s TTL; cleared at the run() choke point whenever a RuntimeReplacedError is minted, on every lifecycle transition, and at every deliberate stop — a sleep cannot race the TTL because SLEEP_AFTER is 20 min of idleness while the memo lives 60 s past activity; the refresh cycle's 10-min cadence always re-probes for real). The same incarnation memos cover the one-fork git-setup script (was three forks per attach/op, /etc/gitconfig rewritten each time) and the per-user staging dir (write/credential paths fold their chown+chmod into one fork).

  22. Mirror mutex — a durable lease judged against the incarnation, and idempotent engine steps (pure src/execution/residentIncarnation.ts and src/execution/residentStepPlan.ts, applied by the Worker's withMirrorLock and its step methods fetchMirror / installDeps / runBuild / snapshot / restoreCheckout): a DO yields at every await, so concurrent requests interleave mid-handler — every mirror mutation (provisioning clone, refresh fetch, attach's verify-fetch + worktree create/remove, sweep removals, the rebuild's stage and swap — the build itself runs off-lock in the staging tree, item 47 — and the snapshot) runs under one mutex with two halves. In process, a promise chain queues takers in arrival order — the fast path within one incarnation. Durably, the row resident:mirrorMutex {holder, incarnation, expiresAt, step} is the truth across incarnations: the resident mints an incarnation id when its isolate starts and again whenever its container runtime is replaced under it — and only then: a lifecycle transition clears the object's memos and never the incarnation, so the cycle that flips the state to refreshing still holds its own lease afterwards — and a taker past the chain reads the row and takes it when free, when the holder's incarnation is not the current one (an isolate swap dropped the chain while the holder's process may still be writing), or when expiresAt — the step's own budget — has passed (the backstop for a holder that hung without a swap); only a live holder of the current incarnation is waited for, and a release deletes only the holder's own row. Attach waits under a named 60s timeout (ATTACH_MUTEX_WAIT_MS) → 503 {error, state, reason:"mirror-busy"} on expiry; a timed-out waiter releases its queue slot so later waiters are never stuck behind a ghost. The same lease shape carries the two in-flight facts the watchdog's stale-mid-flight branch reads — the refresh cycle and the hydration, in resident:inFlight, each alive only for the current incarnation inside its budget, and the branch clears a dead lease only against the holder it read, so a cycle that recorded a fresh lease meanwhile keeps it — and the per-key dependency install (resident:depsLease:<key>: the install's exclusive resource is the key's store entry, never the mirror, item 59; a live holder of the current incarnation is joined, a dead holder's scratch tree and staging dir are swept of build-user processes and removed before the next attempt starts, item 56). Every engine step is a public method that reads the facts it is about to change (a stored record, the disk markers, the store), asks the pure plan whether the work is done — done issues no command, so a second call with the same inputs has no effect — takes its lease, runs its commands under the step's own budget, writes its result and releases; snapshot commits with compare-and-swap on the record it read at the start and answers superseded (its fresh objects dropped) when the stamp moved meanwhile, never a throw; the recorded facts move to the stamp in the same write as the record. The refresh instance drives the steps one per step.do (item 7); /debug info shows incarnation, mirrorMutex and leases.

  23. Inactivity eviction keeps the binding: Each pass processes at most 16 live bindings and stores a cursor after each one. Later passes continue beyond that cursor; evicted rows use no batch space. The configured step budget stays at 21 minutes as the UID pool grows. the sweep step of every refresh instance (sweepWorktrees; TTL eviction plus the clean-idle release of item 16b — every ten minutes on an awake resident, at the idle cadence on a parked one, no timer of its own) considers worktrees whose lastAttachAt is older than the TTL, then removes only those item 71 authorizes (default 7 days; per-resident worktreeTtlDays 1–365 settable at onboard/reconfigure), keeps the user's UID reserved for that same owner until confirmed VM destruction, and KEEPS the binding record marked evicted with its ref and with what the tree held (item 17's record) — the next attach recreates the tree on the same ref, reclaiming its own clean UID when available or taking an unspent one. A sleeping container is not woken to inspect files; a known owner without fresh current-file proof remains bound. Debug ops threads / backdate-thread / sweep-now exist to enumerate bindings and exercise the TTL path without waiting days.

  24. Bot-side selection is warm-gated and probe-per-dispatch: makeExecutor(opts, {threadKey, agent, profile, repo?, ref?}) returns {executor, note?}. Resident selection requires a run whose effective profile is on the repo-resident machine class (execution.md item 7), a configured execution.resident {baseUrl, tokenEnv?, probeTimeoutMs?} (operator bearer read from tokenEnv, default RESIDENT_OPERATOR_TOKEN), AND a resolved ctx.repo — populated by the repo resolver (item 29); undefined means the per-thread path with NO probe (total input contract). Membership is never cached: each dispatch makes one operator GET /status probe. A probe the Worker answered with a 5xx it typed transient (the catch-all 500: the Durable Object reset or lost under the probe) is waited through — re-probed at once and then under the first attach's budget, the run's stop riding in, the wait named on the card — for a fresh run's selection, a resumed run's re-attach and a mid-run relaunch's alike, and never opens the transport breaker below (execution.md item 9). A serviceable state → ResidentExecutor (isServiceable in src/execution/residentState.ts, the lifecycle union shared with the resident Worker so a renamed state is a compile error on both sides): warm with a POSITIVE note = resident · <owner/name> · <ref>@<sha7> taken from the attach answer (the resident path is named on the card, never inferable only from the absence of a fallback note); every OTHER note — a failed attach, an unreachable or unserviceable resident, a repo not onboarded — is also published to the run's stream as a cold_sandbox run note with the same text, after the attach and before the first turn, head material like the rest of the setup (tracing.md item 16), so the run page explains a sandbox run whose attach span carries the resident's grafted steps — the card is not the only witness; refreshing (the resident keeps serving the last snapshot — item 7 — and /attach refuses nothing; the mirror lock serializes the attach against a refresh's fetch and its stage/swap (item 47 — the build runs off-lock, so the wait is git-network-sized, never build-sized), bounded by ATTACH_MUTEX_WAIT_MS → a mirror-busy 503 falls back like any attach failure; so does a disk-pressure 503 — item 55 — whose error carries the budget math onto the card); and degraded only for the one reason class that proves the checkout intact — github-unreachable: … (fetch failed before the checkout was touched); not disk-full: … (item 54 — the checkout may be intact, but no attach can succeed on a full disk, and recording that failure as github-unreachable sent every run to attach and die at git-setup); not stale-mid-flight: …, whose orphaned cycle may have died inside the rebuild (the next refresh instance rebuilds within one bucket anyway) — because an orphaned cycle's state is unknown from outside: the reset/clean, install and build run in the staging tree and a failure there leaves the warm checkout untouched at its old sha (item 47), but a cycle that died inside the swap's lock section (checkout-swap-failed between its two renames, or snapshot-failed / refresh-failed after them with the markers half-written) can leave the checkout torn, and a fresh thread whose lockfile key matches would hardlink that cache; those, and unknown reasons, stay cold until the next cycle rebuilds. A non-warm attach carries an informational note = resident <state> (<reason>) · <ref>@<sha7> — attached to the last snapshot on the card. Gating on warm alone (the original rule) sends every run cold for the whole of every refresh window — with an active default branch (each merge a rebuild on the 10-min cycle) plus each resident deploy's restore, that is most of a working day of resident refreshing — using fresh sandbox. 404/not-onboarded → the ordinary per-thread path (AE4 — no resident exists, nothing degraded) but with a visible note = repo not onboarded as a resident — running in a cold per-thread sandbox; onboard it (`repo onboard <owner/name>`) for a warm, deps-ready environment so the cold fall-through is never a silent surprise; the engine-owned states onboarding (nothing to attach), restoring (disk being rehydrated), down (only a rebuild escapes) → per-thread fallback with note = resident <state> (<reason>) — using fresh sandbox. Every fallback note rides on the dispatcher's status-frame title (a degraded or cold-fallback resident is loud, never a silent stall). A resumed run's re-attach never falls back (ExecutorContext.reattach, run-history.md item 54): the backend its row recorded is the only one consulted, in reuse-only mode (item 66), and a refusal, an unreachable or unserviceable resident, or an answer naming another worktree or pool user than the row's is a WorkspaceReattachRefusedError the dispatcher restarts the run on; the cold_sandbox note is a fresh run's alone.

  25. Probe circuit breaker: probe TRANSPORT failures (a fetch error, or the probe's own deadline — probeTimeoutMs, default PROBE_TIMEOUT_MS 8 s — passing) fall back as resident unreachable (…), and of those only a failure to REACH the host arms the 30s in-process negative cache, so a resident-service outage costs one failed connect, not one per concurrent dispatch. A deadline miss arms nothing: the host was reached and answered slowly (/status fans out to the registry and four resident reads on a Durable Object that may be busy), which is that dispatch's cold fallback and nobody else's — the probe marks it timedOut beside transport, and the same rule holds for the /residents listing. The deadline was 2 s and a miss armed the window; one miss in 1,633 probes on a healthy resident then cost its run a cold, unseeded sandbox and every dispatch of the next 30 s the same, so the deadline is wide enough for the route's fan-out and a miss is no longer read as an outage. A resident that is gone fails the connect long before the deadline, so the wide deadline costs a dead resident's dispatch nothing extra; a wedged one costs its dispatch the deadline once; the run's own stop aborting a probe — the first probe or a re-probe of the blip wait of execution.md item 9 — is the stop, never the outage, and arms nothing; the edge's own page — a 5xx with no Worker document — that the selection's blip wait was spent on — more of the wait's views document-less than Worker-typed, never its last view alone — arms the window at the wait's end (the platform not answering, paid once), while a wait spent on the Worker's typed blip never does. Definite lifecycle answers — including degraded/down — are NEVER cached (the next dispatch must see a recovery immediately). In-process only; the restart-survival invariant is untouched.

  26. Per-repo gate: the dispatcher checks the resolved actor's compiled code access BEFORE executor selection: explicit codeRepos, otherwise the repos fallback, plus additive human and surface grants; delegation intersects code independently of memory ownership. A matching code-only grant admits a listed restrict.repos repository without requiring memory ownership. A credential with repos: all and codeRepos: [] is refused unless another code grant covers the repo. Human baselines admit unlisted code repos; credentials and schedules need explicit code access. A refusal is a named 🚫 reply, never a silent per-thread fallback, and no executor is created.

  27. ResidentExecutor client (attach-on-open): open() attaches before the first tool call so needs-ref / not-onboarded / mirror-busy surface legibly at selection time — the 409 needs:"ref" answer throws a TYPED ResidentNeedsRefError the dispatcher catches for the ask-once flow (item 30). Every route POSTs {resource, threadKey, ...} with the operator bearer and no x-env headers; /exec responses are read as full text (heartbeat whitespace + one JSON document) and parsed from the body, never the HTTP status; a non-zero exitCode is a result (exit N: …), not an error; an in-body {error} without needs is thrown verbatim and never blind-retried. A needs:"attach" error on any op for an evicted worktree (evicted:, not-attached: — the container stands, the binding survives) triggers exactly one re-attach + retry of the same call; a second needs:"attach" is a legible error. The preflight's worktree-missing: (the container disk was recycled since the last attach) says the container under the thread is gone with every process the run had in it: on /exec it is the typed ExecSandboxRestartedError at once, carrying the resident's words, with no re-attach and no retry (saysContainerGone, containerGoneMessage in src/execution/residentWake.ts) — the pi harness reads it as the container replaced under the run (harness-pi.md item 16) — and on the idempotent routes the one re-attach + retry as for any needs:"attach". A reason:"runtime-replaced" answer is handled per item 43 (/exec: the same typed restart at once; /read, /write: re-attach once and retry).

  28. Plumbing: RESIDENT_OPERATOR_TOKEN reaches the bot container through the Worker shim's explicit envVars allowlist (deploy/cloudflare/worker.ts + deploy/secrets.manifest.json, the same path as SANDBOX_TOKEN); the production config (config/config.production.yaml) points execution.resident at the resident Worker's URL. src/cli.ts accepts --thread <key> / --thread=<key> or SWITCHBOARD_THREAD (default stays ephemeral cli:<timestamp>) so repeated CLI invocations act as ONE thread — the vehicle for re-attach/binding-persistence verification.

  29. Repo/ref resolved BEFORE the model turn: src/core/repoContext.ts is the production default behind the dispatcher's CoreDeps.resolveRepoContext seam (still injectable for tests), run only for an agent whose machine class carries a checkout and with the vet that class names (execution.md item 18): the registry probe and listing below for repo-resident, GitHub for repo-cold; none and blank never resolve a repository. An accepted non-review operation target supplies its repository and any accepted ref before message or history extraction, without a discovery probe; a contextual PR cannot replace either. With a repository-only target, an explicitly requested same-repository PR URL, shorthand or bare number may supply its verified open head; conflicting requested PRs refuse before a fetch, while contextual citations remain evidence. Otherwise, extraction sources in priority order: explicit signals in the CURRENT message — an owner/name slug token, a github.com repo URL (Slack markup <url|label> unwrapped; /tree/<ref> yields the ref too), a PR URL or owner/name#N shorthand — then the thread's previously-established repo derived from history like lastThreadDirectives (user turns, last wins; restart-safe, never stored), then none → {} → the per-thread path with no probe (AE4). The thread's own pull request binds the follow-up: when the current message names no PR, the thread's PR is the newer of the last one a person named in a user turn and the one the thread's newest finished run opened or edited (RunRecord.pr, run-history.md item 2, handed in from the dispatcher's one read of the thread's runs, routing-and-config.md item 3) — the turn's timestamp against the run's end, and the run's when either is undated, since it is the thread's own work. The record's PR binds the ref to its head branch when the message phrases none (refFromPr, with prFromRecord saying the PR is the thread's own), and the resident honours it: the attach carries the PR as ownPr (ownPrOf) and a thread bound to the repo default moves onto that branch, once per pull request — the tree provisioned clean at the branch the run's release told the resident it pushed (item 16, item 16a) — so a coding follow-up pushes to the PR and its resubmitted description edits it instead of being refused on the repo default — from whatever ref the follow-up ran on: when the run pushed nothing and its workspace sits on the base (the binding still the repo default, the move refused or the tree recreated there), the post-step edits the thread's PR by number, the body rendered at the PR's own head, rather than refusing "the branch is the base" (recordPrOf, CodingPrTarget.ownPr; pr-description.md item 5). The card reads resident · <owner/name> · <head>@<sha7> · rebound to <head> (this thread's PR #N), or names the refusal that kept the thread where it was (· rebind to <head> (this thread's PR #N) refused: <reason>), which the run's stream repeats as a rebind_refused note — or, once that pull request has merged and its branch is gone, · returned to <default> (the branch of this thread's PR #N is gone) (item 16's second movement). A person's PR keeps the rule below (inherited fail-closed, the ref never rebound, ownPr never sent), and a ref in the message wins over both. A closed or unfetchable PR binds no ref and no pr (prUnpostable says why) — but the record's own PR, once merged or closed, is still handed to the coding post-step as closedRecordPr with the head commit GitHub names for it, so a description resubmitted in its thread edits its body (pr-description.md item 5); a turn's closed PR is unpostable and nothing more; a record's PR of another repository is not the thread's. Addressed repos (without this, a thread bound to one repo by an issue link keeps every later in acme/api run on that repo's resident — a bare slug is weak and could not rebind — while an opening in api carries no signal at all and goes cold): the first token after the word in (outside code) names the request's TARGET — in owner/name anywhere in the message, or a bare in name in the DIRECTIVE position only (nothing but agent:/model: directives or a mention before it — agent:coding in api, …; "the crash is in api, see the logs" is prose even when a repo is called api). Vetted against the resident registry it is STRONG like a URL: it binds a fresh thread and rebinds a bound one, and "last strong wins" holds across URLs and addresses alike (the resolver walks the thread's user turns backwards, vetting each address). A bare NAME resolves only through the registry listing (GET /residents, the admin bearer — the route is read-scoped) and only when exactly one onboarded repo carries it; an unknown (in production) or ambiguous name is prose and binds nothing, a failed listing is no answer. The strength and code-span guards below stand: an address the probe refuses (in try/catch, in docs/reference/specs/memory.md) changes nothing unless the current message also names a bare PR number, in which case resolution stops rather than borrowing the thread’s older repo, a merely-mentioned onboarded slug ("also check acme/web") stays weak, and without a probe (local/dev) nothing can be vetted so an addressed slug stays the weak token it always was. One probe per candidate per resolution. Silence is not a refusal: the probe answers "unreachable" (transport failure, timeout, the outage window) apart from false (not onboarded), and when the current message addresses a slug the registry could not be asked about, the resolver returns unverifiedRepo — never a silent fall back to the thread's old repo — and the dispatcher replies "couldn't verify … the resident registry didn't answer" without starting a run (an unanswered address in history is simply no answer; a fresh thread whose only candidate went unanswered reports unverifiedRepo rather than "not onboarded"). The run card names the bound repo (resident · <owner/name> · <ref>@<sha7>) so a wrong binding is readable from Slack, not only from a sha. Without an accepted operation target, an explicitly requested PR resolves its repository and head through RestGithubApi.getPullRequest using read-scoped credentials — NEVER a gh shell-out from the dispatcher (AGENTS.md invariant 5). Missing or failed credentials stop before an HTTP request; there is no unauthenticated fallback. An open same-repository head also reads its branch tip; an unreadable tip keeps the PR object's head, and closed or fork heads remain frozen. A failed metadata read leaves the head and ref unresolved, and cross-fork head refs are never bound (they don't resolve in the mirror). With an accepted operation target, the fetch runs when the current message explicitly requests a PR of the accepted repo — regardless of ref phrasing beside it; a contextual citation does not trigger it. The legacy non-review path can still fetch a contextual PR extracted as namedPr (agent-review.md §11): a fetched open PR supplies the ref unless an accepted operation ref already fixed it; a prose on main / on branch X in the same PR-target message binds no ref, including when the fetch fails. Only an OPEN pull request contributes the ref hint (refFromPr is set only from an open one; a state the answer did not carry binds nothing): a merged or closed pull request cited as a receipt in a coding ask must not bind the thread's branch to its dead head branch — a ship child bound there has its contract branch refused by the resident's push guard and the commit orphans (issue 1860) — while its facts (pr, headSha, prDescription) still ride as context and a phrase's ref stands. And a coordinator child's ref hint is its contract's branch, always: a PR-derived ref the resolver bound off the child's own request text is dropped by the dispatcher before the attach — the drop refFromPr's contract asks of a consumer that does not bind the PR — with the head sha that pinned the cited PR's head (a review child keeps the binding: the pull request is its target). For a non-review run, a message-cited PR that did not bind the ref (prFromMessage without refFromPr) contributes an attach SHA only when its verified open head matches the accepted operation ref; the dispatcher drops every other headSha before attach, so a merged PR's frozen head stays context — the card, the record — never item 51's expected commit (which the resident could only refuse stale-tip, sending the run cold at the dead commit). (Before, a bound ref skipped the fetch — a re-review saying "rebuilt on main" lost its head SHA and the review was refused.) The ref is a typed slot, never a prose guess: only the head-of-ask routing clause (… in owner/name on branch X: task), a branch:X token, a pull-request URL's fetched head or a /tree/<ref> URL fills it; every candidate is validated against the resident's strict ref pattern (no ../@{/.lock); slugs are validated owner/name shapes, lowercased (resident resource ids are lowercase) — invalid tokens are ignored whole, never partial garbage. Code never establishes a repo: a token inside an inline code span (`unset/unset`, `src/core`) or a fenced code block (pasted logs/diffs) is code or a path being talked about, not a repo switch, so ordinary target extraction ignores it, including URL/PR and typed branch forms; the coordinator's routed-PR conflict gate still checks quoted PR citations. Thus on `main` remains prose (src/core/repoContext.test.ts::a token inside a code span never establishes a repo…, ::a slug inside a fenced … block…, ::backticked prose does not bind a ref…, ::repoFromThread ignores code-spanned tokens in history too). An on <owner/name-shaped> token is only a legacy repository mention when no repository is independently established; it never becomes a ref. Signals have two strengths, and the thread binding is sticky against weak ones: a github.com URL (repo/PR//tree) or owner/name#N is STRONG — unambiguously a repository; a bare owner/name-shaped token is WEAK — it is also the shape of every relative file path and of ordinary prose, and code spans only catch the backticked ones. A thread bound by a strong signal is rebound ONLY by another strong signal (in the current message or a later user turn); weak tokens are consulted only while nothing strong has bound the thread. Otherwise a prose token shaped like a path (e.g. docs/reference/specs/memory.md) would select a cold sandbox for a non-existent repo AND unbind the thread's PR. (src/core/repoContext.test.ts::a bare slug-shaped token never rebinds a thread bound by URL…, ::a bare slug in an EARLIER follow-up…, ::redirecting the thread to another repo BY URL…, ::repoFromThread: the last STRONG user-turn signal wins….) Bare tokens are guarded two more ways. (1) A bare token never overrides a repo the thread already established, whatever its strength — the first binding stays until a STRONG signal replaces it, so a thread opened with a bare in owner/name is not hijacked by later prose (reflection/review-post, try/catch, comment/spec are all slug-shaped). (2) In a thread with no repo yet, a bare token binds only when the resolver's injectable resident probe (resolveRepoContext(msg, history, isResident?) / repoFromThread(history, isResident?)) confirms it names an onboarded resource; the production probe (residentOnboardedProbe in src/execution/factory.ts) is one operator GET /status through the same negative cache as executor selection — any lifecycle state of an onboarded resource is a yes, not-onboarded and every unreachable/error answer is a no (fail-closed). No resident configured, or its bearer unset → no probe → bare tokens bind unvetted as before (local/dev). A refusal is reported when it is the whole story: when nothing binds and the first bare candidate the probe refused exists, the resolver answers { rejectedRepo } and the dispatcher — for a repo-needing agent only — closes the ack card not started (repo not onboarded) and replies 📦 with the slug, repo onboard <slug>, and the github.com URL form as the way to run cold anyway; no attach, no model turn (before: an empty workspace and fatal: not a git repository). A thread that already has a repo carries rejectedRepo only when a refused current-message address and a bare PR number would otherwise select its older repo; otherwise the silence is unchanged. (src/core/repoContext.test.ts::bare prose slugs never hijack a thread …::*, ::a STRONG repo signal in the current message beats the thread's; a bare slug does not, ::repoFromThread: a bare slug binds a thread that has no strong signal — first bind wins….)

  30. Needs-ref binds to the repo default loudly; asks only when it can't: when attach answers 409 needs:"ref" (typed ResidentNeedsRefError) the body names the resident's defaultRef (item 16), and the factory re-attaches ONCE with that ref — the message named no branch, and the cold path already works on the default branch without asking, so the warm path no longer costs an extra human turn on every fresh thread. It is never silent: the note becomes resident · <owner/name> · <ref>@<sha7> (repo default — no branch named), and the binding is made by that explicit second attach, not guessed resident-side. A 409 that carries no defaultRef (a Worker predating the field) keeps the ask-once flow: the dispatcher replies ONE clarifying question naming the repo — 🌿 "Which branch of <repo> …" — before any run card exists and without any provider call (mirrors the named-refusal reply shape); the user's branch:main answer in the thread is an ordinary next message: the resolver reads the typed ref from it and the repo from thread history, and re-attach binds it. A second needs-ref after binding by default is a resident bug and surfaces as a plain named error, never a loop. Either way the binding persists resident-side as the sticky fallback: an attach that explicitly names a different ref replaces it and records that named ref as the new fallback. A hint derived from the repo default or the thread's own pull request keeps item 16's fallback movement instead (the by-default re-attach says refByDefault: true, so the resident records the binding as the kind that may follow the thread's own pull request).

  31. Resident prompt variant, selected after executor resolution: the dispatcher composes the effective system prompt AFTER makeExecutor returns — executor is a ResidentExecutor → the agent's residentSystem (exported as CODING_SYSTEM_RESIDENT / REVIEW_SYSTEM_RESIDENT in src/agents/registry.ts) plus a line naming the resolved repo; any other executor → the agent's own prompt untouched. The variant flows through RunOptions.system; the shared AgentDef is never mutated (concurrent dispatches share it). Variant content: the workspace is a READY git worktree on the thread's bound branch, deps installed, build warm — no cloning, no installs, no repo discovery, no scope-first survey; gh is NOT in the resident image (git + node only), so coding pushes with git through the Git door using its run bearer, calls diff_digest to shape the description, and submits it through submit_pr_description — Switchboard renders the body at the observed pushed head and opens (or edits) the PR from the bot process, so the prompt carries no PR-creation API call at all (pr-description.md item 5). It is told honestly that the Git door may refuse authentication — an auth-refused push is said plainly, never retried blind — and a run that pushed without producing a PR gets the branch compare URL reported honestly by the dispatcher, never a claimed PR. A sandbox seeded from this resident's snapshot (execution.md item 26) has its own variant, chosen the same way after executor resolution; resident and seeded never both hold.

  32. Repo-management commands, fail-closed: repo onboard <owner/name> [--ref <branch>] [--test "<cmd>"] [--build "<cmd>"] [--install "<cmd>"] [--evict-coldest] / repo reconfigure <owner/name> [--ref …] [--test …] … / repo offboard <owner/name> [--dry-run] / repo rebuild <owner/name> [--dry-run] are registry commands (repo.onboard|reconfigure|offboard|rebuild in src/core/commands/repo.ts, command-registry.md item 20; the admin client and the slug/ref validators live in src/core/residentAdmin.ts): answered inline through the registry's chat adapter, never a model turn, reachable as /api/repo.*, MCP repo_* (repo:write) and the CLI, all gated in chat by repoManager = canManageRepos — fail-closed in src/config.ts: repo:write is never a baseline — ADMINS ONLY until granted — because these commands provision billable always-on compute and bind GitHub credentials. A refusal is a 🚫 reply naming the admins. repo list is the command registry's repo.list (src/core/commands/repo.ts; command-registry.md item 17): repo:read scope, chat gate open, the JSON is the resident Worker's /residents body and the text is the historical repo list reply (renderResidentList) — so the same read-only live registry view is now also GET /api/repo.list, MCP repo_list, and CLI repo list; parseRepoCommand still recognizes the verb and handleRepoCommand yields it (null) to the registry stage, and the resident admin client is resolved per call by residentAdminFromConfig (shared with the mutating verbs). Only the known verbs match — prose like "repo onboarding is done how?" flows to the model untouched. Parsing is key="value" tokens (curly quotes from Slack autoformat normalized), slugs are validated owner/name shapes lowercased, refs pass the resident's strict pattern, and omitted onboard commands are DETECTED from the repo root (item 52; the npm table npm install --no-audit --no-fund / npm run build --if-present / npm test — the table the first residents were onboarded with — is only the fallback when the root cannot be inspected, and the reply says so). The commands reach the resident Worker's admin routes via a fetch client using RESIDENT_ADMIN_TOKEN (env name settable as execution.resident.adminTokenEnv); the bearer reaches the bot container through the Worker shim's envVars allowlist (deploy/cloudflare/worker.ts + deploy/secrets.manifest.json), same path as RESIDENT_OPERATOR_TOKEN. Because the resident's /reconfigure replaces the whole command table, a partial chat repo reconfigure merges its patch onto the current table (fetched live from /residents) before posting.

  33. --dry-run is the stateful-op checkpoint: POST /offboard {resource, dryRun:true} answers 200 with {resource, dryRun:true, wouldRemove:{registryRecord, schedules, snapshotBackupIds, backupObjects, r2Objects, threadBindings, container}} — the ids and the count covering every recorded generation (item 7) — the same itemization the real teardown reports, computed READ-ONLY (R2 objects counted, never deleted; schedules/state untouched). POST /rebuild {resource, dryRun:true} answers 200 with the rebuild plan (item 34) and executes nothing. Both chat commands render the plan and say "Nothing was changed", naming the non-dry command to run.

  34. Rebuild is the down→onboarding escape hatch: POST /rebuild {resource} (admin) discards the recorded snapshots, every generation — R2 objects deleted FIRST, before initResident wipes the stored handles (same ordering as teardown) — forgets the SDK's runtime identity and destroys the container (destroy(): SIGKILL, a fresh VM and disk; a rebuild that reprovisioned onto the running container once inherited its wedged runtime, item 64), and reprovisions from scratch through the ordinary provisioning pipeline, reusing the registry record's command table/ref/budget. Answers 202 with {…plan, backupObjectsDeleted, state:"onboarding"}. The registry record (cap slot) and thread bindings are KEPT (worktrees on stale shas are recreated by the next attach); refused with 409 while the engine is mid-flight (onboarding/refreshing/restoring) so two cycles never race one disk. A rebuild whose provisioning then fails follows the ordinary onboard failure paths (named down reason; timeout releases the slot).

  35. Onboard checks installation membership, honestly: when the GitHub App is configured, /onboard requires the repo to already be in the installation's repository list — proven by a repo-scoped mint (the token API refuses repos outside the installation), so the check IS the scoping mechanism it protects; failure → 403 {error:"not-in-installation: …"} BEFORE any registry insert (no cap slot consumed, nothing provisioned); the message says the two things GitHub's 422 can mean (repo not in the installation's repository list, or no repo under that exact name) and where an admin fixes it (the App installation's Repository access settings). When the App is NOT configured the check is SKIPPED — never faked — and the 202 carries warning:"github-app-not-configured: installation membership was NOT verified …", which the chat reply surfaces.

  36. Auto-rebuild on the down transition, budgeted (pure REHYDRATION_FAILURE_RE, isAutoRebuildEligible, autoRebuildDecision, AUTO_REBUILD_BUDGET = 2, AUTO_REBUILD_WINDOW_MS = 24 h in src/execution/residentAutoRebuild.ts; the Worker's goDown → autoRebuildFromDown, the watchdog's down branch, rebuild(…, {auto})): a resident down with a rehydration-flavored reason (r2-restore-failed / snapshot-stamp-mismatch / no-snapshot / runtime-unreachable — the states only a rebuild can escape, since a down resident runs no cycle; the last is item 64's final rung, a recreated container that did not answer either) is rebuilt at the transition: goDown persists down, judges the reason against the history row resident:autoRebuilds (the ISO instants of this resident's auto-rebuilds, pruned to the window) and, with budget left, runs item 34's rebuild at once — destroy, then provisioning armed through its own schedule (the one timer, item 3) — before it hands the ResidentDownError back, so the caller's step ends as it always did and the row reads onboarding by the time anyone polls it. The instant is recorded once the rebuild has started (a refused rebuild spends no budget), and a failure inside the judgement is logged, never thrown — the caller's error is the ResidentDownError either way, and the state is already down for the backstop to find. The stamp names no reopen instant: a stamped row is never judged again and runs no cycle, so nothing resumes when the window passes; only a person's rebuild recovers it. Before this the watchdog counted one strike per pass and rebuilt at three (~30 min at the 10-minute cron): a wait that bought nothing, since none of these reasons recovers on its own — a down row runs no cycle, and a bounded wait standing in for an event is a race (0029). The budget: AUTO_REBUILD_BUDGET rebuilds per AUTO_REBUILD_WINDOW_MS; the one past it stays down with the reason stamped — auto-rebuild budget spent (2 in 24 h) — repo rebuild resets it, written once (a stamped reason is never judged again), so a resident that comes back warm and goes down again on something a rebuild does not fix (a stamp mismatch that recurs, a backup store that keeps failing) stops where a person sees it on the fleet page instead of rebuilding every half hour forever. A person's /rebuild (item 34) and an offboard clear the history; the automatic rebuild keeps it — the history IS the count. Provision-failure downs never auto-rebuild (they would loop against the same broken build). The watchdog is the backstop, not the trigger: its down branch takes the same decision over the same row — for a transition that found no registry record (an offboard mid-flight) or a down from before this rule — and reports action:"auto-rebuilt" with the reason prefixed auto-rebuild (watchdog):; the firing record counts them (· N rebuilt, only when N > 0, live-view.md item 14). A serving pass does not clear the history (a flapping resident is judged with its earlier rebuilds in view); the strike row it used to keep (resident:rebuildStrikes) is retired and deleted on a serving pass. POST /debug {"op":"force-down", "transition": true} fires the transition itself (the rebuild or the stamp happens in the call); bare force-down persists the state alone, so the watchdog's backstop is provable live. /debug info carries autoRebuilds. What the transition does not decide: a degraded streak — item 64's ladder covers the unreachable runtime, and item 67's covers every other failure in the resident's own steps; both end in a down this transition rebuilds.

  37. POST /op is the deterministic modelless path (operator scope): {resource, op, ref?} where op resolves ONLY through the fixed enum {test, build, status} into the onboard-time command table — request text is never interpolated into a shell command (the command STRING is admin-written; the ref rides as a git argv element). ref passes the strict branch pattern at the Worker (400 on failure) and must additionally resolve in the mirror (fetching once under the mutex if unknown; still unknown → in-body unknown-ref); absent ref = the resident's default branch. Not-onboarded → 404 (like /status). test/build stream with /exec's convention (immediate headers, 15s whitespace heartbeats, ONE JSON document over HTTP 200; post-validation errors in-body) and answer {ok, op, resource, ref, sha, summary, stdout, stderr, exitCode, truncated, deps, reconciled, durationMs} — ok is the command's verdict: a failing run is a RESULT (ok:false with a named summary), never an error path. status touches no checkout and no exec at all — DO storage reads only — answering lifecycle + sha + lastRefreshAt (ok = not down; degraded still serves).

  38. Ops run in disposable per-op checkouts, never a thread's worktree: each test/build op transiently allocates a pool user (in-DO set, excluded from AND excluding thread allocations — an op never shares an OS user with a thread), clones --no-hardlinks from the mirror under the mirror mutex into a 700 user-owned /workspace/ops/<uuid>/checkout, materializes deps through the EXACT thread mechanism (materializeThreadDeps: same committed-lockfile key → cp -al from the warm checkout; differing key → scoped token-free install, reported as reconciled:true), runs the table command privilege-dropped with the 20-minute exec ceiling as its budget (OP_EXEC_TIMEOUT_MS = BASH_TIMEOUT_MAX_MS — the deterministic op path has no caller knob, a full suite routinely outlives the 5-minute per-command default, and the client bounds its /op wait at the same ceiling), and DELETES the checkout in a finally (success or failure) before clearing the live op allocation; the UID stays spent until confirmed VM destruction. No snapshot is ever written by an op; pool exhaustion is a named in-body user-pool-exhausted error. Op users count into the seat level like thread users (orchestration-plane.md item 9) — an op-saturated pool reads above and the plane's observation of the refusal maps to the seat condition.

  39. Per-entry effects profiles gate the modelless path: the registry record carries effects?: {<command>: "readonly"|"mutating"} (admin-writable only — /reconfigure accepts it and REPLACES the whole map, {} restores the default; absent entry = readonly, so test/build/status are readonly by construction). /op refuses a non-readonly entry with 409 {error:"op-refused: …"} naming the entry and its profile — the guard rail for future mutating entries; the refusal reaches the user verbatim as a 🚫 reply.

  40. Bot-side Operations seam, two implementations: src/core/operations.ts defines named op → structured {kind:"result", ok, summary, output?} with distinct non-run kinds (refused, not-onboarded, error) — the 3-method Executor cannot express modelless ops. ResidentOperations (src/execution/resident.ts) posts /op with the operator bearer and parses the streamed body (404 → not-onboarded; op-refused… → refused; other in-body error → error; a failing run stays a result). LocalOperations (src/execution/executor.ts) is the dev/CLI implementation against the thread's local workspace dir — honest about its limits: fixed Node conventions (test → npm test, build → npm run build --if-present, status → workspace existence), a requested ref reported as ignored. The dispatcher builds the default per dispatch (resident configured → resident-backed; local execution → local; other per-thread backends → none) and CoreDeps.operations injects for tests.

  41. Deterministic asks answer with zero model turns: (a) the explicit commands repo test <owner/name> [<ref>] / repo build <owner/name> [<ref>] are the registry's repo.test|build (src/core/commands/repo.ts; command-registry.md item 24): bound in stage A by the shared grammar — a hostile ref or slug fails the schema with a named refusal (ref: expected a plausible git branch ref (e.g. main)) before any backend, the chat gate is agentRun (= the coding allowlist, NOT repoManager), canUseRepo is checked inside, machine callers need repo:exec; (b) the natural forms — run the tests on main in acme/api, or run the tests on main as a reply in a thread whose turns name the repository (repoFromThread, shown to the router as the thread's repository) — are prose, so they reach the same repo.test|build through the router's door (routing-and-config.md items 7 and 21): one call to the fast model binds the slug and ref under the command's own schema (a hostile ref fails that schema at invoke, refused by name), the command runs at once as an exec-class write, and the reply leads with the receipt routed: repo test acme/api main; an agent:/model: directive skips the router, so no command binds there. ONLY the model call differs between the two doors — none on the typed form, one router call on the natural form, never an agent run on either: the implicit target agent is coding, so canRunAgent(user, "coding") answers the exact refusal a normal coding request gets, and canUseRepo (item 26) refuses by repo name — both before anything executes. A result posts as ✅/❌ + summary + tail-clipped output; a refusal posts 🚫 verbatim (under the receipt on the routed form); a not-onboarded repo, a backend error or a refused table entry is a named ⚠️ reply on both doors — the routed form leads with the receipt at verbose (routing-and-config.md item 28), and nothing falls through to the agent (the person's next message is the follow-up, routed fresh). On both doors the /op call is bounded by the exec ceiling on both sides — the client's /op wait (src/execution/resident.test.ts::ResidentOperations.run::bounds the /op request with an AbortSignal.timeout at the exec ceiling…) and the server's run budget (deploy/cloudflare-resident/opExecTimeout.test.ts::op exec budget (resident-repos item 38)::the op run itself is capped by OP_EXEC_TIMEOUT_MS) — so a long suite finishes and a hung resident still answers.

  42. Residents dash — the browser twin of repo list (src/channels/residentsView.ts, wired in src/index.ts): GET /residents lists every registered repository, including cold registrations; warm residents have a green highlight and cold rows omit resident-only disk and image gauges — state dot (warm green unless its image report is pending; onboarding/refreshing/restoring or warm with a pending image report amber; degraded/down red; unreachable grey), reason, default ref, short sha, last refresh — each row a link to GET /residents/<owner>/<name>, which shows the interesting engine view for one resident: lifecycle (state/reason/last refresh error/last restore), pinned facts (ref, full sha linked to the GitHub commit, lockfile hash, provisioned/refreshed timestamps), the snapshot stamp (ref/sha/lockfile hash/backup ids), pending schedule counts, thread worktrees (each thread's ref + last-attached sha linked to its commit, deps mechanism, bound/last-attach times, evicted marker — the "which PRs are live on this resident" view, since only the default branch is pre-built), the command table with effects, and registry settings, plus the chat commands to manage it. Both pages sit behind the SAME Cloudflare Access gate as /runs (fail-closed in src/index.ts; they list every repo and its build commands, so never exposed without SSO) and are cross-linked with the runs dash. The registry is read from the admin GET /residents route on EVERY request with the admin bearer — never cached, and the bearer never reaches the page. Off-path: non-GET → 405; resident admin not configured → 503 naming execution.resident; resident Worker non-200 or unreachable → 502 carrying the upstream status/reason (never a 500 with a stack); unknown slug → 404 "not onboarded"; non-slug paths fall through. Since the Vue port the handler serves the shared web shell with the listing/record as the seed (ResidentsIndexSeed/ResidentDetailSeed; slug/live/tone helpers shared by handler and page in src/channels/residentsModel.ts), and the pages render in web/src/pages/{ResidentsIndexPage,ResidentDetailPage}.vue (mobile layouts included): hostile strings are inert by construction (the seed island escapes every angle bracket, the components render text — no v-html), only well-formed slugs become detail links, under the same WEB_HTML_HEADERS (strict CSP + X-Frame-Options: DENY + no-store) as /runs. Tab favicon: the index tab wears the dot favicon (live-view.md item 21) painted the fleet's worst tone — red if any resident is degraded/down, else amber if any is onboarding/refreshing/restoring or warm with a pending image report, else green only when every resident is warm and no image report is pending, else grey (no residents, or an unknown/unreachable one with nothing worse to show; a fleet is "all up" only when all of it is) — so a glance at the tab bar says whether the fleet needs attention. residentsFleetTone (residentsModel.ts) is the one worst-of rule; the shell renders the dot from the seed on the first paint (webShell.ts's pageFavicon) and the page repaints it from each listing its feed pushes. The detail page wears the neutral mark — one resident's state is the page's own dot, not a fleet claim. The index is live, and each resident folds open to what is on it (web/src/components/residents/ResidentRow.vue; the join and the reducer in residentsModel.ts; the feed in src/channels/residentsFeed.ts): the row keeps its glance — dot, slug, state, image report (current/pending/unknown), the facts — and gains the count of runs on the resident and a detail ↗ link; open, it is one row per tree on the resident, each reading left to right as the run · the tree it holds · the clock — the run side in the proportional face (words), the tree side a fixed-width monospace block (branch @ sha over deps · size) so rows align without column labels. A row per live run (the registry's rows whose RunMeta.repo is the slug, oldest first: agent chip, the request, who started it and the thread link under it, a running for stopwatch on the shared wall clock, open ↗ with the run's token), each joined by thread key — the run's RunMeta.threadKey is the binding's threadKey (AGENTS.md invariant 4) — to the worktree the resident holds for it (ref, sha linked to its commit, deps mechanism, the tree's own bytes from the item 55 sample), with when and who started it and the thread link; a run whose attach has not completed has no worktree bound yet; the live bindings no run is using are dimmed rows of the same shape (idle, last use; evicted ones are the detail page's audit rows, not trees); the disk line is the item 55 arithmetic in one sentence (free under any diskBudgetMb cap, room in more trees, sample age — the gauge itself is on the summary). The seed (ResidentsIndexSeed) carries the listing plus the registry's live repo runs under the viewer's runs:read predicate — the same predicate /runs lists through (liveView/viewer.ts), so both dashes show one viewer the same runs — and the server clock. The page then follows GET /residents?stream=1 (ResidentsFeedFrame): the registry's index feed narrowed to the viewer and to runs naming a repo (upsert/removed, the runs index's own frames — a finished upsert takes the run out of its fold), and a residents frame carrying the admin listing whole whenever a tree changes hands or the periodic listing read fires. Run-driven reads use the buses that already carry those moments: the run's dispatch.workspace.attach span end on its own stream (the tree is bound — or the attach failed) and the run's seal (the release at run end, item 17, is done by then: it runs before the reply returns and the seal follows the reply); the feed subscribes to each live repo run for exactly those two frames (one event of replay — the seed already showed anything bound earlier). A 30-second listing read also carries drain holds and image-report changes when no run event fires; the index shows the drain and held repos separately from the warm state. Reads are coalesced (one in flight, at most one queued), a failed read leaves the page on the listing it has, and a disconnect releases every subscription. Off-path: a run that dies without a seal is reclaimed by the resident's own sweep (items 16b/23), and the next listing read shows the tree gone; a resident's state change with no run behind it (a refresh cycle, provisioning) reaches the page on the next periodic listing read or an immediate reload. ?open=<owner/name> opens that resident's fold on first paint (the docs screenshot uses it).

42a. UID diagnostics stay in the API. Dynamic residents report uidAllocation:"dynamic", the historical poolUsersSpent count, no fixed total, and a bounded ownership preview with an explicit completeness flag. The index and detail pages omit UID counters, identity tooltips, UID-history exhaustion warnings and Linux worker IDs for dynamic, older fixed-pool and unavailable UID records. The existing index feed updates current lifecycle, run activity and disk usage. The detail disk view combines home directories into one total. Backend allocation, ownership and diagnostic fields remain available. 43. Exec survives a deploy — a runtime replacement is named, never mistaken for a dead sandbox (a wrangler deploy of switchboard-resident mid-run replaces the ResidentDO isolate and the Sandbox SDK rejects the running command's handle with Process handle refers to a previous runtime incarnation; unclassified, two such exec failures trip the runner's fail-fast and the run aborts as a fake "OOM"). Resident side (run() in worker.ts): the spawn and output-collection phases are separated. A runtime replacement during collect means the process started and its output is lost — never re-run. isRuntimeReplacement classifies one from the pinned @cloudflare/sandbox@0.13.0-next.751.1, typed checks first, wording last: - StaleProcessHandleError — the SDK's own verdict on a process handle; its control-plane wrapper also collapses any interruption it sees into this class. - RuntimeIdentityInactiveError — an exported class the SDK throws raw at ~10 process/exec sites before its adapter would translate it. - OperationInterruptedError with reason runtime_replaced, sandbox_lifetime_changed, or container_stopped. NOT transport_disposed / sandbox_destroyed / recovery_exhausted (real failures) and NOT unknown (no evidence — conservative). - RPCTransportError with kind peer_closed, connection_failed, upgrade_failed, or session_disposed — how a container stop/restart surfaces while output is being collected: the SDK's log-subscription reader deliberately does NOT translate transport loss into an interruption, so the raw transport error reaches run(). NOT invalid_frame / protocol_error (wire-format bugs) or unknown. A mere network blip lands here too, and that is fine: the outcome is the same safe one (re-attach once, never re-run, the model re-checks effects). - NOT the platform's superseded-isolate reset (isDurableObjectCodeUpdateReset): isRuntimeReplacement is deliberately immune to it, so the /exec path never swaps the incarnation on a DO reset whatever the catch's ordering, and the thread run() path intercepts a DO reset FIRST as a control-reset, never runtime-replaced (the control-reset carve-out below), since the container it runs over is unchanged. The DO reset's ONE home is isControlReset; the RESTORE path is the one site that folds it into the replacement disposition, named explicitly beside the union at its own site (isRuntimeReplacement(err) || isControlReset(err)), since a reset under a live restore leaves nothing to land and re-restores onto the container that comes back. - fallback: the SDK's replacement wordings (RUNTIME_REPLACEMENT_WORDING, shared with the refresh classifier in src/execution/residentRefresh.ts — the six SDK messages, Process supervisor is closed (the spawn refusal a stopped container answers until it restarts), the raw workerd binding refusal The container is not running, consider calling start(), and The container just exited — the SDK's answer when the container dies under a live exec, which the bot side had read as a roll since item 65 while the resident's classifier filed it as the step's own failure: five stop-container faults during a wake on a throwaway resident each recorded degraded(refresh-failed: The container just exited), one strike apiece toward item 67's ladder, until the wording joined this list), matched on the error and its cause chain (bounded to 8 like the SDK's own selfAndCauses). That last wording is the container ROLL case, distinct from the container_stopped interruption above: a deploy that rolls the container out (not just swaps the DO isolate) drops it onto an empty ephemeral disk, and a /exec racing that window gets the spawn refused outright — the SDK's own auto-start (runWaking → startAndWaitForPorts) re-throws it untyped when the roll outlasts its port-ready bound, so there is no typed class and the wording is the only signal. It can never match a genuine container crash (container exited with unexpected exit code) or the readiness probe (the container is not listening), which stay ordinary failures. During spawn, the resident retries ONCE on a fresh process sandbox only when the SDK marks the interruption retryable (its own vouch that the process never started, e.g. container still starting); the SDK sets retryable:false for runtime_replaced, so a deploy-time spawn failure is also never re-run. Both unsafe cases answer the NAMED error {error:"runtime-replaced: …", status:409, reason:"runtime-replaced"} on /exec (in-body over the 200 stream, with reason now always forwarded), /read, and /write. Client side (ResidentExecutor.opWithReattach): on /exec, reason:"runtime-replaced" is the typed ExecSandboxRestartedError at once — the executor's one word for a container gone under the thread (saysContainerGone; the same word item 65's wake ends in and item 27's worktree-missing takes), carrying the resident's own words and that the command was not run again (containerGoneMessage), waitedMs 0 — with no re-attach first and never a retry: the command may have started, and the pi harness, the one loop, ends the run on that word for a restart from its request (harness-pi.md item 16), so a recovery before the verdict only delays it (a re-attach blocks through the restore, ~25–60 s) or hides it (the attach refused image-stale or not-serviceable while the replacement reconciles — the 1.230.0 miss on run 5b7e3b50). Not an ExecInfraError, so one deploy never reads as a dead sandbox. On the idempotent routes (/read, /write) the client re-attaches once (proving the new isolate serves the thread) and re-issues the call; a second runtime-replaced there with no successful op between IS an ExecInfraError (a deploy storm or a flapping resident), and any successful op resets the streak. What a container roll costs the run: the run in flight ends interrupted and its request runs again under the same run id (run-history item 54), and its attach drives the resident's recovery — the container is auto-started and the tree rebuilt/restored from R2 (~1–2 min observed) — so the restarted run lands on the warm, restored tree. Before the harness (record 0032) the /exec that hit the roll was handed back to the native loop as text after a blocking re-attach, so the model's next command landed on the restored tree; the native loop is gone and no caller continues on the executor after the word. The control-reset carve-out — a DO reset over a live container is NOT runtime-replaced. A wrangler deploy of the resident Worker CODE (no image change) supersedes this DO's isolate mid-command while the container and every process the run holds in it keep running; folded into isRuntimeReplacement, that made /exec answer runtime-replaced for a reset over a still-running container, so the pi harness judged the container replaced and relaunched a second pi in the SAME container, orphaning the live pi at the recorded pid on every release. The thread run() path intercepts a DO reset FIRST, in both the spawn and collect phases, as a typed ControlResetError — ahead of isRuntimeReplacement, ahead of the memos' clearing and ahead of any incarnation swap, since the container and its memos are exactly as they were — that answers its own {error:"control-reset: …", status:409, reason:"control-reset"} on /exec, /read and /write: the container and its processes are unchanged and the command's outcome is unknown. Client side (ResidentExecutor.opWithReattach): on /exec, reason:"control-reset" is the typed ExecControlResetError at once (saysControlReset) — NOT an ExecSandboxRestartedError (not a replacement, so never the replaced verdict) and NOT an ExecInfraError (not a dead sandbox); the harness seam re-sends an idempotent op once and resolves a write by pi's echo (harness-pi.md item 16), never orphaning a live pi. On /read and /write — the routes CONTROL_RESET_REISSUE_ROUTES names explicitly: /read idempotent by shape, /write a full-content put (the path and the whole content, never an offset, an append, a mode change or a hook), so the same bytes landing twice are the file once; a route that ever writes by delta must leave the set — the client re-attaches once (the DO is fresh after its reset) and re-issues, then surfaces the unknown outcome; any other route's control reset is the unknown outcome at once, never a blind re-run — one rule, controlResetUnderThread, reads the set for every answer, so /exec (outside it) is typed there and a route added to the call sites is judged by the set, never by the order of the checks that follow. runtime-replaced stays reserved for a container runtime actually replaced. - The /exec answer's gate (execThreadImpl → replacedExecAnswer; harness.md item 6's third failure shape): isRuntimeReplacement above is the SDK-replacement half the resident's own steps need — a step interrupted by a runtime replacement is retried onto the container that comes back, and the restore path adds the DO reset to it explicitly at its own site — but the word on /exec is what the harness relaunches a run on, so /exec says runtime-replaced only when the SDK vouched the runtime MOVED (sdkVouchesRuntimeMoved: the typed classes, the interruption reasons and the six wordings of RUNTIME_MOVED_WORDING — never the platform's isolate reset, which is this DO resetting over a container it did not touch) or the resident itself knows the container it held is gone (knowsContainerGone: a restore under way for this resident, restoring — its own knowledge, reliable — and nothing else). Not the Container base's exit state, which was tried and separates nothing, read from @cloudflare/containers 0.3.7 container.js: the rollout's Runtime signalled the container to exit due to a new version rollout: 0 does not match RUNTIME_SIGNALLED_ERROR ('runtime signalled the container to exit:', line 10, a lowercase includes at 53-56), so the monitor's rejection writes stopped (1445-1462) — or nothing, because the harness's next command wakes the container and startAndWaitForPorts replaces this.monitor (1377-1380) before the old callback, guarded by this.monitor !== monitor (1442-1444), ever runs — while an idle stop (the resident's SLEEP_AFTER) resolves the monitor and writes stopped_with_code(0) (1434-1440; stop() itself writes no state, 712-717). Not onStop, which the base calls only from syncPendingStoppedEvents at the next start (1594-1603), for an idle sleep as much as for a roll, so no stop hook is overridden and no stop record kept. Under-saying the word is safe — the harness's own probe (identity, alive) stays the arbiter — and over-saying it is what orphans a process. The resident's own evidence for a gone container (a per-incarnation marker written into the container at first attach and compared on the next spawn failure) is the follow-up this item names, not yet built. A failure that only says the container is DOWN (STOPPED_CONTAINER_WORDING: The container is not running, consider calling start(), Process supervisor is closed, The container just exited) or that the transport was lost under the call (the RPCTransportError kinds) answers what the SDK said — the SDK's own words, no reason — because an asleep or starting container and a network blip answer the same words, and a word given then would relaunch a run into a container that was never replaced (the orphaned process, the relaunch spent). The harness's one more command decides instead, waiting through the restore window. The platform's running flag is never the evidence, for the same reason; and a status read that fails inside the gate is no knowledge — the SDK's words, never an unhandled throw where the 409 was due. The decision is made once, where run() classifies the failure (replacementKnown = sdkVouchesRuntimeMoved(err) || knowsContainerGone()), and carried on the error (RuntimeReplacedError.known): it gates the word on /exec. A classified replacement's two consequences are split at that choke point, because they carry different safety: the per-incarnation memos (clearIncarnationMemos: hydratedVerdictAt, gitSetupDone, stageDirsReady, containerIdMemo, the deps flags — the fact that was stale) clear on every classified replacement, whatever the word will say — cheap to re-derive, and the memo block's invariant is that this one point clears them, so a real roll (where known is false while the harness's next command wakes the container and the wake rewrites the base's state before any exit is recorded) re-probes the disk on its next command; the incarnation (swapIncarnation, the lease fencing token: one isolate paired with one container runtime) is minted only when the replacement is known, because isRuntimeReplacement is the union — the RPCTransportError loss kinds and the down wordings a network blip or a merely-asleep container also produce — and a mint on a blip would make takeMutex read a live holder's row as holder-incarnation-gone and hand the mirror mutex to a concurrent /attach over a running fetch (worktree add over a fetching mirror). Two Durable Object-reset behaviours stand beside the split: the run path's reset (isControlReset → ControlResetError, checked before any of this) neither clears nor mints, so the re-send lands on the same pi; the restore path's fold (isRuntimeReplacement(err) || isControlReset(err)) still mints, because a reset under a restore is a new isolate. controlReset.test.ts scans the entry for the order and both behaviours. /read and /write keep the union's word unconditionally, on purpose: they reach the same run() and can meet the same down wordings, but there the word drives the client's re-attach-and-retry — the attach is what wakes a container that is asleep or starting — never a verdict, nothing relaunches on it, and the harness never reads or writes through them (its seam runs every operation as an /exec script), so gating them would trade a spare re-attach for a read that fails outright while the container starts. The exec route's every RuntimeReplacedError — the preflight's own worktree probe and the command alike — goes through the one gate; execReplacedWord.test.ts scans the entry for it. A DO reset never reaches this gate as runtime-replaced, and never a replacement's consequences: run() intercepts it first as a control-reset (the carve-out above), before replacementKnown is read, before the memos are cleared (clearIncarnationMemos(), every classified replacement's) and before the incarnation is minted (swapIncarnation(), a container known gone), so a reset over an unchanged container never clears its still-valid memos or leases (the exact orphan this item's fix prevents). Neither the gate's vouching set (sdkVouchesRuntimeMoved) nor isRuntimeReplacement lists the isolate reset — a vouch or a replacement judgement on it would say the opposite of that word — so the DO-reset signal lives only in isControlReset, with the restore path naming it explicitly beside the union at its own site; the resident source scan holds all these facts. 44. Deploy preflight — a Worker deploy never lands on an in-flight run (a deploy swaps every ResidentDO isolate and invalidates the Sandbox SDK process handles — Process handle refers to a previous runtime incarnation — which reconcileImage's container-restart deferral cannot see). In-flight activity is now observable: GET /status answers {state, reason, inFlight} and GET /residents carries live.inFlight per resident plus an aggregate inFlight — the sum over residents whose live view answered with a count, null (with inFlightUnknown = how many) as soon as any resident's view failed or carried no count, so a reader can never mistake "unknown" for 0 — (same inFlightCount() predicate as reconcileImage/isIdle: thread exec/read/write, disposable /op users, attaches past their image check, refresh cycles past their gates). npm run deploy in deploy/cloudflare-resident/ first runs node preflight.mjs with RESIDENT_DRAIN_TOKEN. The ordinary preflight reads /healthz within its preparation budget. A verified features.deployFenceAcquisition: v1 selects POST /prepare-deploy-fence with one original acquisition ID. A verified old receiver without that capability selects legacy POST /deploy-fence. Selected staging attempts, explicit force and qualified bootstrap keep their existing legacy protocol and cancellation limits. Malformed, unavailable or unknown capability evidence refuses before mutation; a failed selected v1 preparation never falls back to legacy. Both protocols close registered reattach before reading exact run ownership and refuse executing or unknown owners. The v1 handler withdraws its exact preparation before sealing when its observed admission count is positive, even if that last admission ends before the seal could verify zero. Terminal registrations remain protected without counting as executing. Fail closed: no bearer, unreachable/non-2xx/non-JSON Worker, a resident whose live view errored, a live view without inFlight (a Worker predating this item), or an impossible count (negative / non-integer — a counter bug) all refuse with the operator hint. RESIDENT_DEPLOY_FORCE=1 (or --force on a direct node preflight.mjs) deploys anyway with a warning that still names what it is killing. The decision (decide()) is pure and unit-tested; only main() touches the network. A provisioning resident refuses too; a refresh or restore state without admitted work warns: a resident whose live state is onboarding has provisioning executing, which an isolate swap kills exactly like a thread run and which has no checkpoint to resume from — provision-failed, down, a rebuild to recover — so it refuses by name. refreshing and restoring have the engine executing too, but the swap only INTERRUPTS them: a refresh step is retried by its instance and resumes from its disk checkpoints (below, and item 48), a restore is retried by the next hydrate, which first unmounts and removes whatever the interrupted one left (item 61); they used to refuse like onboarding, and one release deploy spent its whole budget behind a resident stuck in restoring — a state the swap would have cleared — and went red. They are allowed with a WARNING that names the resident and the state when no durable operation admission is active (preflight ok: … — WARNING: mid-cycle: repo:… (restoring) — …), never REFUSED; runs in flight still refuse whatever the state. (Before the classifier the interruption itself was costly: degraded(build-failed: exit 143: Session terminated, killing shell…), self-healed only at the next cycle — a full cadence of cold runs.) A deploy that lands mid-cycle anyway (forced, or a path the preflight cannot see) recovers within a minute: the pure classifyRefreshFailure (src/execution/residentRefresh.ts) recognises a step killed from outside — exit 143, Session terminated, SIGTERM; never a step the cycle itself timed out — as refresh-interrupted: <step> … instead of <step>-failed; the instance step throws it to the engine, whose retry (thirty seconds on, six attempts, item 7) re-enters the same idempotent method, so the row is never degraded by it and the park streak never sees it. A chronic false positive — a repo whose own build output carries the signature every cycle — costs at most the step's six attempts per instance, not a hot loop: the engine's retry cap bounds it. An image-stale container stop (reconcileImage) and a disk-full recycle at the entry gate are retried the same way (CycleRestartError). The gate is an allow-list of known states — warm/degraded/down with inFlight: 0 allow; refreshing/restoring allow with the warning above; onboarding refuses by name; any state the script does not recognize refuses as unknown (the script is plain JS outside the shared ResidentLifecycleState type, so vocabulary drift fails closed). Runs and cycles are reported independently — a resident that is both busy and mid-cycle names both. Since inFlight counts the refresh cycle itself (inFlightCount = runs + refreshesInFlight), a resident that is merely refreshing reads ≥ 1 and would refuse as busy whatever the state rule says; so the live view also carries runsInFlight — the runs alone (thread exec/read/write, disposable /op users, attaches mid-flight; runsInFlightCount) — and the fenced preflight separately judges executingRuns; a Worker without /deploy-fence refuses. The first live sample of the revision (a refresh window on the switchboard resident, the real preflight.mjs every 0.2 s) showed exactly that gap: the WARNING suffix named refreshing on every sample, but each was still REFUSED … (1 in flight) for the cycle's own count. A harness run counts while its process lives in the container, not only while an operator call is in flight: pi runs inside the container between the bot's /exec calls, so the in-memory op counters read 0 mid-run and a deploy once rolled the resident under a live pi sleep. The resident now holds a durable run registration per attached run — written by every successful /attach, cleared by the binding's eviction (/detach, the clean-idle sweep, disk pressure: every release path ends in evictBinding), and no longer counted after the run deadline plus release grace — and the retention counts (GET /residents inFlight/runsInFlight, GET /status, /debug info) add the registrations whose thread has no op currently in flight, so a registration is never counted twice; the fenced deploy read distinguishes terminal retention from execution. Durable rows on purpose: the harness process survives an isolate swap in the container, so a fresh isolate still counts it. isIdle, reconcileImage, and the disk-full recycle all read the bounded registrations: a live registered run protects its container between operator calls, including after the one-hour recent-use floor. An owned registration remains protected until exact preservation permits release; a positively legacy registration uses its deadline plus grace or clean-idle boundary. the sweep also clears it when its binding is evicted. reconcileImage DEFERS its restart while a registration is live (item 16b): a restart is exactly what kills the run the registration protects — one attach's reconcile once stopped the shared container under another thread's registered run. The attach sends a provisional deadline covering the remaining setup window plus the full harness lease. When the lease starts, /run-deadline sets the actual remaining deadline; a resumed run does the same from its saved remainder, and a re-attach sends that remainder instead of another full lease. An in-process relaunch also passes the same run ID through its re-attach. The live-run ledger mints a monotonic attachment fence for its current run ID, thread, and process generation before each resident attach. Attach serializes by thread and records each accepted fence in a per-thread high-water row that survives eviction and binding purge. It accepts a newer fence across run IDs and generations, even when boots share a timestamp; an older delayed attach cannot replace or reopen the newer workspace. Lease updates and /detach require the exact run ID, generation, and fence. An eviction retains that ownership evidence so the ending run can still record its pushed branches after the tree was swept; a stale run cannot record branches or remove its successor’s tree. Legacy callers may release only legacy registrations. Drain, memory, and stale-image gates exempt only a live registration owned by this run and fence or one with a newer ledger fence; a missing, malformed, foreign, or expired row grants no gate exception. 45. Event-triggered reclamation — a finished ref becomes a cleanup candidate within one refresh cycle (gc.ts + reclaimFinishedRefs in worker.ts): idle signals (items 16b/23) collapse a worktree an hour to seven days after its last attach; this reclaims it when the WORK finishes. It is a POLL folded into the existing refresh cycle, not a webhook — decided from what the App is configured for: GET /app answers events: [] and GET /app/hook/config 404s (the App was created with webhooks off, per the README recipe), and turning one on would mean a public unauthenticated ingress with HMAC verification, a webhook secret in a second credential domain, and a UI-side App change, all to deliver a signal the cycle's git fetch --prune already carries for free. Each cycle, after the prune and the warm flip, every LIVE binding whose ref is not the default branch is classified in a CONCURRENT pre-pass over the distinct refs: branch existence comes from ONE git for-each-ref --format='%(refname:short)' refs/heads/ listing of the mirror (parseRefListing in gc.ts — one spawn for the whole pass, not a rev-parse per ref; an unreadable listing leaves membership unknown so the PR lookup decides, never "every branch gone") — a ref absent from it → gone; otherwise ONE GET /repos/<slug>/pulls?state=all&head=<owner>:<ref> per distinct ref, the lookups running concurrently under Promise.all (the cycle's minted repo-scoped token, or anonymous when the App is unconfigured; 10 s budget each) → an open PR wins (open), else merged beats closed (merged/closed), none → no-pr; any transport/non-200/unparsable answer → unknown. The eviction loop itself stays serial — its re-read guards depend on ordering. reclaimDecision evicts gone/merged/closed bindings with no op in flight AND no live run registration on the thread (item 44's registration, written by the attach and bounded by its deadline plus grace or cleared by release: a harness run's process lives in the container between the bot's calls, so the op counters read 0 while its model thinks — a live run whose branch the mirror did not yet hold, a ship unit's unpushed branch, once lost its tree to this pass mid-run; run-held names the keep, and the guard before removal re-reads the registration and its liveness as it re-reads the binding) — and asks nothing about the tree: under item 17 its dirt is never a reason to keep it (a merged PR's tree can still hold unpushed local commits, and a thread that itself ran git fetch --prune after the branch was deleted counts its pushed commits as unpushed against origin/main; whatever the tree holds, no run will see it again — the next attach wipes it). What the tree held is measured once as the thread user right before the removal (the one-spawn worktreeCleanliness, runtime up; git rev-list HEAD --not --remotes counts against the worktree's OWN remote-tracking refs, local snapshots advanced by its own git push that GitHub's squash merge never touches, so a pushed-then-squash-merged branch reads 0 unpushed) and recorded by the eviction — evictedLeftBehind / evictedUnmeasured beside evictedWhy and one log line, item 17's record — so nothing is discarded silently; a slept container has no tree left, so nothing is measured and the pool user is released outright. The default branch is never reclaimed; open/no-pr/unknown keep with the reason named. The eviction is the sweep's evictBinding with the same re-read guards (binding re-read + op counter right before removal; lastAttachAt unchanged), and it now stamps the binding with evictedWhy (merged #N, gone, clean-idle, ttl, detach) — surfaced in /residents live.threads, /debug threads, and the dash's evicted marker — so every eviction is auditable. A pass failure is a log line, never a lifecycle flip. When the runtime is DOWN the mirror is not consulted (a run would wake the container and read an empty disk as "every branch gone"); only the PR lookup can speak. POST /debug {"op":"reclaim-now"} (admin) runs a fetch --prune (bounded by ATTACH_MUTEX_WAIT_MS) then the exact pass, answering {reclaimed:[{threadKey, ref, why}], kept:[{threadKey, ref, why}], fetch}. Under the Workflow lifecycle (item 7) the same pass runs in the cycle's completion, at the end of the instance's snapshot step, with a token minted there. 46. Resident-level LRU eviction — opt-in, default off (pickEvictionCandidate in gc.ts): POST /onboard {…, evictColdest:true} (admin route; a non-boolean value → 400) changes the over-cap answer from 429 to "make room": when the registry refuses with 429, the Worker collects every resident's registry record + live view (collectResidentViews), picks the coldest ELIGIBLE one, swaps the registry rows in ONE input-gated section (ResidentRegistryDO.replace: the victim's slot is freed and the newcomer inserted atomically — a concurrent onboard can never take the freed slot and leave a resident destroyed for nothing; 409 if the victim was offboarded meanwhile or the newcomer already exists), and only THEN tears the victim down through the item-11 teardown minus the registry step (teardownResident, shared with POST /offboard's offboardResident). Eligible = live state warm (idle-parked is still warm; degraded/down/mid-flight/unknown are rejected by name), inFlight === 0 (null — a live view that failed — is rejected, never read as idle), ZERO live worktrees (a live tree may hold uncommitted work; evicted bindings do not count), and last activity — the newest lastAttachAt across all bindings, or provisionedAt/onboardedAt when never attached — older than LRU_FLOOR_S (1 h, = IDLE_AFTER_S) so a repo used minutes ago never goes cold to make room. Coldest = oldest last activity, ties on resource name. The 202 carries evicted: {resource, lastActivityAt, registryRemoved, schedulesCancelled, containerStopped, storageCleared, backupObjectsDeleted, r2ObjectsDeleted, errors}; no eligible candidate → the ordinary 429 with rejected: [{resource, why}] itemizing every resident (e.g. active 12m ago (floor 60m), 2 live worktree(s), state down), so the admin can offboard by hand with the facts in front of them; a refused swap (409/429 from replace) destroys nothing and names the candidate as wouldHaveEvicted. Without the flag nothing changes: the registry's 429 now also names the opt-in. Chat: repo onboard <owner/name> [...] --evict-coldest (behind the same fail-closed canManageRepos gate — the flag provisions AND destroys billable compute) sends evictColdest:true; the reply adds a ♻️ Made room: evicted … line naming the resident and its last use, and an over-cap refusal relays the per-resident reasons as bullets. --evict-coldest on repo reconfigure is refused by name. 47. Lower-contention refresh — the rebuild stages outside the mirror mutex and swaps under it (stageCheckoutScript / swapCheckoutScript / CHECKOUT_FETCH_COMMAND in src/execution/residentRefresh.ts, applied by runBuild): the cycle's rebuild used to run git clean -fdx + build UNDER the mirror mutex — it had to: materializeThreadDeps hardlink-copies CHECKOUT_DIR under the same lock, and a checkout mutating under that cp -al is a torn cache (a false ❌ from repo test) — so a heavy rebuild held the lock for minutes and pushed concurrent attaches past ATTACH_MUTEX_WAIT_MS (60 s) into mirror-busy 503s and cold seeded-sandbox fallbacks while the card read refreshing. runBuild now takes the lock twice, briefly, and never for the build. Before the stage lease, off the lock: any stale build-user writer under the staging path is killed and a failed rebuild's leftover staging tree is dropped — the tree is private to the rebuild and can hold hundreds of thousands of inodes, so its rm never extends an attach's wait. Stage (lease checkout-stage, the section's own sum — two git-network budgets, one per command it covers, item 22's backstop): a swap the container died inside is self-healed (a missing checkout restored from the retired tree), the retired leftover dropped, and the warm checkout is hardlink-copied to /workspace/checkout.staging (cp -al by root, ownership preserved — the one checkout read the lock still covers, exactly materializeThreadDeps' consistency guarantee), then git fetch runs from the mirror into the staging tree (CHECKOUT_FETCH_COMMAND — the mirror read the lock covers: an attach's fetch --prune could delete a ref under its negotiation). Off the lock, in the staging tree: the hard-reset and -x clean (checkoutUpdateCommand, fetch-free now — review 1b's fresh-inode invariant holds exactly as before, the clean removing the staged copy's hardlinked build output so the build allocates fresh inodes), on an install the deps-view re-link to the new store entry (the install itself already runs off-lock in the deps store, item 59, and the unconditional reinstall on a lockfile move is unchanged), and the repository's build under REFRESH_BUILD_TIMEOUT_MS. Swap (lease checkout-swap): two renames — checkout → retired, staging → checkout, the retired tree put back when the second rename fails so the checkout path is never left empty — with the disk markers (deps-key, the installing marker's clear, built) written under the same lease, so the markers move with the tree they vouch for; the retired tree is removed AFTER the lock is released (a large rm never extends an attach's wait), and both side trees are wiped with the workspace on every rm -rf of the disk. The snapshot stamp, READY_MARKER, lockfileKey and the restore path are untouched: CHECKOUT_DIR still holds exactly what the markers and facts say, and now changes only by rename under the mutex. The Workflow instance (item 7) is unchanged: its build step is still one runBuild, whose step budget covers the two short leases plus the off-lock rebuild. 48. Refresh install gate + disk checkpoints — refreshes stay under the attach wait (the complement of item 24's serviceable-state attach): the rebuild on a sha move is decided by the pure planRefresh (src/execution/residentRefresh.ts, imported by the Worker like residentDetach) from the committed-lockfile key at the new sha (item 6's pure-function key, computed from the mirror BEFORE any checkout work) and three disk facts read by readRefreshDisk: the checkout's real HEAD (as worker1) and two root-only markers under /workspace/.resident/ — deps-key (the lockfile key whose install fully completed into node_modules) and built (the sha whose build fully completed). Rules: (a) lockfile key unchanged → rebuild {install:false, clean:"keep-deps"}: git clean -fdx -e node_modules (a git exclude that survives -x, matching workspace packages' node_modules at any depth) removes every other gitignored path so the build still allocates fresh inodes, and the build-written caches INSIDE node_modules (.cache, .vite — NODE_MODULES_CACHE_DIRS, pruned at any depth) are swept too, since those open+truncate in place and the build runs as the inode owner (review 1b holds — nothing installs, and nothing the build writes lands on a kept inode; residual: a build that writes elsewhere inside node_modules is outside the sweep), then build only — seconds, not the 1–3 min unconditional reinstall that pushed refreshes past the bot's 60 s ATTACH_MUTEX_WAIT_MS; (b) key changed, or no deps-key marker (a container from before the markers, or a full clean that was interrupted) → the unchanged full path: git clean -fdx + install + build; (c) checkpoint hit — HEAD, built and deps-key all already match the target → reuse: skip checkout/install/build and go straight to the snapshot (facts + stamp must still move together, so the snapshot is never skipped) — this is what a DO reset mid-cycle (a resident-Worker deploy; the container disk survives it) leaves behind, and it previously cost a full rebuild; (d) HEAD ≠ sha or built ≠ sha with a matching key → keep-deps rebuild (never reuse on a marker alone). Marker discipline: the markers for the steps being redone are removed BEFORE the step starts (built always; deps-key only on a full install), written right after the step completes, written from the stamp on restore (ensureHydrated) and at the end of provisioning, and wiped with the workspace on every rm -rf of the disk. facts.sha is NOT checkpointed early: DO facts and the snapshot stamp are verified against each other on hydrate (item 5), so advancing facts before the snapshot would turn a sleep into snapshot-stamp-mismatch/stale-serving. Each planned cycle logs refresh: <old> → <new>: <action> (<why>) and refresh: <sha> <action> done in <ms>ms for wrangler tail receipts. A cycle a deploy kills mid-step leaves these checkpoints behind and its step is retried by the engine thirty seconds on as refresh-interrupted (item 44), so the retry is a reuse or a keep-deps rebuild a minute later, not a full rebuild ten minutes later. On the Workflow lifecycle (item 7) the instance's fetch step computes the same plan and clears the same markers once per cycle, so a build step the engine retries finds the checkout's markers naming its own work and issues no second build. 49. Test overrides — the over-cap path stays provable at any production cap (POST /debug {"op":"set-test-overrides", cap?, floorS?}, admin scope; gc.ts parseTestOverrides / effectiveLimits, registry limits() / setTestOverrides()): the 429 / rejected[] / eviction behaviors of item 46 are reachable only when the registry is FULL, and RESIDENT_CAP is sized for the team's real fleet — proving item 46 by changing the compiled cap costs a deploy each way plus the floor hour, and would be impossible once the real fleet fills the cap (lowering the compiled cap below the fleet size refuses the team's own onboards). This is the resident's fault-injection pattern (item 13: backdate-thread, force-down, force-onboarding, …) applied to the two limits. Mechanics: the op stores {cap?, floorS?, setAt, build} in the registry DO under testOverrides (outside the resident: prefix, so it never counts as a slot); limits() answers min(override, constant) for each field and is read INSIDE the same input-gated section as the count/insert (onboard, replace), so an override flip can never interleave with an onboard; handleOnboard's LRU pick uses the effective floorS. Guard rails by construction: (a) admin only — not in READ_DEBUG_OPS, so the read token gets 403; (b) an override can only LOWER: cap must be an integer in [1, RESIDENT_CAP], floorS in [0, LRU_FLOOR_S], else 400 naming the range — it can never become a back door past wrangler's max_instances, and a stored value above the constant (a later deploy lowered it) is clamped, never honored; (c) deploy-scoped — the record carries BUILD_MARKER (the /healthz u, bumped every deploy-worthy change) and a different build ignores it (ignored:"stale-build <marker>"), so a forgotten test cap cannot outlive the session that set it; (d) visible — GET /residents reports the ENFORCED cap plus capDefault and, when active, testOverrides {cap?, floorS, floorDefaultS, setAt, build}; repo list appends ⚠️ test overrides active (set <when>): cap N (default M), LRU floor Ns (default Ms) — clear with …; the watchdog summary's cap is the effective one too. An empty body (neither field) clears. The op answers {cap, capDefault, floorS, floorDefaultS, override}. 50. Read-only attach — a read-only run cannot push or traverse the local mirror (the complement of the reviewed-head guard, agent-review.md item 8): a review run that is read-only only "by convention" could fetch another PR's branch and land its verdict on the wrong PR; the guard makes the post-step refuse such a review, the door also limits direct fetches to the bound repository. POST /attach accepts readonly: true (validated boolean; absent = false, so an older bot's body means what it always did — parseReadonly in src/execution/residentReadonly.ts). The bot sends it from the run's effective profile (ResidentExecutor opts readonly, set by makeExecutor iff the profile's identity is read — the review agent; execution.md item 5), never from the prompt and never from the toolset name. On a read-only attach the pure planReadonlyAttach (imported by the Worker like residentDetach) yields: no App token or writable credential for the tree (the read-only run bearer can fetch through the door but cannot push; credentials: "none" — a new honest member of the union, distinct from unavailable = wanted one and could not get it — while the mirror's own recovery fetch for a ref pushed since the last refresh cycle still mints a fetch-only token as root, outside the tree: a read-only attach loses no availability, only the tree's credentials), origin left at the local mirror path (git remote set-url origin /workspace/mirror instead of the GitHub URL) — thread users are denied traversal into the mirror (root:worker1 750, item 19), so git fetch origin/git push origin fail legibly; a direct fetch through the run-bearer door remains possible for the bound repository, while the clone-time remote-tracking refs (origin/<default> and every branch the mirror had) still serve git diff origin/<base>...HEAD, and the tree is scrubbed on every read-only attach, reused tree included (rm -f .git/github-credentials + git config --unset-all credential.helper), because the file was written per attach before this rule. The binding records readonly (absent on pre-field bindings = writable) and AttachOk.readonly echoes it. Mode switch recreates: a live tree built for the other mode (read-only attach on a writable tree, or a writable attach on a read-only one) is wiped before the ordinary dirty/stale checks — a credential-less mirror-origin tree must never serve a writable run, and a GitHub-origin tree must never serve a read-only one; recreating is the simple safe option (a tree carries only scratch state between attaches, and one thread is normally one agent for its whole life). An evicted prior's recorded mode never forces a switch (the tree is gone anyway). Deploy order: resident first (an older resident ignores the unknown field — handleAttach reads named fields only — so a new bot against an old resident degrades to today's writable attach, loudly visible as credentials: "ok" on a review attach), then the bot.

  1. Attach fetches a STALE ref, not only a missing one — the caller names the commit it expects: the mirror is fetched on the refresh cycle (item 7) and, on attach, when the bound ref is missing; a ref present at an older tip (pushed to since the last cycle) would otherwise be cloned as-is, and a review of it is refused by the reviewed-head guard (agent-review.md item 8) — a head the bot had already resolved. POST /attach accepts sha (optional; a full 40-hex lowercase commit — a full sha names exactly one commit, no prefix ambiguity — validated before it can become a git argument: parseWantSha in src/execution/residentHead.ts; absent = the previous behavior, so an older bot's body means what it always did). The bot sends RepoContext.headSha whenever repo resolution found a PR head (ExecutorContext.headSha → ResidentExecutor opts sha), on both the review and coding paths — except when a message-named PR did not bind the ref: for a non-review run, a cited pull request that contributed no ref hint (prFromMessage without refFromPr — merged or closed, or a phrase's ref standing beside it) has its frozen head dropped by the dispatcher before the attach (issue 1860), because no fetch can bring a live branch's tip to a dead PR's frozen head — the sha would only earn the stale-tip refusal below and send the run cold at the dead commit; a review run keeps it (the pull request is its target, the sha the reviewed head's pin). The sha applies to the ref it was resolved for: the pure wantShaForBinding drops it when a refHint was named that is not the thread's sticky bound ref (such a thread can never be at that commit, so it must not pay a fetch on every attach); no refHint means the caller intends the bound ref. Under the mirror mutex the pure mirrorFetchReason({refExists, mirrorSha, wantSha, returnable}) decides, naming why (FetchReason): ref missing → missing-ref, fetch (unchanged); sha named and the mirror's tip of the ref is not that commit → stale-tip, fetch, then re-read the tip; the bound ref is one the binding could return from — returnable, the Worker's one reading of canReturnToDefault before the mutex (item 16's second movement: a rebind's branch or one of the thread's own pushed branches) — → returnable-ref, fetch, so a branch deleted at the origin since the last cycle is found gone at THIS attach and the return decided now; otherwise null, no fetch — a default-bound or person-named ref the mirror holds attaches as the mirror stands, the refresh cycle its freshness. The reason decides what a failed fetch means: missing-ref and stale-tip fail the attach as they always did; returnable-ref only verifies a ref the mirror holds, so its failure is one log line and the attach goes on with the mirror's ref. A read-only attach still gets the fetch-only token for this recovery fetch (item 50's rule covers the stale case as well as the missing one). The attach then clones/reuses at the mirror's tip as before — the stale-tree rule in ensureThreadWorktree (HEAD neither the tip nor a descendant → recreate) wipes the old tree, so a sticky thread binding never pins the previous run's commit. If the tip is STILL not the expected commit after the fetch, the attach is refused (stale-tip, 409 with the resident's state): a push racing this attach or a force-push leaves the tip ahead of the commit; a fetch that failed or was skipped (the mirror busy, item 16's wake-fetch skipped) leaves it behind; either way the resident cannot serve the commit the caller asked for, and it says so instead of running the tip — a run executes at the sha it asked for, or not on this resident (the caller's named fallback runs cold at the requested commit). The tip is read once, under the mutex, and is both what the target is judged on and what the worktree is provisioned at; an unreadable tip with a named commit is refused too, never assumed fresh. Before this rule the attach proceeded on the fetched tip and reported it in sha, leaving the reviewed-head guard to catch a review of it — a coding run had no guard and ran at a commit nobody asked for. The sha belongs to the attach that BINDS the run and to a moveTo (agent-review item 12): the bot's ResidentExecutor sends it on those and on no other — a recovery re-attach (a worktree evicted, a runtime replaced) names no commit and re-attaches the branch as the run left it, since the run's own pushes may have moved the tip past the sha it started on (shaPending, cleared by a successful attach, set again by a move). A ref that is gone after the fetch is not yet a refusal: the pure attachTarget({refExists, wantSha, commitInMirror, tipSha}) decides — the ref's tip whenever the ref exists at the named commit (or none was named); when it does not but the caller named a commit the mirror holds (git cat-file -e <sha>^{commit}; a --mirror clone carries refs/pull/*, so a merged PR's head outlives its deleted branch), the thread tree is cloned --no-checkout and checked out at that commit detached (worktree-detach), reported in sha as any attach; only a ref that is gone with no expected commit, or one the mirror does not hold either, is unknown-ref (the message now says which). Before this a review of a merged PR — its branch deleted on merge — fell back to a cold sandbox that cloned the same commit itself, after the resident had already fetched. Deploy order: resident first (an older resident ignores the unknown field), then the bot.

  2. The onboard command table comes from the repo root, and provisioning reports its outcome into the thread (a bare onboard that assumes npm leaves a pnpm workspace down on npm install with EOVERRIDE and a Terraform repo with no package.json down with ENOENT — for hours, when the acknowledgement says only "watch repo list", which nobody does). Two changes. (a) Detection: repo onboard reads the repo root over the GitHub REST API with the App's read-scoped installation token (githubRepoInspector in src/execution/githubRepoInspect.ts: the tree of the onboard ref, then package.json when the tree has one — two GETs, never a clone, never gh; invariant 5) and the pure detectCommands (src/core/repoToolchain.ts) decides the table: package manager from the packageManager field, else the root lockfile (pnpm-lock.yaml → pnpm, yarn.lock → yarn, --immutable under a .yarnrc.yml, bun.lock[b] → bun, package-lock.json/npm-shrinkwrap.json → npm), else npm; install is the manager's frozen-lockfile form (npm keeps the proven npm install --no-audit --no-fund; every manager detection can name — pnpm and yarn included — is baked into the resident image, execution.md item 10, so a detected install never fails on a missing binary); build/test are <pm> run build / <pm> test ONLY when package.json has that script, else the no-op true (the resident's parseCommands requires both keys; the reply renders it as "none"); a root with no package.json gets NO install and no-op build/test — never an npm command that can only fail. An explicit --install/--build/--test wins per key and is rendered verbatim (an explicit --build true is the operator's command, not a detected "none"). The reply names the toolchain and the reasons (Toolchain: pnpm (package manager from pnpm-lock.yaml; no build script — build is a no-op)); an uninspectable root (no credential, a 404 for a repo outside the installation or an unknown ref, a failed call, no inspector wired) falls back to the npm table WITH a ⚠️ line saying the root was not inspected and which keys took a default (using npm defaults for build and test; when all three were given explicitly the line says nothing was assumed) — the table is never silently assumed. The resident side matches: a table with no install skips the thread-install on a lockfile mismatch (materializeThreadDeps → deps: "none") instead of the old npm install fallback. (b) Outcome: repo.onboard and a non-dry repo.rebuild carry the registry's settle (command-registry.md item 26): after the 202 reply, the chat adapter polls the resident's GET /status (admin client status(resource)) every SETTLE_POLL_MS (10 s) while the state is onboarding and posts a SECOND reply in the thread — ✅ … is warm, or ❌ … failed to provision: <the resident's reason> followed by the hint that fits the reason: when the failure was at install/build/test (a step the table controls), the two commands that fix a bad table (repo reconfigure … --install/--build/--test, then repo rebuild …); for any other step or reason (clone, snapshot, timeout, none recorded) a plain retry (repo rebuild …) naming the resident/Cloudflare side as the likely fault — never the table hint (a failed snapshot put is not a table problem, and the table hint would mislead); a 404 (offboarded meanwhile), a non-200, a transport failure, another state, and the SETTLE_MAX_MS (12 min) bound each produce their own sentence, never silence. The acknowledgement itself now says a report will follow and that repo list shows the live state meanwhile. Best-effort by design: the poll lives in the bot process, so a restart mid-provision loses the follow-up — the resident's state is never in doubt (repo list, the residents dash, the watchdog's deadline), only the notification is.

  3. A failed step names the failure, and the image's toolchain — pnpm, yarn and bun — is pinned to a version, not to a build date (the shape that forced it: a resident down at provision-failed at install: exit 1: [WARN] The "pnpm" field in package.json is no longer read by pnpm … — a warning that cannot fail an install: the same command prints those exact bytes on stderr and exits 0, reproduced inside the resident's own base image. The real error was on stdout and was thrown away, so that failure's cause was unrecoverable). Two changes, one for the report and one for the cause it was hiding. (a) The report never chooses between the streams: describeStepFailure (src/execution/residentStepReport.ts, pure) replaces tail(r.stderr || r.stdout) everywhere a command result becomes an error — assertOk (every provisioning/refresh step), runDepScript (thread dep-cache), POST /read — and reports BOTH streams, labelled, tail-first (a tool's error is its last output), each bounded on its own (400 chars) so one noisy stream cannot crowd out the other; a step that failed silently says no output instead of an empty tail, a timeout still says (timed out), and describing a success throws rather than write a lie into the record. The || was not a style choice but a bug with a family attached: the pnpm/bun family reports through its own logger on stdout, so for those tools stderr-first is exactly backwards. On failure the DO also console.logs stepFailureLog — 4000 chars of each stream into the Worker log (observability is on) — because a 400-char reason is a pointer, not a diagnosis. (b) Pinned toolchains: the resident image installed pnpm@latest yarn@latest, which made the pnpm major a property of the last image build; an image rebuild moved pnpm 10 → 11, and pnpm 11 stopped reading package.json's pnpm field, so a repo keeping overrides / patchedDependencies / onlyBuiltDependencies there fails --frozen-lockfile with ERR_PNPM_LOCKFILE_CONFIG_MISMATCH (reproduced). Both images now name exact versions and assert them at build time (a grep on the installed version), src/deploy/imagePins.ts parses the Dockerfiles and imagePins.test.ts fails if a floating tag returns. pnpm is pinned on the 10 line deliberately: 10.x reads that field, and pnpm ≥10 self-manages packageManager, so a repo pinning 11/12 still gets its own version — the pin is a floor, not a ceiling. That self-management also corrects a claim the Dockerfile used to make: a repo with a packageManager field downloads that version at provision time (a repo pinning pnpm@10.10.0, say), so a pnpm install needs the registry, not just the image. bun is the opposite case: bun does not self-manage packageManager, so the version baked into the image IS the one a repo's bun install runs — and both cloudflare/sandbox bases ship bun as a real binary at /usr/local/bin/bun on the 1.3 line, which rejects a bun.lock at lockfileVersion 3 as Unknown lockfile version (bun 1.4 stamps v3 when overrides carry scoped name@range rules). Taken from the base unpinned, that bun failed deps-install at every attach to a repo pinning bun@1.4.2 (bun install v1.3.14 … "lockfileVersion": 3), and every run on that repo fell to a cold sandbox. Both images now install bun at one exact pin like pnpm's, the base's binary removed first (npm will not write its bun link over a file it does not own, and it has to be /usr/local/bin: a thread's commands run through su, whose PATH is /etc/environment's, not the image's ENV), and the layer's own bun --version | grep -qx proves the bun on PATH is the pin — in the resident once more as worker1, exactly how a thread's install runs. For bun the pin is a ceiling as well as a floor: it must be at or above what any onboarded repo pins, and it moves by a commit, never by a base bump.

  4. A full container disk is disk-full, never serviceable, and the resident recycles it (pure decisions in src/execution/residentDisk.ts, the classifier in residentRefresh.ts). The shape: every attach fails attach-failed at git-setup: exit 4: stderr: error: failed to write new configuration file /etc/gitconfig.lock and falls back cold, while the resident's own state reads degraded(github-unreachable: Failed to write file '/workspace/.resident/git-credentials': ENOSPC: no space left on device …). The obvious hypothesis — a stale /etc/gitconfig.lock left by a killed git — is not this failure: a pre-existing lock makes git config say could not lock config file /etc/gitconfig: File exists and exit 255 (reproduced), and /etc is image layer, recycled with the container. failed to write new configuration file <x>.lock is git's write_error() — a write() on the freshly created lock file failed — and it is reached with exit 4 exactly when the disk is full but an inode is still creatable (reproduced on a full Linux tmpfs: byte-identical stderr). git prints no errno there, which is how a disk full of thread trees read like a lock bug. The defect is twofold: (a) ENOSPC during the cycle's credential-file write was recorded as github-unreachable, a reason on the bot's serviceable allow-list (item 24), so the bot kept attaching to a resident that could not take a worktree, a credential file, or a config lock; (b) nothing freed the disk — the resident stayed that way until the container happened to recycle. Classification: classifyRefreshFailure gains a third class, disk-full: <step> <message> (/workspace: <n> KiB free). A message carrying the errno wording (ENOSPC, No space left on device — Node, git, cp, pnpm all pass it through) is disk-full outright; a kill signature without it stays refresh-interrupted whatever the disk holds; otherwise the Worker runs ONE probe (df -Pk /workspace, only after a step has already failed, never on the hot path) and a free space below DISK_FULL_FREE_KIB (128 MiB — less than one checkout of the largest onboarded tree plus git's scratch, so a fetch or worktree add cannot complete anyway) names the disk; no probe answer is never treated as full. Applied at every step failure the cycle records (the fetch/mint catch included — the github-unreachable site) and to a failed attach step, which flips the resident degraded(disk-full: …) and pulls the refresh cycle to +1 s; disk-full: is deliberately NOT on the serviceable allow-list, so the next dispatch goes cold with resident degraded (disk-full: …) — using fresh sandbox on the card and no attach. The cycle owns the verdict: a cycle entering on degraded(disk-full: …) re-probes before fetching — still below the floor → it decides recovery and returns (a fetch that happened to fit would flip warm, the bot would attach, git-setup would fail, attach would flip it back: a flap loop); space back (a detach or the sweep step freed trees) → the cycle runs and earns warm honestly. Recovery: the disk is a cache, so planDiskFullRecovery stops the container (this.stop(), the same swap reconcileImage does); decided at the cycle's entry gate the recycle throws CycleRestartError, so the engine's retry re-enters the fetch step thirty seconds on and takes the wake path — restoring → restore from R2 → warm on an empty disk — inside the same instance; decided by a failed fetch or step it ends that cycle failed and the next bucket's instance restores; live bindings keep their users and their trees are recreated on the next attach (worktree-missing → attach, item 21). A recycle is refused, with the why on lastRefreshError (… — container kept: <why>) and the degraded reason kept clean, when (checked in this order) the last recycle is inside DISK_FULL_RECYCLE_COOLDOWN_MS (1 h: a disk that refills within the hour is a working set the instance cannot hold — the why says to resize or offboard, and a second recycle would only throw away another restore), when any operation is in flight (the calling cycle excluded from the count), or when a live binding was attached to or used within IDLE_AFTER_S (1 h; an exec bumps lastAttachAt too, item 21) or has a bounded live run registration. The op counter is 0 between a run's tool calls; the registration protects that gap beyond the recent-use floor. Those are the idle-sleep gate's own predicates (item 16b): a platform sleep destroys the disk exactly as a recycle does, so neither runs while a registered run still owns it. Before recycle, item 71 checks every live tree and owner; an uncertain one blocks the whole VM loss. Once allowed, each live tree is measured as its thread user (measureTreeBeforeEviction, runtime up — down, the disk and every tree are already gone), the plan is decided again over fresh facts (an attach or an op that landed during the measurement keeps the container), and each binding — still live, its pool user kept, its tree recreated on the next attach — records what went under its own name beside the eviction record (recycledLeftBehind: {uncommittedChanges, unpushedCommits} when something was there, recycledUnmeasured: <the probe's first error line> when git could not read it — a full disk can refuse the probe's own mktemp — both rewritten on every recycle; recordRecycledTree), with one resident log line per tree that held something (disk-full: <threadKey> tree discarded with the disk — …); /residents live.threads and /debug threads carry both fields. The recovery lives in the refresh cycle only: an attach is in flight by definition, so it never stops the container itself.

  5. Disk is a measured, budgeted resource — never discovered by ENOSPC (pure module src/execution/residentDiskBudget.ts, imported by the Worker like residentDisk/residentRefresh). Item 54 names a disk that HAS filled; this keeps it from filling. Measure: the measure step of every refresh instance, after its sweep (item 7) — every ten minutes on an awake resident, never on an attach's hot path (the admission's free-space term is a live df of its own; the per-part projection it reads from the sample moves only with a cycle) — runs ONE df -Pk /workspace (parseDfKiB: total/used/free) and ONE du -xsk over the parts in duArgv's fixed order — the mirror, then checkout/node_modules BEFORE the checkout (GNU du counts a hardlinked inode once per invocation and charges it to the first argument that reaches it, so the shared deps land on the deps term and the checkout term reads as history + tree + build output), then each live thread's dir (its UNIQUE bytes: own history clone, tree, the item-18 per-thread copies; an install thread's own node_modules), then every pool user's home (homes: a pnpm store or npm cache an install left outside the tree) — and persists DiskSample {at, totalKiB, usedKiB, freeKiB, parts:{mirror, deps, checkout, threads:{<threadKey>: KiB}, homes:{<user>: KiB}, other}} at resident:disk (other = used minus everything itemized: the image, /tmp; a part du could not read is null, never 0). Surfaced as live.disk on GET /residents (and /debug info), as · disk <used>/<total> (<pct>%) on every repo list line, as the gauge on the /residents index row and a Disk section on /residents/<slug> (used/total, free — under the diskBudgetMb cap when one is set — the reserve and its two terms, the headroom as "room for N more hardlinked / M deps-installing trees", the sample time, and every component with homes above 1 MiB), and as · disk max <pct>% (<owner/name>) on the watchdog's firing line (watchdogCheck carries each resident's gauge from storage — the watchdog never touches the container). POST /debug {"op":"measure-disk"} (admin) takes the sample now and answers it. A sleeping container is never woken to measure; before the first measurement of an incarnation the dash says "not measured yet". Measured on a freshly restored pnpm-workspace resident with one hardlinked thread tree: df 15 086 920 KiB total / 4 262 360 used; mirror ≈ 371 MB (the checkout's .git), deps 2 244 052 KiB, checkout rest 462 888 KiB, the thread tree 541 860 KiB unique, /home/worker1 4 KiB — there is NO pnpm store on a restored disk (the store is not in the snapshot) and where an install has run the store hardlinks node_modules on the same ext4 filesystem (link count 2 on every .pnpm file = checkout + the one thread; a store copy would read 3), so deps are never stored twice; du of the readable tree (3 880 312 KiB) + the root-only mirror accounts for df used to within 10 MB. Budget admission (admitThreadDisk, before the mirror lock — making room takes it): a thread whose tree is on disk is reuse (0); otherwise the cost is projected from the LAST sample's checkout parts — hardlink = parts.checkout when the committed lockfile at the ref's mirror tip equals the warm checkout's key (a ref not yet in the mirror is projected as hardlink, the common case), install = parts.checkout + parts.deps when it differs — and the tree is created only when free − reserve ≥ projected (checkDiskAdmission), where free is this attach's own fresh df minus the projections of attaches admitted but not yet on disk (diskCommittedKiB, released when the attach settles), capped at diskBudgetMb − used when the record sets diskBudgetMb below the disk (the first thing that ever read the field; effectiveFreeKiB), and reserve = staging + floor with staging 0.6 × (mirror + checkout) — the archives the cycle stages on the container's own disk (/var/backups/<id>.sqsh, lz4 squashfs, the pair concurrently, removed after the upload), the same ratio instanceSizing.test.ts sizes the instance with (imported, so they cannot drift); the deps are NOT in it since item 61 PR B excluded node_modules from the checkout archive, and the deps-store entry's own archive (item 61, once per lockfile key) is counted as 0.6 × deps only while that backup is in flight (depsBackupInFlight, the Worker's depsBackupsInFlight set), never as a standing charge — before this the standing term multiplied the deps too, and a resident with 6.9 GiB of deps held 5 GiB for an archive of about 1 GiB and refused every attach with 6.7 GiB free and floor max(1 GiB, 5 % of total) (1 GiB ≈ two hardlinked trees of slack for what no projection sees — git pack scratch, a cp -al falling back to a plain copy, /tmp — and 8× item 54's 128 MiB, so admission always speaks before the failure classifier has to). With no sample yet the projection is null and the tree is admitted while free clears the reserve (a fresh resident is never refused on a guess; a blind admission past the floor is still refused); no df answer → admitted (unknown is never refused, item 54's rule). Making room: when it does not fit, the deps store's spares go first — every complete entry no live tree, install or entry backup references (planDepsEviction with DEPS_STORE_MAX_UNREFERENCED_UNDER_PRESSURE = 0, coldest first; the sweep's DEPS_STORE_MAX_UNREFERENCED = 1 warm spare is a cache worth minutes on a future attach, a refused tree is a run falling to a cold sandbox now), removed with their entry backups (item 61 PR B) and df re-probed; then live trees are ordered by orderEvictionCandidates — never the requesting thread, never a tree with an op in flight, never the default branch (the one most likely re-attached; reclaimDecision's rule), never one attached within DISK_EVICT_MIN_IDLE_MS (10 min: longer than any gap between a run's tool calls, shorter than the clean-idle release's hour) — coldest lastAttachAt first; each candidate passes item 71's preservation decision, then its tree is measured as its thread user (worktreeCleanliness) for the eviction record — under item 17 its dirt is never a reason to keep it, and no keep token names dirt — the sweep's re-read guards run (binding unchanged, op counter 0), then evictBinding removes the tree (why disk-pressure, the counts or the probe's failure recorded beside it) and df is re-probed, until it fits. Refusal: still short → 503 {error: "disk-pressure: …", state, reason: "disk-pressure"} — the same shape as mirror-busy, so ResidentExecutor.attach throws and makeExecutor falls back to the cold sandbox with resident attach failed (… disk-pressure: …) — using fresh sandbox on the card (item 24) — and the error text IS the math: need <projected> for a new tree (<kind>), but <free> free[ under the <cap> diskBudgetMb cap] minus the <reserve> reserve (snapshot staging <s> + floor <f>) leaves <x> — short by <y>; evicted <n> deps-store spare(s) (<GiB> back); evicted <n> idle tree(s) (<GiB> back): <keys>; kept <m>: <key> (<why>), … (diskPressureReason; an unmeasured projection says so; spares evicted with no tree following read evicted no idle tree, nothing at all evicted nothing). Never a lifecycle flip: the checkout is intact and every existing tree keeps serving; only new trees are refused. Eviction also clears the user's leftovers: every evictBinding (sweep, reclaim, detach, disk-pressure) removes ~/.local/share/pnpm, ~/.cache, ~/.npm, ~/.yarn, ~/.bun under the pool user's home (threadUserCacheCleanArgv; never the build user's) — an install thread's pnpm store is its tree's hardlink source (0 unique bytes while the tree lives, ALL of them once it is removed) and a pool user is an arbitrary slot, so a store left behind is almost never reused and 16 users × a 2.3 GB store is the whole disk. Sizing (instanceSizing.test.ts, re-derived with these inputs): image 552 MB + mirror 380 + checkout 2772 + reserve (staging 1891 + floor 1074) = 6669 MB base; a hardlinked tree 555 MB (measured whole); an installing tree 2853 MB; the 20 000 MB instance (4 vCPU / 12 GiB / 20 GB, the platform's custom-type ceiling: the CPU is the decision — 16 threads sharing one core was the fifty-concurrent-runs plan's D6 — memory follows the vCPUs at 3 GiB each, the disk follows the memory at 2 GB per GiB) mounts as ~18 858 MB usable → 16 hardlinked + 1 deps-installing trees, or 12 + 2, fit with the reserve; neither the mixed workload (16 hardlinked + 2 deps-installing trees = 21 255 MB) nor 32 hardlinked trees fits this ceiling, so admission is mandatory at every size (the previous 16 000 MB, 15 449 usable, fit 10 + 1 or 5 + 2). Known unmodeled transient: a lockfile change on the default branch while threads hold the old deps (their inodes survive the checkout's git clean) costs one deps term until they release; the reserve (2.9 GB) covers a reinstall (2.3 GB) but not a reinstall plus a concurrent snapshot staging.

  6. No build-user step starts beside a process the last one left behind, and no wait abandons a live process (pure decisions in residentRefresh.ts / residentStepReport.ts; applied in the Worker's buildUserRun and run). The shape that forced it, on a busy resident: main moved six times in twenty minutes with a lockfile change, so every cycle was a full install sharing the 1 vCPU with two thread runs; the install crossed the 5-min budget, the supervisor's kill lagged past the SDK's 30 s output grace, proc.output() rejected Process output did not complete within 330000ms, the cycle recorded refresh-failed: … — and npm install kept extracting into the checkout. The next cycle (no deps marker on disk — full install) ran git clean -fdx over it: warning: failed to remove node_modules/dayjs: Directory not empty on exactly the packages being written (git unlinks while iterating, an entry added underneath it survives the rmdir), checkout-update-failed, 600 s later the same again with a second orphan sharing the CPU — degraded and every run cold for as long as main kept moving. Reproduced negative: git clean -fdx on 300/1000/3000-entry directories inside the resident's own filesystem (ext4, git 2.34.1) succeeds — the leftovers are a concurrent writer, not the filesystem. Sweep: buildUserRun first runs killStaleBuildProcessesCommand(BUILD_USER, CHECKOUT_DIR) as root — steps on the checkout are strictly sequential, so a live worker1 process whose cwd is inside the checkout at step start is by definition a leftover (an abandoned wait, or a step orphaned by a Worker-only deploy resetting the DO while the container kept running); the scope matters because the same user's deps-store installs (item 59) run in their own scratch trees, in parallel, and are live work. Survivors are named on stdout (the Worker log shows WHAT was still running: <step>: killing stale worker1 processes under /workspace/checkout: <pid> <cmd>), then kill -KILL (SIGTERM would let npm keep writing under the clean), then a bounded wait until every matched pid is gone; a process that survives SIGKILL for 5 s fails the step as <step>-stale-sweep-failed — nothing starts beside it. Kill on abandoned wait: run catches the SDK's ProcessWaitTimeoutError from output(), kills the process (SIGKILL), waits up to KILL_EXIT_WAIT_MS for its exit, and returns abandonedWaitStepResult — timedOut: true, the observed exit status or -1 with "no exit status observed", "killed" in the report — so it is the step's own timeout (install-failed: exit 137 (timed out) …), never refresh-failed and never an interruption. Budget: the refresh install runs under REFRESH_INSTALL_TIMEOUT_MS (10 min, twice the build's): a full switchboard install is ~4 min on an idle vCPU and thread runs share it; a timed-out install spends the whole budget and leaves no deps, so the next cycle repeats it — a slow install is strictly cheaper than a killed one, and the cycle is background work (runs keep attaching to the last snapshot). The clean itself is unchanged (git clean -fdx, item 48): it was never the defect.

  7. A timed-out install resumes, and every cycle failure is logged (the second half of item 56's incident, after its sweep: with the checkout swept clean of the orphan, every later cycle still found no deps marker, planned the conservative full clean, wiped node_modules and began the same cold npm install from zero — four cycles in a row, none finishing inside the budget while thread attaches paid their own full install on the same vCPU). (a) The deps-installing marker (INSTALLING_MARKER, the lockfile key) is written before the install step and removed after the deps key lands; a cycle that ends in between leaves it behind, and planRefresh then plans rebuild WITH install on a keep-deps clean — npm reconciles the partial tree to the lockfile, so an install longer than one budget converges over cycles instead of restarting. A marker for another key is stale evidence (full clean, as before); a landed deps key always wins (depsMatch is checked first). Safe for the hardlink invariant (item 48): threads only link deps whose key the deps marker vouches for, and no marker vouched for the partial tree. (b) onRefreshAlarm logs refresh: cycle failed — <classified reason> for every failure, not only StepErrors: a failure between steps (an SDK error, the markers, the snapshot) used to reach only the state entry, which the next cycle's failure overwrites — the install timeout that started the incident had left no trace once the follow-up cycle failed. Proof: src/execution/residentRefresh.test.ts::planRefresh: a timed-out install resumes instead of starting over plus the live row below. Both hold on the Workflow lifecycle (item 7): the instance's install step writes the same installing marker before installDeps, and every step failure of the repository's own passes through the same refreshFailed, so the refresh: cycle failed — <reason> line is logged whichever scheduler ran the cycle.

  8. A lockfile-diverged thread reconciles on top of the shared cache, never from an empty tree (pure decision planThreadDeps + threadDepsMechanism in src/execution/residentDepCache.ts, the reconcile cost kind and RECONCILE_DEPS_RATIO in src/execution/residentDiskBudget.ts, applied in the Worker's materializeThreadDeps and admitThreadDisk). Before: a thread whose committed lockfile differed from the warm checkout's key ran the install from nothing — install 1.94 GiB projected per tree, 5+ min on the 1 vCPU it shares with the refresh cycle and every other thread, so a burst of seven reviews within 13 minutes exhausted the disk (three of them diverged) and a ship run's /attach died before its install finished. Now the diverged tree is seeded from the checkout by the item-18 mechanism (hardlinked node_modules, mutable paths swapped) and the install runs on top: npm/pnpm replace a changed package with fresh inodes and can never write through a shared one (worker1-owned, write bits stripped), so the delta is all the tree costs in bytes and time. Admission projects such a tree as checkout + RECONCILE_DEPS_RATIO × deps (0.25 — a bound above the typical few-package delta, to be re-measured against du of live reconciled trees; unmeasured deps stay null, never a guess), and the residents page's headroom line counts lockfile-diverged (reconciling) trees at that cost. A diverged tree whose command table has no install is not seeded (deps:"none"): nothing could reconcile it. The attach answer names the mechanism reconcile whatever the seed's own mechanism was. Proof: src/execution/residentDepCache.test.ts::planThreadDeps …, src/execution/residentDiskBudget.test.ts::projecting …, web/src/pages/residents.test.ts::ResidentDetailPage::item 55: the Disk section …, plus the live row.

  9. Dependencies live in one content-addressed store per resident, behind one primitive (pure module src/execution/residentDepsStore.ts, the view via depCacheScript's nodeModulesSrc in residentDepCache.ts; applied in the Worker's materializeDeps / linkDepsView / adoptCheckoutDeps / sweepDepsStore). Before this, deps were installed IN PLACE and PER CONSUMER — provisioning into the checkout, the refresh cycle into the checkout again on every lockfile change, a thread whose branch lockfile differed from the checkout's into its own 1.9 GiB tree (item 18's install mode). Three install sites, three budgets, none aware of the others; item 56's refresh spiral and item 58's ship attach that died mid-install both lived in the seams, and one resident ran 32 installs in a day for ~14 distinct lockfile keys because "older than main's lockfile" and "different from main's lockfile" were the same comparison. Store: /workspace/deps/<lockfileKey>/node_modules — the key is the item-48 hash of the committed lockfile (a pure function of the commit; anything not 64 lowercase hex is refused as a path segment), the entry holds EVERY node_modules the install produced — the tree's top-level one plus each outermost nested one at its tree-relative path (deploy/<worker>/node_modules; nestedNodeModulesListCmd, .git never a source, copies inside a listed node_modules stay with their parent) — exactly as the install produced them, plus .complete (written LAST, inside the entry so the rename carries it) and .used (touched on every hit — the LRU clock). The nested dirs are entry content because npm installs a workspace's conflicting versions into the workspace's own node_modules and the lockfile mandates those paths: an entry that carried only the top-level dir left every view short — the packages hoisted for a nested dependent lost their dependent, npm ls --omit=dev marked them extraneous and attributed their subtrees to production, and the root repo's licenses:check failed on @img/sharp-libvips-linux-x64 (LGPL, a dev-only dependency via miniflare) on resident trees while the same commit passed on a clean npm ci. Views hardlink the nested dirs by the same mechanism as the top-level one (each named on a nested=<rel> line, its tool caches swapped per root), gated on the tree having the parent directory; the harden strips owner write from nested files too; the entry archive (item 61) is the entry dir minus its markers (DEPS_ENTRY_BACKUP_EXCLUDES — .complete is the restore commit's to write, LAST), restored into the scratch root, and an archive from before nested coverage fails that commit and falls through to the installer as any bad archive does. The root repo's npm run check:deps-drift (scripts/deps-drift-check.mjs, part of check:consistency) names this drift class directly — a lockfile package path missing from the tree, a node_modules entry the lockfile does not know — before any downstream gate trips on a symptom. Proof: src/execution/residentDepsStore.test.ts::the install scratch tree and the store commit::commit: every OUTERMOST nested node_modules the install produced moves into the entry at its tree-relative path…, src/execution/residentDepsStore.test.ts::deps-harden: the install's node_modules is made owner-read-only, and a tree with no lockfile may hold none::script: nested workspace node_modules are hardened like the top-level one…, src/execution/residentDepCache.test.ts::depCacheScript nested node_modules (a lockfile that installs into a workspace's own node_modules)::runs for real…, src/depsDriftCheck.test.ts::deps-drift evaluateDepsDrift::the incident's shape is named…. An entry is immutable once complete: its files are owner-read-only (deps-harden, chmod u-w before it becomes visible) so even worker1's build fails EACCES on a write through a shared inode instead of mutating every consumer's tree — Flyweight (GoF): many consumers, one tree by identity. The primitive materializeDeps(key, sha, installCmd, budget): planDepsMaterialization — complete entry → hit (touch .used); an install for that key already running in this incarnation → join its promise (never a second install; a complete entry beats a stale memo); else install: git clone --shared --no-checkout of the mirror into .scratch-<attempt> (objects via alternates, seconds), checkout --detach <sha>, chown to worker1, the repo's install command as worker1 in that scratch tree under REFRESH_INSTALL_TIMEOUT_MS, harden, then depsStoreCommitScript as root: mv scratch/node_modules → .staging-<key>-<attempt>/, mv staging → entry (atomic; a racer that finds the entry complete drops its staging and keeps the winner), touch .complete .used, rm -rf scratch. An install that produced no node_modules is a failure when the commit has a lockfile (there was something to install) and an empty entry when it has none (depsHardenScript, emptyOk for NO_LOCKFILE_KEY — the sha256 of the empty ls-tree listing): a repo whose install is a no-op (true, a terraform tree) gets an empty node_modules owned by the build user, and every consumer's view links an empty directory — without it a rebuild of such a repo dies at deps-harden (find: '…/node_modules': No such file or directory), and only dropping its install command brings it back. The install runs outside the mirror lock — it touches no consumer's tree — so a full install no longer holds every attach behind the mutex (the staging step item 47 asks for). Parallelism: distinct keys install concurrently up to nproc (depsInstallSemaphoreSize, read once per incarnation, unreadable → 1) — never a constant; on today's 1 vCPU that is 1 by arithmetic. Consumers: provisioning materializes main's key and links the checkout as a view (linkDepsView = depCacheScript with the entry as nodeModulesSrc + the item-18 tool-cache swap, whose copies are made writable again — chmod -R u+w — because cp copies the hardened mode bits); the refresh cycle on a lockfile change materializes the new key before taking the mirror lock, then under it cleans (-x, item 48) and re-links the view; a thread attach materializes its branch's key and links — a thread on a stale branch, a resident lagging main, or a key that flipped mid-life all hit an entry main built, and only a branch that genuinely changes deps installs (once, for every thread on that key). The attach answer's deps is hardlink/copy for every key now; install is history. Adoption: a checkout that holds node_modules for a key with no entry — a disk from before the store, or a fresh R2 restore (the snapshot carries the checkout's tree, not the store) — is adopted under the mirror lock (adoptCheckoutDeps: the same commit script with keepScratch, then the view): a rename plus hardlinks, seconds, same inodes before and after, so threads that linked from the checkout earlier are untouched. Eviction (sweepDepsStore, after every disk measurement): depsStoreListScript lists entries with complete flag, size and .used; protected = the checkout's key, every live binding's depsKey, every install in flight; planDepsEviction removes debris (incomplete, nothing in flight) first, then the coldest complete spares beyond DEPS_STORE_MAX_UNREFERENCED (1 — the item-55 sizing leaves room for about one ~2 GiB entry beyond the checkout's; the attach's disk-pressure path keeps none, DEPS_STORE_MAX_UNREFERENCED_UNDER_PRESSURE, item 55), and every .scratch-*/.staging-* leftover once nothing is in flight. Disk accounting (item 55): parts.deps is now the store's total (du charges the shared inodes to it, measured before the checkout and the threads), so checkout reads as history + tree + build output as before; the install projection (checkout + deps) over-projects when the store holds several entries — conservative, unchanged. Snapshot: the checkout archive EXCLUDES its top-level node_modules and each entry has its own archive — item 61's content-addressed snapshots (the store no longer "rebuilds lazily" on a wake: the warm key's entry is restored, or installed when it has no archive). /attach streams like /exec: heartbeat whitespace then ONE JSON document over HTTP 200, a refusal's status in the body (ResidentExecutor.attach reads it there; pre-validation 400/404 keep real statuses), so an attach that waits on an install cannot lose the connection the way a plain response did live (fetch failed at 272 s). Not here: pre-warming keys the mirror fetch sees, the resident as an R2 producer, CI as the producer (the key then gains the image digest, and the miss path tries R2 before the local installer).

  10. Purging synthetic bindings (load-harness.md item 8; pure decision src/execution/bindingPurge.ts, imported by the Worker like residentDiskBudget.ts). Item 23 keeps a binding after eviction on purpose, so a load run that attaches fifty load:<runId>:<i> threads would leave fifty evicted rows on /residents/<slug> and in /debug threads forever. POST /debug {"op":"purge-bindings", "resource", "prefix"} (admin) deletes the EVICTED bindings whose threadKey starts with prefix and answers {purged: [keys], keptLive: [keys]}. selectBindingsToPurge refuses (HTTP 400, the reason named) a prefix that is not a whole thread-key namespace or longer (load:, load:r1: — never empty, never a bare partial namespace) and refuses the production namespaces outright (slack:, http:, mcp:, cli:), so a purge can only ever touch synthetic keys; a live binding under the prefix is never deleted, only listed as keptLive. Nothing on disk is touched: an evicted binding has no worktree and no pool user.

  11. A restore is judged by the bytes still arriving, never abandoned to a clock (pure judgeRestoreProgress + RESTORE_POLL_MS / RESTORE_STALL_MS / RESTORE_MAX_MS in src/execution/residentRefresh.ts, applied by the Worker's restoreWithProgress in doHydrate). The shape that forced it: a ~2 GiB checkout restore completes in 481 s — the same snapshot had transferred in 47 s and 104 s earlier the same day — but a fixed 300 s budget (R2_TRANSFER_TIMEOUT_MS, item 7) has already declared it failed, a second hydrate runs rm -rf over the tree the first one is still filling, and the resident goes down(r2-restore-failed) on a disk with 11.5 GiB free until the watchdog rebuilds it from GitHub. The SDK's restore cannot be cancelled or observed, so the wake path observes the disk instead: every RESTORE_POLL_MS it samples du -xsk of the SDK's backup work directory (/var/backups, where presigned downloads first land as download.* part files before the final <backupId>.sqsh appears) PLUS the target (where extraction lands); it counts positive changes from a pre-restore baseline, so cached archives do not count and temporary-file cleanup does not hide later growth; while observed bytes grow it waits, however long; it fails as stalled when nothing has been written for RESTORE_STALL_MS (the clock runs from the start until the first byte, and a sample du could not take neither counts as growth nor resets it) or as capped past RESTORE_MAX_MS — ONE deadline for the whole hydrate, shared by the wait for a previous attempt's restore and both restores, so the worst-case restoring span is the cap itself, under the watchdog's 30-min stale-mid-flight window, and the verdict stays the wake path's. The two restores run sequentially (the SDK serializes backup operations on one queue; a concurrent pair only let the second one's clock run while it waited). Restores this incarnation started are tracked in pendingRestores; a later hydrate awaits them (bounded by what remains of that same deadline) BEFORE clean-before-restore, and goes down with the disk untouched if they will not settle — a clean never races a writer. On success the size and rate are logged (<what>: <GiB> in <s> (<MiB/s>)) so the constants can be revisited from evidence. The snapshot upload keeps its fixed budget (there is no target to observe from the resident's side). Proof: src/execution/residentRefresh.test.ts::judgeRestoreProgress … plus the live row. A restore that goes down takes the container with it: pendingRestores keeps the next HYDRATE off the directories, but a down resident's only exit is a rebuild, and provisioning owns the same directories — a restore the wake path has given up on lands into the checkout a rebuild has just cloned and linked; tar overwrites in place through the deps store's hardlinks (item 59), resetting every hardened entry file from 444 to the archive's 644 (.package-lock.json, the tool-cache copy the view does not share, alone stays 444) and rewriting freshly cloned tracked files. Same lockfile key, so identical bytes; a different key would be corrupted silently. So both down exits of the wake path — a stalled/capped restore, and the bounded wait for an earlier attempt's stream expiring — first stop the container (clearIncarnationMemos + stop(), the disk-full recovery's own move; the disk is ephemeral and the stream dies with it) and say so in the reason (… — container stopped so the transfer cannot land on a rebuild); and provisioning waits for pendingRestores (bounded by RESTORE_MAX_MS) before its clean-workspace, failing the provision as await-restores rather than cloning over a writer. The bytes do not travel through the Durable Object (pure backupTransferMode in src/execution/residentBackupTransfer.ts, applied in takeSnapshot and reported on GET /healthz). The SDK's localBucket: true is its documented LOCAL-DEVELOPMENT mode, in which the DO reads the archive from the R2 binding and pumps it to the container over the control RPC (and the reverse on upload), so a 128 MB isolate sits in the data path of every transfer — and a 1.16 GB checkout restore exceeds the isolate's memory while the progress probe's du shares the connection: the isolate is reset, the transfer dies with it (backup.restore … interrupted because the runtime changed), the state stays restoring with no down, and only a manual force-down + rebuild brings the resident back. In presigned mode the DO signs GET/PUT URLs and the container moves the bytes itself (downloadBackupParallel, resumable from prepareRestore's existingSize); the SDK's requirePresignedURLSupport reads exactly CLOUDFLARE_ACCOUNT_ID, BACKUP_BUCKET_NAME (wrangler vars: {{account}}, {{script}}-cache — the bucket name MUST equal the BACKUP_BUCKET binding's, which the offboard sweep deletes through) and R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY (secrets, deploy/secrets.manifest.json, an R2 API token scoped Object Read & Write to the one bucket). backupTransferMode(env) is presigned (localBucket: false) only when all four are non-blank; anything missing is local with the missing names — fail-closed to local mode, never to a broken snapshot — logged once per snapshot and shown on /healthz as backupTransfer (+ backupTransferMissing). The mode rides on the SDK handle, so a snapshot restores the way it was taken; records from before the switch keep restoring through the DO until the next cycle re-snapshots. Once the DO is out of the data path, item 61's du probe during a download is harmless. Content-addressed snapshots (pure CHECKOUT_SNAPSHOT_EXCLUDES, depsBackupStorageKey, DEPS_BACKUP_TTL_S, depsBackupsToDrop and the restore branch of planDepsMaterialization in residentDepsStore.ts; applied in takeSnapshot, backupDepsEntry, restoreDepsEntry, doHydrate, sweepDepsStore, teardown). Without this every refresh cycle re-uploads the whole checkout — hundreds of MB to over a GB, a dozen times a day on a busy repo — and every wake restores it whole, although since item 59 the node_modules inside it is a hardlink view of an immutable store entry: the one part that never changes once written. Now the checkout archive is taken with excludes: ["node_modules"] (the top-level view only; mksquashfs anchors a bare pattern at the archive root), and each entry is archived ONCE, right after deps-commit or deps-adopt (backupDepsEntry: the entry's node_modules dir, off the caller's critical path, 180-day TTL, recorded on the DO as resident:depsBackup:<key>; presigned mode only — local-bucket mode would put the DO in the data path of a deps-sized upload; the key is protected from eviction while the upload runs). A wake restores mirror + tree as before, then materializeDeps(stamp key) with the new restore backing: the archive comes down into a private .scratch-<attempt> (the SDK extracts wherever the handle's dir says, so the handle is re-pointed), judged by restoreWithProgress like every restore, then the same commit script an install ends with — staging, atomic rename, .complete LAST — so a partial download never becomes an entry; a restore that fails for any reason (expired, missing, stalled) drops the record and falls through to the installer, which records a fresh archive. A snapshot from before this change still carries node_modules and is adopted on restore as before. The wake decides by what the CHECKOUT holds, not by what the store holds: a checkout with node_modules after the adopt (an old snapshot) is done; one without gets materializeDeps — a hit when the entry survived on disk, else the archive, else the installer — and the view linked; the deps checkpoint is written ONLY with the view in place, so a wake whose materialization fails (or whose registry lookup fails) goes warm with the tree whole and the checkpoint unwritten, and the next refresh cycle's plan repairs it (no deps marker on disk — full install) — a cache miss is never a down. The materialization lives INSIDE the hydrate's one deadline (planWakeDepsBudget): the restore is judged against that deadline, the installer's budget is the smaller of its own and what remains, and under WAKE_DEPS_MIN_MS nothing starts — so the worst-case restoring span is still RESTORE_MAX_MS. Every other caller of the restore backing (an attach) gets min(budget, RESTORE_MAX_MS) from now, never longer than it would wait for an install. A presigned restore is a MOUNT, and the resident extracts it (pure residentRestoreExtract.ts: restoreMountDir, extractRestoreScript, unmountRestoreScript, unmountAllRestoresScript; applied by the Worker's restoreExtracted for the mirror, the checkout and a deps entry, and by the unmount-restores step before every clean-before-restore / clean-workspace). In presigned mode the SDK's restore does not extract the archive, it mounts it — squashfuse on the .sqsh under /var/backups/mounts/<id>_<ts>_<rand>/lower, fuse-overlayfs at the handle's dir with a writable upper beside it — and leaves the .sqsh in /var/backups (the SDK's own comment: unsquashfs extraction is its LOCAL-DEV path). A mount point breaks code written for one ext4 filesystem: rm -rf /workspace/mirror → Device or resource busy (the resident loops degraded on its wake path), chown -R after a restore → a copy-up of every inode (chown-failed: exit 143 (timed out)), du -x → the mirror and checkout measured as ~1 MiB, and every hardlink or mv between the checkout and the deps store crossing devices. So the handle is re-pointed at a staging mount beside the target (<target>.restore-<attempt>, still under /workspace where the SDK allows it), judged by bytes arriving like every restore, then extractRestoreScript puts a real tree in place — unsquashfs -n -no-xattrs straight from the downloaded .sqsh when the image has squashfs-tools (the Dockerfile installs it; the log says extract: unsquashfs), cp -a out of the mount while an older image lacks it (extract: cp) — unmounts the staging mount and then every squashfuse lower under the SDK's <backupId>_* dirs (found by the backup id, not by the overlay's options — fuse-overlayfs exposes no lowerdir= in /proc/mounts, so an unmount that looked there would leave the lowers mounted, each pinning its unlinked .sqsh), removes those dirs and the .sqsh, and renames the extracted tree in LAST, so a failure leaves neither a half target nor a mount. After a wake the disk is exactly what it was before presigned mode: one filesystem, hardlinks and renames work, du -x is right, chown -R is cheap. The mount model could later be turned to our advantage (a per-consumer fuse-overlayfs over one squashfuse lower is copy-on-write isolation without hardlinks) — a separate design. Eviction drops the archives of the entries it evicts (depsBackupsToDrop); teardown deletes every archive; a rebuild keeps them, so provisioning after a rebuild restores the warm key instead of installing it. POST /debug {"op":"deps-backups"} (read scope) lists the records. The disk budget's staging term (item 55) counts the entry archive (0.6 × deps) only while an entry backup is in flight, the cycle's mirror + checkout archives always; re-deriving both from the measured sizes of the last archives is the next step. A killed extract is an interruption, not a snapshot failure (the incident that named it: a resident under a release's deploy roll): the roll SIGTERMed checkout-restore-extract 300 ms in; its exec came back a normal result — exit 143: no output — and only the cleanup execs 10 ms later carried the SDK's container is not running wording, swallowed as the best effort they are; restoreFailureDisposition read a snapshot failure and the resident went down(r2-restore-failed: exit 143: no output — …) on a snapshot nothing was wrong with, until a person rebuilt it nine minutes later. Now the disposition reads the kill signature the instance step's classifier already reads (INTERRUPTION_SIGNATURE: exit 143, SIGTERM, Session terminated) as restore-interrupted, and restoreCheckout asks the runtime after any failure that was not a typed replacement (isRuntimeActive; a typed replacement is false without asking): a runtime that is gone took the disk the extract wrote to with it, so the verdict is restore-interrupted: … — the container stopped under the restore whatever the exit code said; a (timed out) still wins (the stream may be running on a live disk). The verdict travels on a StepError carrying the reason, which classifyRefreshFailure reads as an interruption (^restore-interrupted:, at the start of the message only) — the instance step retries onto the new container, as item 5's live row shows for the SDK-worded case.

  12. Resident text is made safe at the seams, never at the surface. A resident's reason, error and summary strings are built from remote output (git, npm, the container's shell) and once carried other threads' identifiers, and they reach card titles, Slack replies, repo list and stored records. One helper, residentText (src/execution/residentText.ts: strip terminal control sequences, redact credential shapes, cap), is applied at three seams and nowhere else — with two caps by field: reason, stateReason (the lifecycle reason beside state on a thread route's 503 — the same remote text reason carried there before it, execution.md item 9) and summary at RESIDENT_TEXT_CAP (300, a card note or a repo list row), error at RESIDENT_ERROR_CAP (1000, a whole reply), because a refusal such as item 35's carries the admin's path to the fix and GitHub's own body past the first 300 characters and a cap that cuts them defeats the message: on the resident at the write (setResidentState, recordRefreshError) and at the exit (json() and the heartbeat-streamed document, through sanitizeResidentBody, which rewrites only error, reason, stateReason, summary and a stderr that mirrors error); on the bot at the parse (parseResidentBody, the /status probe's own body read, the admin client), permanently, because a reason stored by an older resident survives that resident's deploy and its rollbacks. The probe's state is validated against the lifecycle union plus not-onboarded/unknown, never echoed. The disk-pressure refusal names sizes, counts and keep tokens (DiskKeepWhy, with other as the caller's fallback) and never another thread's key or its free text. Step-report tails are stripped and redacted before the cut. /op stdout and stderr are redacted before the clip. Every card note is one line (oneLine), and the setup-failure close and reply are redacted. The same rule closes the GitHub half of the class: every client that slices a response body into an error redacts first. stdout from /exec is tool output and keeps its publish-time redaction; resident-minted ids and refs are outside the class.

  13. Every attach and op answer carries its step trace (tracing.md item 19). AttachOk.trace and OpRunOk.trace — and a ThreadErr's, on a refusal — list the commands the Worker ran for that request — runOk's named steps, the op's own command, each mutex_wait — as offsets from the request's start with their exit and timeout, bounded (64 steps, 8 KB) and sanitized (src/execution/residentStepTrace.ts, shared with the bot). The collector is per request (an AsyncLocalStorage around attachThread and runOp), so two concurrent attaches never share steps and a refresh cycle records none. The bot grafts them under the span that made the call; nothing else reads them, and an older bot ignores the field. The same steps are the children of the resident's own resident.attach / resident.op root on its span log, which joins the bot's trace when the request carried one (tracing.md item 22).

  14. A container whose runtime never answers heals itself — the unreachable probe is named, counted and escalated (pure isRuntimeUnreachableSignal, runtimeUnreachableRung, runtimeUnreachableReason, SDK_CONNECT_TIMEOUT_MS, WAKE_PORT_READY_MS, SDK_RUNTIME_RECORD_KEY in src/execution/residentRefresh.ts; the Worker's run(), escalateRuntimeUnreachable, recreateContainer, forgetRuntimeIdentity, debugRecreateContainer). The incident this names: from one moment on, every sandbox.exec against one resident rejected after exactly 30 s with AbortError: The operation was aborted from inside the SDK's wake path (RuntimeBootstrapProbe.probe → ContainerControlConnection.fetchUpgradeAttempt: the WebSocket upgrade to the container's control port 3000 was never answered; the 30 s is the SDK's module constant DEFAULT_CONNECT_TIMEOUT_MS, not ours). The platform saw a healthy container; the in-container runtime was silent. Each refresh instance failed after three such aborts (the ready-stamp probe, the disk probe the classifier added, the measure step's df) and recorded degraded(refresh-failed: The operation was aborted); the watchdog created the next instance every bucket; nothing escalated for forty minutes. An admin stop-container did not help — stop() is a SIGTERM the SDK signals and never waits on, and a wedged runtime ignored it — and neither did a /rebuild, which reprovisioned onto the same running container; an offboard (the first destroy(), a SIGKILL) and an onboard did. Not one of record 0029's alarm-chain incidents: the scheduler ran every bucket; what it ran could not reach the container. Named: at the exec choke point (run()), a spawn-phase rejection that is not a replacement (isRuntimeReplacement first) and carries the SDK's connect abort on its cause chain — the AbortError name, or the DOMException's exact message when a wrapper copied only that (isRuntimeUnreachableSignal; a command's own Aborted (core dumped), a timeout, a crash never match) — is RuntimeUnreachableError, whose message is the reason runtime-unreachable: the container's control port did not answer within 30 s (attempt N of 6); the classifier (classifyRefreshFailure with the caller's count) answers the runtimeUnreachable outcome with that reason and never guesses from the wording on its own (the bare message without the caller's word stays refresh-failed: …, as before); a disk probe is never consulted for it (another 30 s abort). Counted in storage, not memory: resident:runtimeUnreachable {count, firstAt, lastAt}, one more per unanswered connect (every exec counts — the attach path's too), cleared by any exec whose process spawned; the isolate a deploy or an eviction swaps does not reset it. The ladder (runtimeUnreachableRung, decided where a refresh step's exec found the port silent; every rung records its reason as lastRefreshError and the state, logs one line naming the rung and the count, and is not evidence about the repository — NON_EVIDENCE_REASON, never the park streak, never serviceable): rung 1, attempts 1–2 — degraded(runtime-unreachable … — the step is retried) and the step thrown to the engine, whose retry re-enters it thirty seconds on (the short re-arm a refresh-interrupted gets); rung 2, attempt 3 — the incarnation swapped, stop() sent (cheap; enough for a runtime that still honours signals), degraded, thrown; rung 3, attempts 4–5 — recreateContainer: the SDK's runtime identity forgotten (SDK_RUNTIME_RECORD_KEY, so the SDK's destroy skips the runtime cleanup it would otherwise wait out against the silent port, and a destroy that fails midway leaves no identity to re-adopt), the VM destroyed (destroy(): SIGKILL, a fresh disk), every snapshot and entry backup kept, the incarnation swapped, degraded with the pending restore named — a restoring marker with no restore running would hold the cron off for the stale bound, so the retry's wake path flips restoring itself, finds no runtime and restores mirror, checkout and deps from R2 (minutes, where a rebuild is half an hour); rung 4, attempt ≥ 6 — a recreated container did not answer either: the VM destroyed and down(runtime-unreachable …), a reason REHYDRATION_FAILURE_RE matches, so item 36's auto-rebuild strikes apply and the rebuild (now on a fresh container, item 34) is the last resort. Under the engine's six attempts (item 7) one instance runs the whole ladder — about twenty minutes from the first unanswered connect to down — and a count that outlives the instance carries into the next bucket's; the housekeeping steps skip a resident whose row exists (stopped: runtime-unreachable), so an instance costs one abort, not three. The wake budget: the first exec after a (re)start is bounded by the SDK's physical start — WAKE_PORT_READY_MS (3 min; the template's SANDBOX_PORT_TIMEOUT_MS, the SDK's env knob, 10 s to 600 s, default 90 s) waiting for the control port to accept a request — plus ONE fixed 30 s connect; a slow boot spends its time in the port wait and is not counted (item 5). The operator (item 13): POST /debug {"op":"recreate-container"} is rung 3 on demand, answering {recreated:true, restoreStartedAt} and running the restore through the ordinary wake path in the background — a failure the wake path did not record itself is recorded as degraded, so restoring is never stranded; refused mid-flight and on down. stop-container is unchanged. /debug info (read scope) carries runtimeUnreachable: {count, firstAt, lastAt, rung} | null. A full reset uses repo rebuild --full (admin POST /reset) to destroy the VM and remove resident state while keeping its registry settings. Offboard followed by onboard (item 11) performs complete removal and requires registration again. Both reuse confirmed teardown and keep the SDK schedule table; incomplete backup cleanup retains its handles and blocks provisioning until a checked full rebuild completes it. Not reproducible live end to end (no way to wedge port 3000 from outside a healthy container); the proofs are the unit ladder, the source guards and the recreate-container live row below.

  15. A container that exited under a live run is a pause, not a dead sandbox: the client waits for the wake before it counts a strike (pure isContainerRolling, wakeDecision, wakeWaitBudget, sandboxRestartedMessage, WAKE_WAIT_MAX_MS, WAKE_POLL_MS in src/execution/residentWake.ts; ResidentExecutor.awaitWake, attachOnce, attachRefusal in src/execution/resident.ts; ExecSandboxRestartedError in src/execution/executor.ts). The shape that forced it: the container rollout that follows a resident Worker deploy stops each resident's container and starts it on the new image three to ten minutes after the deploy, whether or not the image changed. A coding run admitted inside that window had its next /exec refused in-body not-serviceable: The container just exited (the thread preflight's ensureHydrated failing at the container start, before the command ran), twice in a row; the bot read both as ExecInfraError, the native loop's two-strikes rule aborted the run with "Aborting instead of retrying into a dead sandbox", and under a minute later the refresh instance had restored the snapshot and the resident was warm again. The deploy preflight (item 44) guards the other direction only. Now, on a refusal that names a rolling container (isContainerRolling: a not-serviceable: whose detail is the platform's The container just exited or Container is starting, or a runtime-replacement wording shared with item 43's RUNTIME_REPLACEMENT_WORDING, or attach's image-stale:), the client first asks for the engine view with the operator token, GET /status (the only view that scope opens: runtimeUnreachable lives on /debug info, a read-scope route the operator bearer cannot call), and decides (wakeDecision). The container is coming back while the engine holds the snapshot it wakes from: restoring (a wake already running), any serviceable state (item 24's isServiceable: warm, refreshing, degraded(github-unreachable)), or a degraded reason the engine retries on its own (the resident's NON_EVIDENCE_REASON: stale-mid-flight, restore-interrupted, runtime-unreachable). A definite answer no wake recovers from (down, onboarding, a repo failure, not onboarded, an off-table state), or a Worker that did not answer at all, is the strike it always was. While waiting it polls /status every WAKE_POLL_MS (5 s) and re-attaches (attachOnce, under the attach's own default clipped to the run's remaining clock where the caller carries one — the one bound every attach request has, the rolling wake's re-attach that recreates the worktree included; the budget and this ceiling bound the probing, never a request — execution.md item 9) as soon as the engine says it serves — after a container's exit the engine view lags, so the first re-attach follows the first pause; in a wait a transient refusal began (execution.md item 9) the first serving view is current and is re-attached into at once: the attach's own hydrate is the wake when nothing else has started it, and a refusal that still names a rolling container keeps the wait going. The resident going down mid-wait, a re-attach that does not answer, or a spent budget ends the wait with the strike, naming what was last seen and what began the wait (its classification code: container-exited here, transient-refusal for item 9's wait); any other attach refusal is the attach's own error; a hard stop rejects at once with the stop's one typed error (aborted, classified as the transport), from the pause, a probe or the re-attach alike. The budget bounds the WAIT — the probing — and is the command's own (wakeWaitBudget: the bash tool has already clipped timeoutMs to the run's remaining wall clock minus the write-up reserve) under a three-minute ceiling, so a run never probes past its wall clock; each attach REQUEST inside the wait is bounded by the attach's own default (the exec default, BASH_TIMEOUT_MS), clipped to the RUN's remaining wall clock less the write-up reserve where the executor carries the run's clock (ResidentExecutorOptions.remainingMs, the run control's lease clock the run loop starts on the harness's lease event; inside the reserve, or under an attach's floor past it, no request opens — the typed refused, on which a relaunch ends the run on its budget, the lease read again before any refusal the attach's own failure would become; a resumed run's probe wait and wake budget are clipped to the lease less the reserve) and never by the operation's budget or the command's timeoutMs — a re-attach that recreates the worktree may outlast a short command's clock, never the run's (execution.md item 9). After the wake the idempotent routes (/read, /write) are re-issued; /exec is handed back to the caller as ExecSandboxRestartedError (not an ExecInfraError) carrying sandboxRestartedMessage, the wait and the fresh worktree's ref@sha, since the re-attach recreated the tree from the mirror and uncommitted changes and unpushed commits from earlier in the run are gone with the old container's disk. The native loop settled that call with a synthetic result and went on; the pi harness reads it as the container replaced under the run — pi ran inside that container and is gone with it — and ends the run interrupted with a sandbox_restarted note, its request dispatched again under the same run id (harness-pi.md item 16) — that is the behaviour; a relaunch of pi in the replacement container is an unscheduled option recorded in record 0038's third amendment. Deliberately not done: a longer strike budget (it would keep every genuinely dead sandbox alive for minutes), and deciding from the exec refusal's own state field (it is the state the DO held a moment before the refusal; the wait needs the view as it changes).

  16. A resumed run's attach reuses the thread's worktree as it stands and refuses by name a tree it cannot keep; nothing is wiped (pure parseReuse and decideWorktree in src/execution/residentReuse.ts, imported by the Worker like residentReadonly; the bot side is ResidentExecutorOptions.reuse, ResidentReuseRefusedError and the factory's reattachWorkspace). Background: item 17's discipline treats every tree as a leftover of an earlier run, which is right for a fresh run and wrong for one that comes back after a bot restart (run-history.md item 38): the attach the resumed run made wiped its own dirty tree under a still-living pi, or raced that run's in-flight install, failed at worktree-clean, and the bot moved the run onto a cold sandbox silently. The shape: the bot sends reuse: true on a resume alone (a fresh attach, and an older resident, see the body they always did); the Worker parses it before anything runs (absent → false, a boolean → itself, anything else → 400) and hands it down attachThread → attachThreadBody → attachThreadCreate → ensureThreadWorktree, which measures the tree as the thread user (exists, readable, dirty, HEAD; the ancestry probe only for a provisioning attach whose HEAD is not the target) and runs the pure decision: reusing plus readable → reuse, whatever the dirt or the HEAD, the tree untouched; reusing plus missing, unreadable or built for the other mode (item 50) → refuse with the reason, thrown as ReuseRefusedError before the one rm -rf is reached and answered 409 {error: "reuse-refused: <why>", needs: "recreate"} after the allocation's rollback; provisioning → item 17 byte for byte. The client types the 409 as ResidentReuseRefusedError; the factory turns every failure of a re-attach into WorkspaceReattachRefusedError and provisions nothing in its place (item 24). The answer also names the container the tree is in (container: the kernel's boot id, read once per incarnation and memoized, so a replaced runtime re-reads it), for the run's row (harness-pi.md item 8).

  17. A cycle that keeps failing in the resident's own steps heals itself; one that fails in the repository's command keeps parking (pure REPO_COMMAND_STEPS, isRepoCommandStep, stepOfFailedReason, parksOnRepeat, INFRA_STREAK_RECREATE_AT = 3, INFRA_STREAK_DOWN_AT = 5, infraStreakRung, infraStreakReason in src/execution/residentInfraStreak.ts; RefreshFailure.step; the Worker's refreshFailed, noteInfraStreak, the gate's park condition, refreshComplete): the split item 64 drew for one signature, drawn for every refresh failure. A step that runs the repository's own command (the onboard-time command table: deps-install, build, test) failing is evidence about the repository — the head is broken — and nothing a rebuild does changes that: provisioning would run the same command and fail the same way, and a down runs no cycle, so the resident would never notice the head being fixed. Those stay degraded(<step>-failed …), the gate's park streak (DEGRADED_PARK_AFTER_CYCLES, item 16b) slows them to the idle cadence, and they heal on the first cycle after the head is fixed. Every other step is the resident's machinery — git against the mirror (fetch, checkout-update), the probes and markers, the restores, a failure between steps (refresh) — and a failure there that repeats is the container or the disk gone wrong, which a rebuild does fix. The week before this rule: one resident sat degraded(deps-install-failed: … configured to use bun) for three cycles and parked — right, a command-table fix was the cure; another failed one cycle with checkout-update-failed: fatal: Could not parse object and self-healed — had it kept failing, nothing would have escalated, ever. The ladder over resident:infraStreak ({step, count, firstAt, lastAt}, one writer noteInfraStreak): counts 1–2 record and let the next cycle try (degraded(<step>-failed …), as before); count 3 recreates the container (item 64's rung 3, recreateContainer: the VM destroyed, snapshots kept, degraded(infra-streak: 3 consecutive cycles failed in the resident's own steps (last: <step>, since <t>) — the container was destroyed, snapshots kept; the next cycle restores from the snapshot), and the next cycle's wake restores mirror, checkout and deps from R2 onto a fresh disk); count 4 is the recreated container's one cycle of its own; count ≥ 5 destroys the VM and goes down(infra-streak: … a recreated container failed the same way; rebuilding) — a reason REHYDRATION_FAILURE_RE matches, so item 36's transition rebuild fires at once, under its budget. A completed cycle, a repo-command failure (our machinery got that far) and the down rung clear the row. infra-streak: is a non-evidence reason (NON_EVIDENCE_REASON): the gate never parks on it and always runs the next cycle; and the park streak itself now counts only reasons parksOnRepeat allows — a repo-command step's failure, or a reason naming no step (github-unreachable) — so a resident-step failure is never parked, since parking would only slow its own ladder. A disk-full failure keeps its recovery (item 54), before any counting. The fetch step names GitHub only for GitHub's failures: refreshFetch used to record every failed mirror fetch as github-unreachable — a serviceable reason (item 24) that parks on repeat — so a mirror broken on disk read as GitHub being down and never climbed; the first live row of this item found exactly that (degraded(github-unreachable: 'origin' does not appear to be a git repository …), streak unmoved). Now the pure fetchFailureIsMirrors (MIRROR_FAILURE_WORDING: the remote gone from config, not a git repository, bad object, corrupt, unreadable or empty objects, a .git/ file it cannot open) routes a fetch that failed on the mirror through refreshFailed as the resident step fetch — degraded(fetch-failed: …), never serviceable, counted by the ladder; everything else (DNS, timeouts, HTTP errors, refused credentials, a repository GitHub says is not there) stays github-unreachable, GitHub's to fix and the resident's to wait out. /debug info carries infraStreak: {step, count, firstAt, lastAt, rung} | null. Fault injection (admin scope, test residents only): POST /debug {"op":"infra-streak","count":N} writes the row (0 deletes it), so one real failure lands on the rung under test; {"op":"break-mirror"} removes the mirror's config on disk (its remote with it), so the next cycle's fetch fails in a resident step — the corrupt disk the recreate rung restores from the snapshot. Not the objects: the wake check (readyStamp) tests objects/ and the checkout's .git, and a missing directory is a disk to restore, never a fault that reaches the fetch — the first live attempt healed itself that way before any step failed.

  18. A container that did not accept the connection is a wait on the resident too (runtime-busy). The thread sandbox's execution.md item 28, on the resident: when the platform's own accept allowance for the SDK's control-port connect runs out before the SDK's 30 s timeout — under a command saturating the cores, and on an idle container too (item 28) — the container port's fetch throws a plain Error, Container is taking too long to accept the connection; the application could be overwhelmed with load — inside the wake path, before any process started; the token names the refusal, never the cause the platform's words guess at. It was seen on a resident settling after a roll, as resident /exec: followed by the platform's words, and it ended the review. It is neither a replacement (the runtime did not change) nor the silent port of item 64 (the container accepts again in moments), so it is counted as neither. The contract: (a) the Worker names it at the exec choke point's SPAWN alone (run(), after the reset and the replacement checks, before the unreachable count) as the typed SandboxRuntimeBusyError (src/execution/sandboxErrors.ts, the recognizer isRuntimeBusySignal over the cause chain); the thread data plane (execThreadImpl, readThreadFileImpl, readThreadBytes, writeThreadFileImpl, and threadRejectionErr for a throw that crossed the stub) answers it as runtimeBusyErr — a 503 with reason: "runtime-busy" and the system cause, streamed in-body on /exec like every failure — and the catch-all reads the word as the platform's transient; a running command's output failure is never named so, since the process exists and a re-send would run it twice. (b) The client (callWaitingOutBusy, around every send opWithReattach makes) re-sends the identical request on the token after 3 s, 5 s, then 10 s, inside the command's own budget capped at RUNTIME_BUSY_WAIT_MAX_MS (5 min), and throws ExecCapacityError naming the refusal when the wait is spent — never ExecInfraError, never a re-attach (the worktree is as it was); a hard stop ends the pause at once with the run's one typed aborted error. (c) An older Worker's bare platform words — the typed 500 with no token — stay an ordinary infra answer after one send: the token decides, never the words. (d) The refresh cycle's own execs meet the same typed word from run(), and the cycle yields to it: classifyRefreshFailure reads the token anywhere in the failure's text (carriesRuntimeBusyToken — a StepError or the fetch step's mint prefix keeps it; the platform's bare words alone are not the token) as busy, refresh-yielded: <step> …, decided before the disk inference and with the disk probe skipped; the instance step answers stopped (runtime-busy), releases the cycle lease and puts back the settled state the fetch step recorded under its refreshing (resident:refreshingFrom, rewritten before every refreshing write) — warm, or the degraded an earlier cycle earned — and the Workflow skips the sweep and the measure, which exec on the same container; the next cron firing tries again. Never degraded, never a rung of item 67's ladder (three cycles of it once destroyed the container under the very run that kept it busy), never the ladder of item 64, and never a retry into the same busy container.

  19. The fleet drain — a deploy closes the fleet to new runs, the runs in flight end, the Worker upload lands, changed containers reconcile, the fleet reopens (record 0059; drain.ts, the routes and the gate in worker.ts, the client's wait in resident.ts; the deploy side is release-and-deploy.md item 31). Item 44's preflight refuses while any resident has a run in flight, and under steady traffic — ship children holding a repo for up to their 45-minute lease, new ones starting as old ones end — no minute is ever quiet: three releases of one evening lost five 30-minute waits and left the resident two releases behind the bot. So the wait is made to end by closing the door it waits on. The drain is one record in the registry Durable Object — { since, until, by, reason }, key drain, outside the resident: prefix — set by POST /drain {minutes?, reason?, by?} (minutes an integer in [1, 90], default 60; a second drain replaces the first until the upload fence is set; replacement then refuses) and cleared by POST /undrain (idempotent, answers cleared) — both of the drain scope, with POST /reconcile, legacy POST /deploy-fence, POST /prepare-deploy-fence, POST /abandon-deploy-fence-acquisition, GET /deploy-fence-acquisition?id=<uuid> and the existing selected POST /staging-seal-window beside them. The separate RESIDENT_DRAIN_TOKEN bearer passes these eight drain-scoped routes and no operator, read or admin route (admin passes them too), so the release deploy holds no more than the drain and its swap need: a leaked CI secret can close a fleet — 90 minutes per post, for as long as its holder keeps posting until the token is rotated — or restart idle stale containers (state the disk never holds), and can never offboard, rebuild or read one. The reopen comes after the applicable image proof, never at preflight-ok: an authenticated unchanged named application and complete current image reports permit the Worker-only upload to skip /reconcile; otherwise the deploy runner posts /reconcile after its Worker deploy and before /undrain (release-and-deploy.md item 31) — the handler walks the registry and asks every resident's Durable Object to reconcile its container onto the current image (reconcileForDeploy, item 16b's reconcile under the deploy's own name; a failing resident degrades to its own error row) — because a fleet reopened seconds after preflight-ok admitted runs into containers the reconcile then restarted under them; a resident confirmed inactive reports current at once — its next start uses the deployed image by construction — on every reconcile, so a pending marker written after one idle pass is consumed by the next; an activity-probe failure proves neither state and preserves the pending marker and drain hold: deploy and refresh answer deferred for a later reconcile, while a new attach with that marker is refused stale; a resident that answers deferred (something in flight, or a registered run) or error restarts on its own next quiet refresh cycle (never on an attach, item 16b / issue 2101 — and while the report is owed every new attach is refused onto the seeded sandbox, so the resident can go quiet). An active resident's pending report remains through the stop and is sent only after the replacement's own hydration reaches warm, and the reconcile's deferral on a live registration (item 16b) is what keeps even that later restart from landing under a run. GET /residents carries it as draining (null for none). The gate is the Durable Object's, in attachThreadTraced after hydration and before the image reconcile: while the drain is in force an attach without a live registration owned by the attaching run (item 44's row, bounded by its deadline plus grace) is answered as the streamed refusal { error: "draining: …", status: 503, draining: <record> } — a new attach is a new run — and an attach for the run that owns the live registration passes: a rolled container's re-attach (item 65), an evicted worktree's recovery attach, a resumed run's reuse attach are the runs the drain waits for, and refusing them would hold the fleet closed on the very run it is closed for. An ordinary drain lets already attached runs finish /exec, /read, /write, /publish, inspection and /detach; the upload fence closes new admissions to each process-bearing route and waits for admitted work to finish before the Worker swaps. (A registration whose release never came grants no new run an exception; its deadline or the clean-idle sweep ends its protection. A registry the Durable Object cannot read while issuing an admission permit refuses the call; unknown admission cannot authorize an upload.) The end is the record's own: liveDrain(stored, now) reads a record whose until has passed, or a shape this build cannot read, as no drain — Durable Object storage survives the isolate swap the deploy performs, so a drain nobody lifted must expire by itself, and it does within its own minutes (at most 90). The client waits (ResidentExecutor.attach): a draining answer — read by its record on a 503, never by its words, so it is never the platform's transient of item 27 — is waited through with one re-attach every DRAIN_POLL_MS (30 s) under the run's own lease less DRAIN_LEASE_RESERVE_MS (10 min, what the attach and the work after it need), DRAIN_WAIT_MAX_MS (60 min) with no lease — and under the fallback's own cost where one stands behind the attach (issue 2101: a run refused by the drain waited 18 minutes at its attach for the whole deploy — the preflight's wait, the swap, the image gate — then did the job in a seeded sandbox that stands up in about two): the factory's first attach passes drainBoundMs = DRAIN_FALLBACK_WAIT_MS (DRAIN.fallbackWaitMs, 3 min), and the first drain turns that bound into one absolute deadline carried through every drain/wake hand-off, so a drain refusal falls to the seed after at most a few minutes even across repeated deploy resets, while a resumed run's re-attach and a mid-run recovery — which have no fallback, their worktree lives on the resident or nowhere — pass none and keep the lease's bound; every error thrown out of the wait carries the drain's share (drainWaitOf), so the fallback's selection still publishes the run's drain_wait note; the run's stop ends the wait at once; the binding carries the wait as wokeAfterMs, so the card's opening note says the run started late and for how long; an answer that is no longer the drain is judged as a first answer is (admitted, transient → the wake wait with what the budget left, else the attach's own refusal); the budget ending with the fleet still drained is the typed ResidentDrainingError (refused), naming how long the run could wait and when the drain says it ends. Inside a wake wait's re-attach (item 65) a draining answer hands the wait to this drain wait rather than spending the wake budget on it: the deploy's own isolate swap lands inside its own drain, so an attach that meets the Durable Object reset enters the wake wait — but the drain record survives the swap, the re-attach that follows meets it, and the run then waits out the drain under its lease instead of striking at the wake budget's minute and falling back cold on exactly the requests the drain exists to hold. The hold has liveness (issue 2044: the reopen gate's first live deploy held the whole fleet 34 minutes on a container the platform never cycled): the first hold stamps holdsUntil — the cycle's measured bound (DRAIN.cycleBoundMinutes, item 65's three-to-ten-minute roll), clamped to until — and liveDrain reads a held record past it as no drain: the fleet reopens by construction, whoever died between the reconcile and the lift, with the registry's alarm (armed at the bound, the earlier end) naming the stale containers in a warning (staleHolds); an active one restarts on its own next quiet refresh cycle, while one confirmed inactive reports on the next reconcile because its next start already uses the deployed image. A replacement start, rebuild or fresh provision counts as the report only after it reaches warm: the hydration path and runProvisioning then report their own fresh container current — each runs on the image the deployed Worker pins by construction — so a mid-drain replacement or rebuild reopens the fleet instead of leaving the hold standing. The waits are legible (issue 2044's second and third defects; issue 2101's third: 18 minutes with zero events between the attach span's start and its end): the client's drain wait paints waiting for the deploy to finish · N min through onSetupNote on every poll and clears it at the wait's end; the drain wait and the attach's wake wait are each one child span under the attach span (dispatch.workspace.attach.drain-wait, dispatch.workspace.attach.wake-wait, named on the run page by ATTACH_WAIT_NAMES in displayNames.ts), failing with the typed error's classification — the refusal's words stay on the card note and the attach error, never on a span (tracing.md item 2); the binding carries the drain's share as drainWaitMs beside wokeAfterMs — and the sandbox fallback after a drained attach carries it on the selection (ExecutorSelection.drainWaitMs) — and the dispatcher publishes either as the run's drain_wait note (run-friction.md reads it as its own category); GET /residents carries each resident's imageReport (pending names a container whose post-deploy report is owed) and repo list leads with the drain and marks every row drained — attach refused, so warm is never printed bare for a repository no attach can reach; a typed repo test/repo build on a drained fleet is answered op-refused: the resident fleet is drained … at once instead of running or hanging. Why the drain lives in the Worker and not the bot: the bot is one of the fleet's callers, not the only one (the CLI, a load harness, a second bot generation mid-roll), and the deploy that needs the door closed runs outside every bot; the registry is the one place every attach already passes. For the orchestration plane (orchestration-plane.md item 9) the registry posts the drain as a drain level — the set posts above and arms ONE alarm at the record's until, /undrain posts below, and the alarm posts below for a drain nobody lifted — each post riding the registry's answers (planeOutbox) until superseded, so the plane's resident-drain window opens and lifts on the drain's own life.

  20. The resident guards its own memory (pure memoryGuard.ts; the wiring in worker.ts; the token MEMORY_PRESSURE_REASON in src/execution/sandboxErrors.ts, read by both sides like runtime-busy). A container at its cgroup memory cap wedges its own control port and every run on it loses its work, and before this item no log line carried a memory number. The reader: one cgroup v2 reading — memory.current, memory.max, cpu.stat's usage_usec from /sys/fs/cgroup — taken as ONE exec (CGROUP_READ_ARGV through the exec choke point run(), so it shares item 68's classification) behind the small CgroupReader interface (a fake in tests), at every /exec and /attach start and on the measure step of the refresh instance the watchdog cron creates (item 7 — the tick that exists; the watchdog's own pass reads storage alone, item 55's rule, and a sleeping container is never woken to be measured). Each sample is ONE structured log line — used bytes, cap bytes, percent — so Workers Logs has the number; no polling loop anywhere. The gate, two named constants over the last reading: at or above MEMORY_SOFT_LIMIT_PCT (80) a NEW /attach is refused 503 {error: "memory-pressure: resident near its memory cap, N% used (… bytes), queued …", state, stateReason, reason: "memory-pressure"} — the same shape as mirror-busy, so the bot falls back cold or waits legibly and the card says why (item 24; ANSWER_OWN_WORDS keeps the token out of the lifecycle reading); at or above MEMORY_HARD_LIMIT_PCT (90) a NEW /exec is refused by name with the same numbers — while every command already running finishes, never killed: the gate sits at each route's start (exec before its preflight; attach after the drain gate and before the image reconcile, so a refusal never restarts a container) and kills nothing. A run already in flight — registered from attach to release (item 44) — re-attaches THROUGH the attach gate exactly as it passes the drain (one registration read decides both): a rolled container's re-attach, an evicted worktree's recovery and a resume belong to the runs the gate holds the resident open for, and refusing them would strand the warm worktree and lose the very work the gate protects; the sample (and its log line) is still taken. A reading gates only the container it measured: an inactive runtime invalidates the guard's reading before the gate asks (sampleMemory → invalidate() — a container that no longer runs holds no memory, and a fresh hydration memo can skip the restore that would wake it), and a typed runtime replacement under the probe (RuntimeReplacedError from run()) travels as MemorySampleUnavailable(…, invalidates) and forgets the reading too, so the route falls through to its own runtime-replaced answer instead of a memory-pressure off a dead container's numbers. An UNREADABLE cgroup (a missing file, unparsable output) logs once and disables the gate for the incarnation rather than refusing everything; any other transient read failure (the container busy under the probe, MemorySampleUnavailable) keeps the last reading and the gate. The gauges: the last reading is persisted (resident:memory) and carried on GET /status (memory), GET /residents (getResidentInfo) and the watchdog answer, so the residents page and the bot can show it; GET /healthz deliberately stays without it — it touches no DO (item 1), and the reading is per-resident. The soft side is also the resident's memory level for the orchestration plane (orchestration-plane.md item 9): at or past the soft line the level reads above, carried on every answer's levels document.

  21. One preservation decision before automatic workspace loss. Sweep, detach, finished-ref reclaim, disk pressure, fresh attach replacement, idle sleep, image reconcile, disk-full recycle and automatic runtime recovery call the same resident decision before they remove a checkout or its VM. A known run stays bound when its owner is live, uncertain, provisional or mismatched, regardless of deadline or TTL. Removing a present tree requires the exact terminal run and matching generation, fence, branch and current private Git tree. A saved checkpoint must match the current HEAD with no later tracked or non-ignored untracked changes. A terminal run, including a Ship review child without a coding settlement, can also release its clean tree when it matches the resident's recorded attach HEAD and has no unpushed commits; a changed or missing attach HEAD does not qualify. Missing or malformed Git proof keeps the binding and names the blocker. For eviction alone, an exact terminal owner may retire a binding whose whole thread directory is independently absent on a warm active VM outside hydration and recreate admission, with a sole UID claimant, no other UID directory, empty staging and no UID process; the eviction rechecks absence under the mirror lock, skips directory and home-cache deletion, records the missing private tree as unmeasured and first spends a missing historical UID. An unreadable or present directory never qualifies, and this exception cannot authorize VM loss or fresh attach replacement. The resident reads one exact owner from the state Worker, which checks live ownership before stored history and does not apply list retention. Truly untracked legacy caches remain eligible for normal cleanup. The decision does not claim to preserve ignored files, files outside the Git tree, explicit admin destruction or platform loss.

  22. Terminal metadata survives a lost detach response. The watchdog applies retained terminal workspace versions before lifecycle gates, without waking a container. Fresh attach replacement also applies the same reconciliation before its preservation decision, so an already settled predecessor does not require a watchdog tick. Registration transfer carries the prior physical owner on the same binding before materialization; capacity refuses transfer without dropping references. The resident commits bounded branch facts and exact terminal proof before acknowledging the matching ledger version. A lost acknowledgment retries from durable proof. Cleanup requires exact-owner acknowledged high-water readback after every pending revision has retired; seeing the oldest terminal version does not prove exhaustion. Current live evidence, incomplete mapping, mismatched physical identity and capacity refuse acknowledgment. Retained proof survives history and obligation retirement; it never substitutes for the private-tree checks in item 71.

  23. A live review owner may advance its clean checkout. A read-only same-ref attach with an explicit target commit may replace its old checkout under the existing thread and mirror locks only when the request, registration, durable fence and canonical live ledger identify the same run and physical binding. The current incarnation and sole UID claimant must match; no thread operation or UID process may remain. The current Git tree must still be at the recorded attach head with no tracked, non-ignored untracked or unpushed changes. A complete numeric process list permits one second of exit grace, followed by the entire exact-owner, fence, physical binding, tree and process check again. A process still running, incomplete output or changed evidence preserves the tree; the grace never signals a process or replays a command. Missing or mismatched evidence preserves the tree. This authority does not authorize named-ref replacement, own-PR rebind, default-ref return or terminal cleanup.

  24. An admin may retire an explicitly discarded, absent terminal workspace. The request pins the thread, ref, UID, recorded boot, attach time and run generation/fence. A request without the expected generation/fence reports the exact terminal owner for confirmation without side effects. A durable discard fence blocks old callbacks and new attachment while pending. Exact terminal ownership, the high-water fence, sole UID claimant, settled native operations and the existing full absence proof on a stable warm boot are required. For legacy runs without an exact settlement, only a matching exact-owner absent answer permits a second canonical final-history read; live rows take priority, and the final non-provisional run must match the ID, thread and repository. This exception applies only to explicit admin discard, creates no settlement or acknowledgement, and does not relax automatic preservation. A present tree, cold runtime, changed target or unknown observation remains held. Retirement is idempotent and preserves history, publication evidence and the high-water owner. A permanent per-thread retirement fence rejects cached older or ownerless binding snapshots; only an active higher-fence attachment may create a new physical workspace. The exact discarded owner’s incomplete evidence remains archived in the loss receipt and cannot become a private-work obligation of the successor; unrelated predecessor obligations remain protected. A receipt retry rechecks the durable retirement and high-water fence using storage only; it reports no current boot and cannot start a VM. Physical reset proof remains a separate observation. The spent UID remains spent until confirmed VM destruction; this operation cannot reset it.

  25. Identity allocation does not impose a small historical-owner cap. New owners receive fresh unprivileged Linux identities on demand after a durable, serialized ownership reservation. The exact same owner may reclaim its identity; another owner never may. A cold-start reattach verifies or recreates that recorded account before user-scoped work. Account provisioning must confirm the generated name, numeric UID and private home before model commands run. Failed or uncertain provisioning retains its reservation and native outcome fence. Legacy ownership and high-water records remain protected. New owners skip the entire precreated legacy roster, including unspent names; confirmed VM destruction resets usage while allocation stays beyond surviving bindings and the prior high-water mark. The separate active-owner limit, disk, memory, cancellation and preservation checks remain in force. Privileged, malformed and unrecorded users never gain command authority. Historical identity usage is reported without a fixed pool denominator; platform UID bounds and actual resource pressure remain valid limits.

  26. An attach refusal names its class as a setup word (record 0097; the words in src/core/runSetup.ts, shared by both sides; attachRefusalCode and preservedWorkspaceCode in deploy/cloudflare-resident/threadErr.ts; attachRefusalCodeOf in src/execution/resident.ts). The Worker's attach answer carries code beside its sentence and reason whenever one word names the refusal. attachThread, the one attach exit, fills it from typed fields: a fleet drain's record is fleet_draining, and a reason that names one class maps to its word. The sites whose reason two classes share set the code themselves: the deploy fence (deploy_fence, beside the drain's draining), a pending or unverified predecessor settlement (settlement_pending, beside workspace-preserved, kept through a worktree step's error), an unverified workspace or pool-UID owner (owner_unverified) and a failed attach step (attach_failed). A refusal no word covers (mirror-busy, stale-tip, busy, disk pressure) carries none. The bot stamps the code on the attach error it throws: a definitive refusal's own (its absent or unknown code as fallback_other), the drain wait's fleet_draining, or, for a transient refusal whose wait ended with no definitive refusal, the transient refusal's code. An absent or unknown code reads as fallback_other, never as a guess from the sentence, so the Worker and the bot deploy in either order.

Validation criteria ​

The resolver's literal result objects are the independent proof for current-message PR flags, absent inherited flags, fetch-failure flags and closed-thread refusal. Their dedicated flag-only and closed-PR subset tests add no separate contract; the PR URL/shorthand and re-review fail-closed proofs below retain those obligations.

CriterionProof
PR-head resolution uses the normal read-scoped GitHub facade and fails closed when credentials are absent or cannot be resolved; it never falls back to an unauthenticated request. Current same-repository branch tips, frozen closed and fork heads, validated size and description facts retain their existing authority. Failures log only closed credential/read/decode/head codes, HTTP status when known and a bounded hexadecimal GitHub request ID; provider bodies, messages and credentials are excluded.[unit] src/core/repoContext.test.ts::resolveRepoContext: read-scoped PR lookup::*; src/core/repoContext.test.ts::PR size for the digest-coverage guard::*; src/core/repoContext.test.ts::PR title and body for the description artifact::*; src/execution/githubApi.test.ts::RestGithubApi — pull request reads::*
Quiet eviction acknowledges the removed repository and preserves backup and prefix-cleanup failures[unit] src/core/commands/repo.test.ts::repo onboard::quiet eviction names the removed repo and preserves cleanup errors
Cold registrations appear beside warm residents; warm rows are highlighted, cold rows have no resident-only gauges, and listing frames update both modes[unit] web/src/pages/residents.test.ts::ResidentsIndexPage::shows cold registrations beside highlighted warm residents and keeps both current; src/channels/residentsView.test.ts::createResidentsViewHandler::serves the index shell from a LIVE registry read on every request (never cached), the listing passed through as the seed
75: dynamic identity ownership accepts more than 250 distinct owners while refusing cross-owner, privileged, malformed or unrecorded execution[unit] src/execution/residentPoolSpends.test.ts::dynamic resident UID ownership::*; deploy/cloudflare-resident/uidOwnership.test.ts::indexed resident UID ownership::*; deploy/cloudflare-resident/runRegistration.test.ts::resident workload admission::creates fresh identities for 252 completed operations through the native admission boundary; ::keeps a failed account creation reserved and its unknown native outcome occupied; ::reports work capacity independently of 250 retained terminal owners without touching the VM; deploy/cloudflare-resident/runRegistration.test.ts::exact-thread owner reconciliation::restores the exact retained owner's OS account before reattaching after a cold VM start
75a: the UID section of a resident status report reads existing state without migration or native work; missing or corrupt UID state is unverified, valid state keeps its issued count and bounded owner preview. This report-body guarantee does not cover Resident or SDK constructor effects during cold-object activation.[unit] deploy/cloudflare-resident/runRegistration.test.ts::resident UID status report::*; deploy/cloudflare-resident/uidOwnership.test.ts::stored resident UID report::*
74: explicit admin retirement checks exact terminal ownership (or matching exact absence plus canonical legacy final history) and stable physical absence, fences late requests and retries, and preserves publication, history and spent UID ownership[unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::explicit absent-workspace discard::*
2: the shipped registry admits five residents, refuses a sixth with 429, accepts cold metadata without using a resident slot, and preserves existing residents above the cap[unit] deploy/cloudflare-resident/repoRegistration.test.ts::metadata-only repo routes::admits five residents, refuses a sixth, accepts cold metadata, and preserves existing residents above the cap
19a: separate workload limit, terminal-file retention, lingering processes, unknown-owner occupancy, concurrent final-slot admission, dynamic operator UID provisioning, native restart markers and acknowledged operator completion[unit] deploy/cloudflare-resident/runRegistration.test.ts::resident workload admission::*; deploy/cloudflare-resident/runRegistration.test.ts::a run's registration is held from attach to release::refuses new work before materialization but permits the exact live owner's reattach
Read-scoped workload inspection uses the admission counter's unique owners and records each reservation, operator, binding, original native record and observed UID contributor. It reports build, logical repository, physical resident, observation interval, cached or unknown boot, exact run ownership and native record hashes, a bounded UTF-8 document and explicit owner, source and detail omissions. Oversized detail is compacted before identities are dropped. Partial reports retain only members of the counter itself, never terminal bindings recorded for context. UID history is joined only for counted owners; retained allocations never add workload slots. Capture completeness is not health, recent progress, termination, cleanup authority or proof of a UID leak. A primary read can use its legacy resource address. A secondary read requires matching stored member identity; missing, unreadable or mismatched identity stays unknown and incomplete. A full admission refusal logs its original bounded observation; a later read cannot reconstruct a historical roster. Inspection never starts an inactive VM or changes admission, kill or recovery outcomes. Its explicit manual probe mode preserves the durable runtime recovery counter on both successful and failed observation, including output and custody guard failures; ordinary admission and maintenance still account for recovery. Guarded observation-failure and diagnostic-sink warnings carry only their event, resource or refusal target, closed reason and bounded error type, never the exception payload. A warning failure preserves the original partial report or refusal. A successful manual audit logs one bounded summary of the exact returned repository, physical resident, interval, count, completeness, omissions and closed issues without owner payloads; a failed summary sink preserves that report and its original obligations.[unit] deploy/cloudflare-resident/runRegistration.test.ts::resident workload admission::audits the same unique owners as admission, with original native evidence and terminal trees excluded; ::labels separate members and their admission refusals with the stored physical resident ID; ::does not label an uninitialized or mismatched secondary as the primary; ::keeps workload evidence incomplete when the resident identity cannot be read; ::keeps an incomplete process or UID observation explicitly unverified without clearing holds; ::an early audit failure warns without exception payload and a failed warning cannot replace partial evidence; ::logs one bounded successful audit summary and a failed log leaves its report and original obligations unchanged; ::bounds audit output and records the original full-capacity contributors without admitting work; ::bounds both audit and original refusal JSON bytes while retaining all sixteen compact owner identities; ::an incomplete observation lists only the counter's captured members, excluding retained terminal bindings; ::reads an inactive resident without creating a native runtime; ::joins a held native owner's recorded UID without turning retained history into workloads; ::a delayed diagnostic owner read does not block new admission or rescan the captured count; ::fresh native owner reads are deduplicated and bounded to the returned contributor projection; ::a delayed old owner observation cannot overwrite the current workload hint; ::does not attribute a foreign or moved-fence canonical response to the held native owner; ::a failed diagnostic sink leaves the full-workload refusal and held owners unchanged; ::manual audit preserves*; ::manual unreachable observation preserves count two so the next ordinary failure uses the stop rung; ::ordinary admission and maintenance retain their original unreachable accounting after an audit; deploy/cloudflare-resident/nativeOperation.test.ts::resident workload audit HTTP::*; src/core/commands/repo.test.ts::repo.workload::*; src/core/residentAdmin.test.ts::makeResidentAdminClient (real fetch client)::reads workload evidence through the existing debug route without retry or a mutation fallback; src/channels/commandContract.test.ts::migrated commands over MCP — grants (AE12)::repo_workload requires repository access on the resolved MCP actor; [gap] Audit all held owners through deployed native MCP; historical refusals without a captured roster remain unproven.
Workload inspection defaults to the primary. repo workload <slug> --resident N resolves a one-based active member from the registered listing after repository authorization, routes through the dedicated replica debug endpoint, and accepts only that physical resident ID in the reply. Pending, missing, duplicate, unreadable and mismatched targets refuse without a fallback or retry; the audit changes no owner or native outcome.[unit] src/core/commands/repo.test.ts::repo.workload::selects an active second resident and rejects a report for its sibling; ::refuses a pending or unknown resident before requesting its workload; src/core/residentAdmin.test.ts::makeResidentAdminClient (real fetch client)::reads a selected member through the replica debug route without replaying a refused request; deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet lifecycle::routes workload inspection to the selected member and rejects raw placement fields; deploy/cloudflare-resident/runRegistration.test.ts::resident workload admission::labels separate members and their admission refusals with the stored physical resident ID
23: the configured sweep page fits its Workflow timeout without platform clamping[unit] deploy/cloudflare-resident/refresh.test.ts::the refresh module — the Workflow entrypoint leaves worker.ts, which re-exports it for the binding::the sweep timeout covers its configured page budget without platform clamping
23: a 32-binding pool is swept in bounded durable pages; protected first-page bindings cannot starve an eligible later binding, and evicted history consumes no page space[unit] deploy/cloudflare-resident/pausedWorkspace.test.ts::paused unpublished work at resident removal::sweeps a larger pool in bounded durable batches without starving later bindings
42a: resident summaries and details show current lifecycle, run activity and disk usage without UID counters, UID tooltips or Linux worker IDs; the disk detail combines home directories into one total[unit] web/src/pages/residents.test.ts::ResidentsIndexPage::shows current run and disk status without historical UID counters or worker IDs; web/src/pages/residents.test.ts::ResidentDetailPage::shows lifecycle and disk details without historical UID counters or worker IDs
73: exact live same-ref owner advances a clean known physical checkout; dirty, untracked, changed-head, incarnation, fence, busy, process and missing-ledger evidence preserve bytes[unit] deploy/cloudflare-resident/workspaceAdvance.test.ts::exact-owner review workspace advance::*
73: a refused head advance records the failed owner, live binding, private tree, process or owner recheck in the existing attach step trace; it preserves the same refusal and bytes without exposing owner values; an unavailable trace collector grants no permission[unit] deploy/cloudflare-resident/workspaceAdvance.test.ts::exact-owner review workspace advance::records the refused advance check without releasing bytes…, ::keeps the original refusal shape when a trace collector is unavailable
72: terminal metadata is retained before ACK and down-runtime gates; lost ACK retries; live, mismatched and over-cap facts refuse; registration transfer retains its predecessor; reconciliation logs the thread key, count of predecessor records actually removed and remaining count, without claiming checkout replacement[unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::resident terminal metadata reconciliation::*
71: admin-only POST /debug with op: "inspect-preservation" and exact input: {threadKey, ref, head, runId} observes the existing terminal owner's registration, fence, physical binding, sole UID claimant, empty process list and strict Git counts without waking or changing the workspace; native boot identity, bounded complete streams and final binding revalidation are required; the existing release decision retains all settlement/version-ACK guards and the diagnostic never releases a binding[unit] deploy/cloudflare-resident/dependencyInspection.test.ts::retained resident preservation inspection::*, src/execution/residentCleanliness.test.ts::bounded native private-tree observation::*; [gap] Deploy the reviewed diagnostic and observe the original retained binding before claiming live release proof.
71: a live or uncertain owner survives deadline, TTL and cleanup; a matching terminal saved checkpoint permits release, as does a terminal review child without a coding receipt whose clean tree remains at its recorded attach HEAD; later edits, untracked files, moved HEAD, an unverified tree and partial fences block release[unit] deploy/cloudflare-resident/pausedWorkspace.test.ts::paused unpublished work at resident removal::*
71: a terminal run whose thread directory is independently proven absent on a warm active VM outside hydration and recreate admission may retire its empty binding during eviction, recording that private tree evidence was lost; the exact fence, sole UID claimant, clean directory and staging scans, and empty process list are required; absence is rechecked under the mirror lock without deleting a directory or home cache; an unreadable or present directory, a live or uncertain owner, and automatic VM loss remain blocked[unit] deploy/cloudflare-resident/pausedWorkspace.test.ts::missing private tree recovery::*
71: sweep, detach, ref reclaim, pressure, attach replacement and automatic VM loss share the guard; delayed owner reads and mirror waits cannot remove a changed binding[unit] deploy/cloudflare-resident/pausedWorkspace.test.ts::one preservation guard covers automatic loss paths::*, deploy/cloudflare-resident/pausedWorkspace.test.ts::paused unpublished work at resident removal::*
71: the state Worker reads the exact live owner before terminal history; provisional or malformed records never authorize release[unit] deploy/cloudflare-memory/runLedger.test.ts::exact owner evidence for resident preservation::*
61: the sandbox consumes the same committed dependency entry format as the resident: root and workspace views retain their repository-relative paths and symlinks; tracked source leaves retain their current contents or absence while untracked dependency siblings are restored in the same directories; missing workspaces are omitted, and destination escapes or tracked-ancestor conflicts refuse before any dependency replacement[unit] src/execution/seedPlan.test.ts::resident dependency entry consumed by a seeded checkout::*; src/execution/seedPlan.test.ts::dependency entry materialization::*
70: the pure gate — one reader output parses into used/cap/percent (an uncapped max gates nothing); below both thresholds nothing changes; between them a new attach is refused memory-pressure … queued with the percent while exec runs; at the hard threshold a new exec is refused with the numbers in its error; each sample logs one structured line with used bytes, cap bytes and percent; an unreadable cgroup logs once and disables the gate; a transient MemorySampleUnavailable keeps the last reading, an invalidates one (a runtime replaced under the probe) forgets it without disabling, and invalidate() forgets an inactive container's reading the same way; the guard exposes verdicts alone — no kill[unit] deploy/cloudflare-resident/memoryGuard.test.ts::*
70: publication and credential inspection also sample the exec gate; inspection returns only opaque incomplete counts on refusal. The wiring — the reader is one exec of CGROUP_READ_ARGV through run(), sampled only on an active runtime and persisted; the exec gate runs before the preflight and the attach gate between the drain and the image reconcile, both answering the mirror-busy 503 shape with reason: "memory-pressure"; a registered run's re-attach passes an ordinary drain but waits under a deploy fence, using the same registration read; an inactive runtime invalidates the reading before the gate asks; the measure tick samples beside the disk; /status, /residents and the watchdog answer carry the gauge from storage; no new timer or polling loop[unit] deploy/cloudflare-resident/memoryGate.test.ts::*
70: the log line, live — every sample lands in Workers Logs as memory: used <n> bytes of <n> bytes cap (<p>%) …, at exec/attach starts and once per measure tick[agent] wrangler tail switchboard-resident (or the Workers Logs dash) during a coding run on an onboarded repo: each /exec logs the line; ten minutes idle-free, the measure step logs it too, beside the disk: line.
69: POST /drain's body → the record — defaults, minutes in [1, 90] as an integer, trimmed words with caps — refused by name, never clamped; liveDrain reads an expired record, nothing, or a foreign shape as no drain; the refusal is a 503 whose error opens draining: and names the reason, the asker, the end and that the run waits, with the record beside it[unit] deploy/cloudflare-resident/drain.test.ts::parseDrainRequest — the record a drain asks for::*, ::liveDrain — the drain in force, or none::*, ::drainRefusal — what /attach answers while drained::*
69: the wiring — /drain, /deploy-fence, /reconcile and /undrain are POST routes of the drain scope (the drain bearer or admin; hasScope passes admin everywhere and every other scope only its own routes), the registry offers get/set/clear under a key outside the slot prefix, the Durable Object's attach consults the live drain after hydration and before the image reconcile and exempts only a live registration owned by the attaching run during an ordinary drain, /residents carries draining[unit] deploy/cloudflare-resident/drain.test.ts::the Worker's wiring (by scan)::*
69: upload admission — a durable registry permit starts before attach, bound-thread commands, detach and deadline extension, admin lifecycle work, each scheduled watchdog pass, and each refresh step; a fenced registry issues no permit, and the deploy quiet read refuses while any permit remains. The pre-upload Worker version cannot replace or lift the fenced drain or reconcile it; an unknown permit remains a refusal, never an inferred idle state[unit] deploy/cloudflare-resident/runRegistration.test.ts::deploy registration activity::holds a durable admission across already-bound work and rejects a quiet read with admitted work, deploy/cloudflare-resident/runRegistration.test.ts::deploy registration activity::admits scheduled refresh steps and watchdog lifecycle work before either can start after the quiet read, deploy/cloudflare-resident/preflight.test.mjs::resident deploy preflight — decide()::refuses retained work without a live fence, an executing owner, or unknown ownership
69: upload phase — the fence stores the invoked Worker version ID and fence timestamp from the version metadata binding. /reconcile acquires a permit before reading the drain; an old-version call holds that permit across resident work or is refused by the fence. These normal reconcile and undrain routes require a version whose ID differs and whose creation timestamp follows the fence, even when its build stamp matches. Manual fence release is a separate admin operation below. Missing metadata refuses.[unit] deploy/cloudflare-resident/drain.test.ts::deploy upload phase proof::*, deploy/cloudflare-resident/runRegistration.test.ts::deploy registration activity::admits reconcile before its drain read and distinguishes the deployed Worker version; resident npm run typecheck.
69: manual pre-upload fence release uses admin-only GET /deploy-fence-recovery and POST /release-deploy-fence. The administrator independently verifies the original controller is stopped and the native target is unchanged, supplies an HTTPS receipt and both explicit attestations, and selects the full stored-record hash. Fresh owner/lifecycle/process probes and a final registry transaction preserve changed records, membership, work obligations, maintenance holds and a different actual Worker version. The transaction stores the original fence and evidence hash in an audit, then removes only the swap fields. The drain, work records, files and native state remain. A lost commit reply permits only an exact audit read. Normal undrain and upload checks remain separate.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::manual deployment fence release::*; [gap] Controlled live operator release on a containing Worker build with independent controller and native receipts.
69: operator fence recovery emits bounded read, release and refusal events through the existing deployment event logger. Events identify the valid selected record hash, actual Worker version, explicit attestation flags, HTTP status and closed refusal class. An uncertain mutation emits its commit phase and a fixed storage/RPC failure class before the original audit read. Private request values, receipt URLs and exception text are excluded.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::manual deployment fence release::releases only the exact pre-upload fence and retains the drain and an independent audit, ::records closed refusal and uncertain-commit classes without private request or exception text
69: orphan permit recovery — GET /deploy-admissions keeps the compact admin inventory; GET /deploy-admissions?id=<id> quotes the exact original start and full flat-record SHA-256 read-only. POST /recover-quoted-admission requires id, expectedStartedAt, expectedRecordSha256 and an HTTPS link to an operator-verified external request-end receipt. It fences new work, independently reads every resident owner/lifecycle and pool-user process count, and compares the original record again inside the same-fence transaction. It atomically preserves the original record and receipt URL hash in a request-ended/operator-attested audit before removing only that permit. Neighbors, native outcomes and private obligations remain unchanged. Changed or corrupt records, prior audits and unknown or busy views refuse. Legacy HTTP and RPC recovery requests refuse deletion; quoted recovery uses a separate address and method with no fallback to older code. A lost mutation reply permits one exact audit read, never another recovery mutation; absence, changed or unreadable evidence stays unverified. Operator abandonment and unobserved request ending are unavailable; caller assertions, upload/version changes and process snapshots cannot substitute for a qualified containment producer. Recovery has no automatic timeout.[unit] deploy/cloudflare-resident/drain.test.ts::targeted deploy admission recovery::*; deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::*; deploy/cloudflare-resident/runRegistration.test.ts::deploy registration activity::recovers one admin-selected orphan only under a fresh fence and independent idle probes; resident npm run typecheck. [gap] Live request-ended recovery requires an operator-verified original request association/end receipt. Executable abandonment remains open until an independently qualified containment/retirement producer or separately reviewed bounded staging risk contract exists; source tests and deployment do not provide that witness.
69: the reconcile inside the drain window — /reconcile walks the registry and asks each resident's reconcileForDeploy; every resident holds the drain before writing its pending marker, so no report outruns its hold. A resident confirmed inactive reports current immediately on every reconcile because its next start uses the deployed image by construction — including when the marker was written after an earlier idle pass — and the report reaches the registry without starting the container. A failed activity probe proves neither state, so deploy and refresh answer deferred with the pending marker and drain hold intact for a later reconcile, while a new attach with that marker is refused stale. An active, quiet container is cycled without trusting the pool-user probe; neither its stop nor a probe of the fresh start reports (issue 2101), and its pending marker survives until the replacement's hydration reaches warm. A failing resident becomes its own error row and hold; a live run registration defers the cycle; an attach never stops an active container and only an owned, live registration permits re-attach. Deploy reconciliation enters the same admission fence as an idle pool recycle before its first await: one already holding the fence defers its image report, and a reconcile already in flight makes the recycle fail its idle proof. In the incident's two-idle, two-restarted fleet, the idle reports and hydrated replacement reports remove their own holds and the last report reopens the fleet before the backstop[unit] deploy/cloudflare-resident/drain.test.ts::the Worker's wiring (by scan)::the reconcile route is a drain-scope POST dispatched to its handler…, ::the Worker's wiring (by scan)::an idle DO marked pending after its reconcile still reports current on the next cycle, ::the Worker's wiring (by scan)::a failed activity probe preserves the pending marker and drain hold, and refuses a new attach, ::the Worker's wiring (by scan)::a report from a non-running container reaches the registry, ::the Worker's wiring (by scan)::the image reconcile defers while a run registration is live…, ::the Worker's wiring (by scan)::an attach never stops an active container, and its replacement reports only after hydration (issue 2101), ::the reopen gate — the fleet reopens on the last container's new-image report, never on the reconcile's return (issue 1931)::the two-idle two-restarted fleet reopens on its last report, before the backstop, deploy/cloudflare-resident/runRegistration.test.ts::the preflight-facing counts see registered runs the op counters miss::isIdle protects a registered run beyond an hour of inactivity but permits sleep after its bounded deadline; [gap] Exercise concurrent deploy reconciliation and idle pool recycle on an isolated VM, confirming the image hold persists until current-image restore.
69: image progress binds the exact timestamped Worker build and the authenticated caller's application, version and image target. A completed report from another artifact or target requires a fresh member decision. A different target cannot overwrite an earlier pending report, even across a Worker rebuild. Same-target retries preserve deferred cycles and replacement hydration. Missing or foreign targets refuse before admission; native, owner and private-work guards still apply.[unit] deploy/cloudflare-resident/imageReconcileState.test.ts::resident image target decisions::*; deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::refuses missing, malformed and foreign image targets before reconcile admission
69 (issue 2044): the hold's liveness — the first hold stamps the cycle bound clamped to until and later holds keep it, a held record past the bound is no drain (liftAsked or not), staleHolds names the reopen's stale containers and nothing otherwise, a pre-bound record keeps until as its only end, and the incident's replay (drain, one held container that never reports, the lift asked, the bound elapsing) ends in a reopened fleet with the container named[unit] deploy/cloudflare-resident/drain.test.ts::the hold's liveness — a cycle that never happens reopens the fleet with the container named, never a silence to the record's own until (issue 2044)::*
69 (issue 2044): the wiring — holdDrainFor stamps the bound and arms the registry alarm at it, the alarm's reopen warns naming the stale containers, a fresh provision reports its own container current and clears the marker, /op answers a drained fleet with op-refused: … drained … before any work, and the admin view carries imageReport[unit] deploy/cloudflare-resident/drain.test.ts::the Worker's wiring (by scan)::the hold's liveness is wired (issue 2044)…
69 (issue 2044): the waits are legible — the drain wait paints waiting for the deploy to finish · N min through onSetupNote on each poll and clears it at the wait's end, the binding carries drainWaitMs beside wokeAfterMs from the first draining answer and from the wake path's drain hand-off alike, the dispatcher publishes it as the drain_wait run note with the rounded minutes, and repo list leads with the drain and marks every row instead of printing a bare warm[unit] src/execution/resident.test.ts::ResidentExecutor.attach during a fleet drain (item 69)::the wait says so on the card (issue 2044)…, ::ResidentExecutor.attach during a fleet drain (item 69)::a drained fleet is waited for…, ::ResidentExecutor.attach during a fleet drain (item 69)::a Durable Object reset that lands during a live drain…, src/core/dispatcher.test.ts::executor provisioning by agent resources::a binding that carries drainWaitMs publishes the run's drain_wait note…, src/core/commands/repo.test.ts::repo.list::issue 2044: a drained fleet leads the reply…
69 (issue 2101): the drain wait is bounded by the fallback's cost and visible — a first attach with a fallback behind it passes drainBoundMs; the first drain turns it into one absolute deadline carried through every drain/wake hand-off, so repeated deploy resets cannot renew the allowance, and the wait ends in the typed error after DRAIN_FALLBACK_WAIT_MS, never the deploy's whole length, the error carrying the full drain share; the factory's fallback selection carries drainWaitMs and the dispatcher publishes the drain_wait note from it; the drain wait and the attach's wake wait are each one child span under the attach span, ending ok when the fleet reopens and failing with the typed classification from every stop point, including the wake's initial probe, never the refusal's free text; both span names have display names[unit] src/execution/resident.test.ts::ResidentExecutor.attach during a fleet drain (item 69)::a fallback bounds the drain wait by its own cost (issue 2101)…, ::ResidentExecutor.attach during a fleet drain (item 69)::one fallback deadline survives drain/wake hand-offs…, ::ResidentExecutor.attach during a fleet drain (item 69)::the drain wait is one span under the attach span (issue 2101)…, ::ResidentExecutor.attach during a fleet drain (item 69)::a drain wait that runs out fails its span with the typed classification and no remote words (issue 2101), ::ResidentExecutor.attach during a fleet drain (item 69)::the wake wait after a transient refusal is its own span under the attach span (issue 2101)…, ::ResidentExecutor.attach during a fleet drain (item 69)::a stop during the wake wait's initial status probe closes the wake span as failed, src/execution/factory.test.ts::makeExecutor resident selection::a drained fleet's first attach falls to the sandbox after the fallback's own bound…, src/core/dispatcher.test.ts::executor provisioning by agent resources::a selection that carries drainWaitMs without a binding…, src/core/trace/displayNames.test.ts::display names::the attach's drain and wake waits have display names under the attach prefix
69: the client — a draining answer is keyed on the record, not the words; a drained fleet is waited for with one re-attach per poll and the binding carries the wait; the wait is the run's lease less the reserve and ends in the typed ResidentDrainingError naming the wait and the drain's end; an answer that is no longer the drain ends the wait as the attach's own refusal; a Durable Object reset met during a live drain enters the wake wait but its re-attach's draining answer hands the wait to the drain wait — the lease's budget and the typed error, never the wake budget's strike and a cold fallback; the run's stop ends it at once[unit] src/execution/resident.test.ts::ResidentExecutor.attach during a fleet drain (item 69)::*
69: live — a deploy under running work: the resident step logs fleet drained … until …, a run asked during the drain shows attaching the workspace… until the deploy lands and then a note after waiting Ns for the resident, the runs that were in flight finish, the step deploys once /residents reads zero in flight, and fleet reopened closes it[agent] On a release that lands while a coding child is live: read the deploy job's resident lines and the card of a review asked mid-drain.
65: the refusals that name a rolling container (the incident's The container just exited, Container is starting, the runtime-replacement wordings, attach's image-stale:) are recognized, and none of the refusals other rules own[unit] src/execution/residentWake.test.ts::isContainerRolling: the refusals that name a container gone for a moment::*
65: the wake decision over the engine view: wait on restoring, the serviceable states and the transient degraded reasons; strike on down, onboarding, a repo failure, not onboarded, an off-table state, or an unreachable Worker, naming it[unit] src/execution/residentWake.test.ts::wakeDecision: the engine view says whether the container is coming back::*
65: the wait budget is the command's budget under the three-minute ceiling, the ceiling with no budget, never negative[unit] src/execution/residentWake.test.ts::wakeWaitBudget: the wait is the command's budget under a three-minute ceiling::*
65: the model is told the wait, the fresh ref@sha, and that uncommitted and unpushed work is gone[unit] src/execution/residentWake.test.ts::sandboxRestartedMessage: what the model is told about the tree it comes back to::*
65: a container exit during a live restore waits for warm, re-attaches and hands /exec back as ExecSandboxRestartedError with the wait and the binding, never infra; the /status probe names the resource[unit] src/execution/resident.test.ts::ResidentExecutor waits for the wake (item 65: …)::a container exit during a live restore waits for warm, re-attaches, and hands /exec back as a restart (ExecSandboxRestartedError); never infra
65: a container exit with a definite non-recovering engine view is infra at once, on every call; an unreachable /status is infra; a refusal that is not a rolling container never probes[unit] src/execution/resident.test.ts::ResidentExecutor waits for the wake (item 65: …)::a container exit with a definite non-recovering engine view is infra at once, on every call, ::an unreachable /status is a strike too: nothing says the container is coming back, ::a refusal that does not name a rolling container keeps the old rule: one strike, no probe
65: the wait respects the command's budget and the three-minute ceiling (a wake that never comes is the strike when the budget is spent, naming what was last seen), ends with the strike when the resident goes down mid-wait, and a hard stop ends it at once with the stop's one typed error (aborted, classified as the transport)[unit] src/execution/resident.test.ts::ResidentExecutor waits for the wake (item 65: …)::the wait respects the command's budget: a wake that never comes is a strike when the budget is spent, not at the ceiling, ::the wait is capped at the three-minute ceiling whatever the command's budget, ::the resident going down mid-wait ends the wait with the strike naming it, ::a hard stop during the pause ends the wait at once with the stop's one typed shape…
65: a re-attach refused image-stale mid-wait keeps waiting and the next one binds; any other attach refusal is the attach's own legible error; the idempotent routes re-issue after the wake[unit] src/execution/resident.test.ts::ResidentExecutor waits for the wake (item 65: …)::a re-attach refused while the container is still rolling (image-stale) keeps waiting; the next one binds, ::any other attach refusal mid-wait is the attach's own legible error (here: not onboarded), not a strike, ::the idempotent routes re-issue after the wake: /read waits, re-attaches and answers the content
65 live (human-gated): a coding run admitted inside the container rollout that follows a resident Worker deploy finishes. Its card shows the sandbox_restarted note with the wait, the model re-checks the tree and goes on, and runs list shows it completed with a PR or an answer, never "Aborting instead of retrying into a dead sandbox"[human] Deploy the resident Worker (npm run deploy in deploy/cloudflare-resident), then within three minutes post a coding ask on an onboarded repo; watch the card and GET /status of that resident through the rollout.
66: reuse parses absent→false, boolean→itself, anything else→named 400[unit] src/execution/residentReuse.test.ts::parseReuse…::*
66: a reusing attach keeps a dirty tree and a stale one and refuses a missing, unreadable, other-mode or refChanged tree by name; a provisioning attach keeps item 17's discipline (ref change, mode switch, missing, unreadable, dirty and stale recreate; clean at the tip or a descendant reuses)[unit] src/execution/residentReuse.test.ts::decideWorktree: a reuse-only attach keeps the tree as it stands::a refChanged tree is refused even when it is readable at the same commit, src/execution/residentReuse.test.ts::decideWorktree: a fresh attach…::*
66: the client sends reuse: true only when asked to re-attach, types the 409 needs recreate as ResidentReuseRefusedError, and records the answer's container on the binding[unit] src/execution/resident.test.ts::ResidentExecutor.open (attach-on-open)::sends reuse:true in the attach body…, ::a 409 needs:"recreate"…, ::records the container identity the attach answered…
66: the factory's re-attach sends reuse: true and answers the resident executor with the container on its binding; a fresh run sends no reuse; a refusal, an unreachable or unserviceable resident, a mirror-busy 503 or an answer naming another tree or user is a WorkspaceReattachRefusedError and no sandbox is provisioned; a recorded sandbox binding re-makes the per-thread executor with zero resident calls[unit] src/execution/factory.test.ts::makeExecutor resident selection::on a resume (ctx.reattach): re-attach, never re-provision::*
66: Worker wiring against the pure decision: handleAttach parses reuse, the flag reaches ensureThreadWorktree, the one rm -rf follows the decision, a refusal is a 409 needs recreate after the rollback, the answer carries the memoized boot id[unit] deploy/cloudflare-resident/reuseAttach.test.ts::*; resident npm run typecheck
66 live (human-gated): a coding run mid-edit on a resident survives a deploy restart with its edit intact: the resident log shows the resume's attach with no worktree-clean step and recreated: false, the run's record a resumed note and no cold_sandbox note, and git status in the worktree still lists the run's change; a run whose tree is removed under it (as root, rm -rf the thread's worktree between the drain and the reclaim) shows the 409 reuse-refused in the resident log and a resumed note saying the run restarts from its request[agent] after deploy, per run-history.md item 54's live row.
16 (named ref authority): a ref the ask names directly (refHint present without refByDefault or an ownPr-derived hint) wins over any sticky binding and is persisted as the binding's named authority even when the ref is unchanged; only an absent hint or the by-default hint may retain the sticky ref. The bot client fails closed when a named attach answers another ref. A changed ref forces the old tree to be recreated even when both refs point at the same commit; a reusing attach refuses rather than serve the other ref's tree. The replacement row remains provisional until every attach step succeeds: a disk-admission throw restores an evicted prior binding before it escapes, and a failure after recreation discards the provisional checkout, leaving the old binding and checkout together.[unit] src/execution/resident.test.ts::ResidentExecutor.attach over a heartbeat stream (item 59: an attach that waits on an install must not lose the connection)::a re-issued child after an aborted attempt rejects the previous sticky ref, while an unnamed ask may keep it, src/execution/resident.test.ts::ResidentExecutor.attach over a heartbeat stream (item 59: an attach that waits on an install must not lose the connection)::rejects a stale returned record for a mismatched named ref, while an own-PR-derived return remains valid, deploy/cloudflare-resident/rebindAttach.test.ts::a named ref replaces the sticky fallback before the worktree is provisioned::*, deploy/cloudflare-resident/rebindAttach.test.ts::a named ref replaces the sticky fallback before the worktree is provisioned::a disk-admission throw restores an evicted prior binding instead of publishing the failed replacement, deploy/cloudflare-resident/rebindAttach.test.ts::a named ref replaces the sticky fallback before the worktree is provisioned::an explicit attach naming the already default-bound ref records named authority without replacing the checkout, so an own-PR hint cannot move it, src/execution/residentReuse.test.ts::decideWorktree: a fresh attach keeps the dirty/stale discipline::a named ref replacing the sticky ref recreates even when both refs point at the same commit, src/execution/residentReuse.test.ts::a named-ref replacement is provisional until the attach succeeds::an attach that fails after recreating discards the new checkout and leaves the old binding on its matching checkout; resident npm run typecheck
16 (named ref authority) live: re-issue agent:ship after a coding attempt dies; the next coding child's card, the pipeline row and the pull request head all name the new attempt's ref, never the dead attempt's resident-bound ref.[agent] runnable when worker:bot is at or beyond this change's merge sha; re-issue a died ship attempt and compare the child's card, pipeline row and PR head.
16 (the thread's own PR): the pure plan — a default-bound thread with a live tree is measured; a ref a person named (boundBy: name, or a binding made before the field whose ref is not the default) is never moved by the legacy own-PR plan; a rebound thread never moves again; an evicted binding is branch-absent; a resumed run's attach, no hint, no binding yet, or a binding already on the branch move nothing; a new binding records default only for the by-default attach of the default branch, and a pre-field binding reads as default iff its ref is the default[unit] src/execution/residentRebind.test.ts::rebindPlan: whether the binding may move, read off the binding alone::*, src/execution/residentRebind.test.ts::boundBy: how a binding's ref was chosen::*
16 (the thread's own PR): the branch must be the thread's own — the memory of a release decides by itself, tree or no tree; without it, a local branch of the surviving tree; a branch the tree never made, a missing or an unreadable tree is branch-absent; the tree's dirt and HEAD are never judged; ownPr and refByDefault parse absent → none/false and a malformed value → a named 400[unit] src/execution/residentRebind.test.ts::rebindVerdict: without a memory of the branch, the surviving tree decides a measured plan::*, src/execution/residentRebind.test.ts::rebindPlan and rebindVerdict when the thread's tree is gone: the remembered branch decides::a live binding's plan carries whether the branch is remembered…, src/execution/residentRebind.test.ts::parseOwnPr: the…::*, src/execution/residentRebind.test.ts::parseRefByDefault: the…::*
16 (the thread's own PR): Worker wiring against the pure decision — handleAttach parses ownPr (its ref through the one ref pattern) and refByDefault; the plan runs before the ref is chosen; under the mirror mutex the row is re-read and re-judged, a branch nothing remembered is looked for in the surviving tree with one probe as the thread user, and every allowed plan goes through the mirror check (moveOntoOwnBranch) with a token minted before the mutex; the rebind runs no status, no HEAD probe, no checkout, no fetch or reset in the tree, no clone, no rm -rf, no new path, no new pool user; the binding records boundBy and rebound; the answer carries rebound or rebindRefused[unit] deploy/cloudflare-resident/rebindAttach.test.ts::rebindToOwnPr moves the binding, or names why it stands — a decision about the binding alone::*, deploy/cloudflare-resident/rebindAttach.test.ts::moveOntoOwnBranch: the mirror must hold the branch, then the row moves and the attach provisions the tree::*, deploy/cloudflare-resident/rebindAttach.test.ts::the binding and the attach answer carry the record::*; resident npm run typecheck
16 and 17 (the tree after a move): the attach after a rebind provisions the tree at the moved ref as it provisions any tree — the refChanged fact reaches the pure decideWorktree, so the one rm -rf and clone run even when the old and new refs have the same tip; a dirty or stale tree is recreated with no exception at the binding's stored path[unit] deploy/cloudflare-resident/rebindAttach.test.ts::the attach after a rebind provisions the tree at the moved ref as it provisions any tree::*, src/execution/residentReuse.test.ts::decideWorktree: a fresh attach keeps the dirty/stale discipline::*
16 (the thread's own PR): the client sends ownPr and refByDefault only when set and records a well-formed rebound / rebindRefused / returned off the answer; the factory sends ctx.ownPr beside the hint and the head, says refByDefault: true on the by-default re-attach, never sends ownPr on a resume, and the card reads · rebound to <ref> (this thread's PR #N), · returned to <default> (the branch of this thread's PR #N is gone) — · returned to <default> (this thread's branch <branch> is gone) when the answer's returned names no pull request, read as the move back all the same, while a malformed number reads as no move — or · rebind to <ref> (this thread's PR #N) refused: <reason> beside the binding line[unit] src/execution/resident.test.ts::ResidentExecutor.open (attach-on-open)::sends ownPr and refByDefault in the attach body only when set, ::records the answer's rebound or rebindRefused on the binding…, src/execution/factory.test.ts::makeExecutor resident selection::the thread's own pull request (ctx.ownPr) and the resident's answer to it::*, src/execution/factory.test.ts::makeExecutor resident selection::needs-ref WITH the resident's defaultRef…, src/execution/factory.test.ts::makeExecutor resident selection::on a resume (ctx.reattach): re-attach, never re-provision::a resume never asks the resident to move the tree…
16 (the thread's own PR): end to end — a follow-up in a thread whose newest finished run opened a pull request resolves the record's PR (prFromRecord, ownPrOf), attaches with refHint = its head branch, sha = its head commit and ownPr, and the card says the resident rebound the thread; a refused rebind is said on the card and as a rebind_refused note on the run's stream (head material), and the run goes on where the binding is[unit] src/core/repoContext.test.ts::resolveRepoContext: the pull request the thread's own run opened binds the follow-up (resident-repos item 29)::ownPrOf: the record's PR…, src/core/dispatcher.test.ts::repo/ref resolution + resident prompt selection::a follow-up in a thread whose newest finished run opened a pull request attaches with that PR as ownPr…, ::a rebind the resident refused is said on the card and on the run's stream…
16 (the thread's own PR) live (human-gated): in a thread whose coding run bound the repo default, created a branch, pushed it and opened a pull request, a directive-less follow-up's card reads … · <branch>@<sha7> · rebound to <branch> (this thread's PR #N); the resident log shows rebound <default> → <branch> …; the tree is provisioned at it; /debug threads shows the binding with ref: <branch>, rebound set and the same worktreePath as before (the same pool user when the tree survived the run's end, a fresh one when the release evicted it); the attach's trace shows worktree-clone unless the tree was already clean at the branch's tip; a push from the follow-up lands on the PR; a second follow-up shows no second move[agent] after deploy: run the coding flow above on an onboarded repo, then the follow-up; read the card, the resident log and POST /debug {"op":"threads"}
29 (and 16's ref hint): only an OPEN pull request named in the message binds the ref (refFromPr) — a merged or closed one contributes no ref hint while its facts still ride as context and a phrase's ref stands — and a coordinator child's attach ref hint is its contract's branch, always: a PR-derived ref off the child's own request text (and the head sha that pinned it) is dropped before the attach, so the push guard allows the unit branch (issue 1860)[unit] src/core/repoContext.test.ts::resolveRepoContext: PR-source flags for ship::a merged PR URL in a plain ask yields no ref hint — the PR's facts (pr, headSha) still ride as context (issue 1860), ::a closed-unmerged PR contributes no ref hint either, and an explicit … beside a merged PR binds the phrase's ref with refFromPr unset (a contract child attaches on its contract branch); src/core/dispatcher.test.ts::coding PR post-step (docs/reference/specs/pr-description.md)::a coordinator's contract child whose request text cites a PR attaches on the contract branch — the PR-derived ref never rebinds the attach, so the push guard allows the unit branch (issue 1860)
29/51 (the merged half): a message-cited PR that did not bind the ref (prFromMessage without refFromPr) contributes no attach sha for a non-review run — the dispatcher drops headSha before the attach, so a merged PR's frozen head never becomes item 51's wantSha (stale-tip → a cold run at the dead commit): the coordinator child attaches at its contract branch's tip, the plain ask at its bound ref's tip (issue 1860)[unit] src/core/dispatcher.test.ts::coding PR post-step (docs/reference/specs/pr-description.md)::a coordinator's contract child whose request text cites a MERGED PR attaches on the contract branch with NO expected commit — the merged PR's frozen headSha never rides the attach (issue 1860), ::a plain coding ask citing a merged PR attaches with no expected commit — the frozen headSha is context, never the attach's wantSha (issue 1860)
16a (the release's word): pushed parses absent → nothing, a list of {ref, pr} → itself, a malformed entry or more than the cap → a named 400; remembering appends and moves a re-pushed branch to the end with its current PR and time; exceeding the cap refuses before detach or eviction, retaining all prior facts instead of truncating them[unit] src/execution/residentRebind.test.ts::parsePushed and rememberOwnBranches…::*
16a (the release's word): Worker wiring — handleDetach parses pushed and checks every ref through the one ref pattern; detachThread checks registration ownership before remembering branches or evicting a live binding; the binding declares ownBranches and a re-allocation after eviction carries it[unit] deploy/cloudflare-resident/rebindAttach.test.ts::the owning run's pushed branches survive release::*; resident npm run typecheck
16a (the release's word): the bot releases with the saved producer branch mapping, independently of display events. An accepted push without a verified mapping holds ordinary release; metadata edits never prove a push. The client sends nonempty mappings only.[unit] src/core/codingPrPostStep.test.ts::runCodingPrPostStep (callable with explicit inputs)::no description + accepted push + an existing PR lookup → note names changed code without claiming a metadata edit, src/core/runRecord.test.ts::the pull request on the record (docs/reference/specs/run-history.md item 2)::the pull request carries the head branch…, src/core/reviewRound.test.ts::attachRoundWorkspace (explicit AgentDef → attach + paired release)::passes the ledger claim, run and generation through attach and release, src/core/runLedger/inMemory.test.ts::resident attachment claims::mints increasing fences only for the current ledger owner, including same-second reclaims, src/execution/resident.test.ts::ResidentExecutor.release — return the thread's pool user when a run ends::sends the run's pushed branches…, src/core/dispatcher.test.ts::coding PR post-step (docs/reference/specs/pr-description.md)::a coding run that opened a pull request releases its workspace with…
16 (the thread's own PR, tree gone): an evicted default-bound thread whose own run pushed the branch plans a recreate; one whose runs never pushed it is branch-absent naming why; the memory never overrides the named-ref, already-rebound and resume guards; a live binding's plan says whether the branch is remembered, and the memory decides by itself — a tree that turns out missing, or that never made the branch, moves all the same — while a live tree whose binding remembers nothing falls back to its local branch[unit] src/execution/residentRebind.test.ts::rebindPlan and rebindVerdict when the thread's tree is gone…::*
16 (the thread's own PR, tree gone): Worker wiring — an evicted binding's recreate plan and a live binding whose branch is remembered both skip the local-branch probe and go through the mirror check; the mirror is fetched when it lacks the branch (a token minted before the mutex) and a branch still missing after the fetch is a branch-absent refusal, the binding kept; the move is recorded on the row and nothing on disk is touched — the attach clones the tree at the new ref[unit] deploy/cloudflare-resident/rebindAttach.test.ts::rebindToOwnPr moves the binding, or names why it stands — a decision about the binding alone::every allowed plan…, deploy/cloudflare-resident/rebindAttach.test.ts::moveOntoOwnBranch: the mirror must hold the branch, then the row moves and the attach provisions the tree::*
16 (the thread's own PR, tree gone) live (human-gated): in a thread whose coding run bound the repo default, created a branch, pushed it, opened a pull request and had its clean tree released at the run's end, a directive-less follow-up's card reads … · <branch>@<sha7> · rebound to <branch> (this thread's PR #N); the resident log shows rebound <default> → <branch>; /debug threads shows the binding with ref: <branch>, rebound set, ownBranches naming the branch and its PR, the same worktreePath and a fresh pool user; the attach's trace shows the tree cloned at the branch (worktree-clone), and a push from the follow-up lands on the PR. Negative control: a thread whose run pushed nothing keeps … · <default>@<sha7> · rebind to <branch> (this thread's PR #N) refused: branch-absent[agent] after deploy: run the coding flow above on an onboarded repo, wait for the run's release (/debug threads shows evicted: true with ownBranches), then the follow-up; read the card, the resident log and POST /debug {"op":"threads"}
16 and 17 (a dirty tree at the run's end) live (human-gated): in a thread whose coding run bound the repo default, created a branch in its tree, pushed it, opened a pull request and left an uncommitted tracked edit, the run's card label ends … 1 uncommitted change(s) and 0 unpushed commit(s) left behind — discarded at the run's end, its record carries a work_left_behind note, the release evicts the tree (/debug threads shows evicted: true with ownBranches naming the branch; the bot log's [release] line and wrangler tail switchboard-resident name the counts), and a directive-less follow-up's card reads … · <branch>@<sha7> · rebound to <branch> (this thread's PR #N) with worktree-clone in its trace and an empty git status --short in its workspace — the run starts clean at the branch. Negative control: the same thread whose release was refused busy (a command still running at the run's end) keeps the dirty tree; the follow-up still reads rebound to <branch>, its trace shows worktree-clean then worktree-clone, and its git status --short is empty[agent] after deploy: run the coding flow with a prompt that ends by editing a tracked file without committing, read the card and the run page, confirm /debug threads, then the follow-up; read its card, trace and workspace
16 (the second movement): a rebound, default-bound binding still on the branch its move named may return, and so may a binding on a branch this thread itself pushed (ownBranches) whatever boundBy says — bound by name, by default, or made before the field; a ref a person named that the thread never pushed, a binding never moved, one already returned, one on the default, or one no longer on the moved branch may not, and returnToDefault answers undefined for them; the return puts the ref back on the default, makes the binding read as bound by default, records returned on it (the pull request the rebind's, else the remembered branch's), stamps a rebind's rebound.returnedAt, answers the move back, and leaves the thread free to follow its next pull request (rebindPlan measures again)[unit] src/execution/residentRebind.test.ts::canReturnToDefault and returnToDefault: a binding whose own branch is gone goes back to the default::*, src/execution/residentRebind.test.ts::rebindPlan: whether the binding may move, read off the binding alone::a thread already rebound is never moved again — until its move is returned…
16 (the second movement): Worker wiring — the create step reads canReturnToDefault once off the binding before the mutex and hands it to the fetch decision (mirrorFetchReasonFor, the mint pre-check and the fetch under the lock alike), so a returnable ref the mirror still holds is fetched for and its existence re-read for real before the mirror is trusted, a verification fetch that fails being one log line and the mirror's ref while a fetch for a missing or stale ref fails as before; it asks the same fact only on an unknown-ref target after the fetch, re-reads the row under the mutex and hands it whole — rebound and ownBranches alike, no Worker-side narrowing to a rebound row — to the pure returnToDefault, whose undefined (a person-named ref the thread never pushed, or a row another attach moved meanwhile) is answered as the row stands with nothing written; its row is written, the default re-targeted and the tree provisioned there through the same worktree step at the same path; a binding that may not return still throws unknown-ref; the row's final write and the answer speak of the returned binding, the answer carrying returned; the binding declares returned beside rebound and a re-allocation after eviction carries it[unit] deploy/cloudflare-resident/rebindAttach.test.ts::a binding whose own branch is gone from the mirror returns to the default, and the run starts clean there::*; resident npm run typecheck
16 (the second movement, its timing): over the pure chain the create step is wired from — mirrorFetchReason, the fetch's effect on the mirror, the ref re-read, attachTarget, canReturnToDefault, returnToDefault — a rebound thread, or one bound by name to a branch it pushed, whose branch was deleted at the origin while the mirror still holds it is fetched for and returns to the default at that attach; a returnable ref still at the origin is fetched for and kept; a default-bound thread attaches without a fetch; a ref a person named that the thread never pushed is trusted from the mirror without a fetch and, once the cycle's prune removes it, is unknown-ref, never swapped for the default[unit] src/execution/residentRebind.test.ts::the second movement is decided at the first attach after the deletion, never at the next refresh cycle::*
16 (the second movement) live (human-gated): in a thread rebound onto its own pull request's branch, merge the pull request and delete the branch; a directive-less follow-up within a minute of the deletion — before the refresh cycle's prune could have removed the branch from the mirror — has a card that reads … · <default>@<sha7> · returned to <default> (the branch of this thread's PR #N is gone), never <branch>@<sha7> at the deleted branch's stale tip; the resident log shows <branch> is gone from the mirror … — returned to <default>; the tree is provisioned there, /debug threads shows ref: <default> with rebound.returnedAt and returned set, and the run works on the default, never falling to a cold sandbox; a description-only follow-up there edits the closed pull request's body in place, its reply saying the pull request is closed with no word of a push and no compare URL (pr-description.md item 5). Negative control: a thread a person bound to a branch (on branch <x>) that no run of the thread pushed, whose branch is then deleted, still fails its next attach unknown-ref and falls cold, the card saying so[agent] after deploy: run the coding flow, merge its pull request with branch deletion, then the follow-up within a minute of the deletion; read the card, the resident log and POST /debug {"op":"threads"}
16 (the second movement, a branch bound by name) live (human-gated): in a thread whose ship request ran a one-unit plan — its coding child bound the unit branch by name (plan/<id>/u1), pushed it and opened the pull request, the release recording the branch in ownBranches — merge the pull request with branch deletion; a follow-up agent:coding in that thread within a minute of the deletion attaches on the warm resident and its card reads … · <default>@<sha7> · returned to <default> (the branch of this thread's PR #N is gone), the resident log shows plan/<id>/u1 is gone from the mirror … — returned to <default>; the tree is provisioned there, /debug threads shows ref: <default>, boundBy: default and returned set with no rebound, and the run works on the default, never falling to a cold sandbox; a later own pull request from that thread then moves it as a rebind. Negative control: the same as the row above[agent] after deploy: in a thread on an onboarded repo, ship a one-line change (agent:ship in <owner>/<repo>: <task>), wait for the unit's pull request, merge it and delete its branch, post a follow-up with agent:coding within a minute of the deletion; read the card, the resident log and POST /debug {"op":"threads"}
16a: Detach refuses an operation in flight and uses item 71 before removing the binding or checkout; a blocked release names the reason and does not claim the work was saved[unit] deploy/cloudflare-resident/pausedWorkspace.test.ts::paused unpublished work at resident removal::*; deploy/cloudflare-resident/releaseAtRunEnd.test.ts::detachThread releases the tree whatever it holds, and names what it discards::*
17 (every eviction records what the tree held): evictedTreeOf — the counts when the probes found something, unmeasured naming the probe's failure when git could not read the tree, nothing for a clean or missing tree; evictedTreeSentence names each; the Worker's measureTreeBeforeEviction is the one reader of the probe for evictions, evictBinding stamps evictedLeftBehind / evictedUnmeasured beside evictedWhy and logs one line, and the live view (/residents live.threads, /debug threads) carries both fields[unit] src/execution/residentCleanliness.test.ts::evictedTreeOf and evictedTreeSentence: what an eviction records about the tree it removed::*, deploy/cloudflare-resident/dirtNeverKeeps.test.ts::every eviction records what the tree held::*
16a and 17 (the loss is never silent): the run loop reads the tracked changes and unpushed commits the run leaves in the same observation as the head — a failed probe leaves the count unset, an empty status is zero — and a coding run that left something gets a work_left_behind note on its record before the stream finishes and the loss on its card label, while a run that left nothing gets neither; the release mode pairs if-idle to a writing identity, always to a read identity or a hard stop[unit] src/core/codingPrPostStep.test.ts::observeCodingWorkspace::counts the tracked changes and the unpushed commits the run leaves…, src/core/codingPrPostStep.test.ts::workLeftBehindOf, workLeftBehindSummary and workLeftBehindLabel::*, src/core/dispatcher.test.ts::coding PR post-step (docs/reference/specs/pr-description.md)::a coding run that leaves uncommitted changes or unpushed commits gets a…, src/core/reviewRound.test.ts::releaseModeFor (per-agent attach/release pairing)::*
64: the signal — the production AbortError by its name, or the DOMException's message when a wrapper copied only that; a command's own abort, a timeout, a replacement, a crash and a non-error are not; the signal and the replacement family are disjoint[unit] src/execution/residentRefresh.test.ts::runtime-unreachable (a container whose control port never answers is named, never a command failure)::the production AbortError is the signal — by its name, and by the DOMException's message when a wrapper copied only that, ::nothing else is: a command's own abort, a timeout, a replacement, a crash, a non-error, ::the signal and the replacement family are disjoint — the Worker asks about a replacement first, and a replaced runtime is never counted as unreachable
64: the ladder — attempts 1–2 re-arm, 3 stops, 4–5 recreate, ≥ 6 down, every boundary; a count under one is a caller bug[unit] ::the ladder: attempts 1–2 re-arm, 3 stops the container, 4–5 destroy it and restore from the snapshot, 6 and beyond go down, ::a rung needs a count of at least one — zero and negative counts are a caller bug, not a rung
64: the reason names the port, the SDK's 30 s and the attempt out of six — never the bare The operation was aborted; each rung appends what it did; no rung's reason is a serviceable degraded reason[unit] ::the reason names the port, the SDK's 30 s and the attempt out of the down threshold — never the bare \The operation was aborted`, ::each rung appends what it did, so the state reason says what happened and what the retry does next, ::is never a serviceable degraded reason — the bot goes cold without attaching to a runtime that cannot answer`
64: classifyRefreshFailure with the caller's count → the runtimeUnreachable outcome with the named reason, whatever the step and message, over any disk probe; without the caller's word the classifier does not guess from the wording; a timeout wording stays the step's own failure and a timed-out restore stays down[unit] ::classifyRefreshFailure with the caller's count → the \runtime-unreachable` outcome with the named reason, whatever the step and message, ::the caller's count wins over a disk probe: no probe is consulted and the disk is never named, ::without the caller's word the classifier does not guess from the wording — the message stays the step's own failure, as it did in production, ::a timeout wording stays the step's own failure and a timed-out restore stays down — the ladder never widens those`
64: the 30 s is the installed SDK's DEFAULT_CONNECT_TIMEOUT_MS; the runtime identity key is its RUNTIME_RECORD_KEY, which its own stop and destroy delete, and a destroy without a stored identity skips the runtime cleanup; WAKE_PORT_READY_MS is 3 min inside the SDK's bounds for SANDBOX_PORT_TIMEOUT_MS and above its default — an SDK bump that moves any of them fails here[unit] ::the 30 s is the INSTALLED SDK's connect timeout (\DEFAULT_CONNECT_TIMEOUT_MS`), not ours — an SDK bump that moves it fails here, ::the SDK's runtime identity lives under the INSTALLED SDK's RUNTIME_RECORD_KEY, deleted by its own stop and destroy — the resident's explicit forget names the same key, ::the wake budget: the first connect after a start waits WAKE_PORT_READY_MS (3 min) for the control port through the SDK's env knob, inside the SDK's bounds and above its default`
64: the Worker counts in storage under one key with one writer and one eraser; ordinary run() asks about a replacement first, counts the unreachable signal off the cause chain and clears on a spawned process; explicit manual workload inspection preserves that counter as specified above; the classifier and the step route the typed error before any disk probe; the housekeeping steps skip a resident whose row exists[unit] deploy/cloudflare-resident/runtimeUnreachable.test.ts::the unreachable probe is named and counted at the exec choke point::*
64: the ladder in the Worker — the rung from the pure ladder over the persisted count, the reason recorded and logged; re-arm degrades and throws; stop swaps, signals, degrades and throws; recreate forgets the identity, destroys, keeps every snapshot and throws; down destroys and goes down with a strike-eligible reason; no rung's reason counts toward the park streak[unit] deploy/cloudflare-resident/runtimeUnreachable.test.ts::the ladder — each rung acts once, logs once and hands the step back to the engine::*
64: recreate-container is admin-scoped by construction (absent from READ_DEBUG_OPS), dispatched and listed; it refuses mid-flight and down by name, destroys through recreateContainer and starts the restore; stop-container is unchanged; /debug info answers runtimeUnreachable with its rung[unit] deploy/cloudflare-resident/runtimeUnreachable.test.ts::the admin op and the read view::*
64 / 34 / 11: rebuild forgets the identity and destroys before it reprovisions; teardown deletes every key and never deleteAll()s (the SDK's table survives); initResident rolls the state row back when arming fails[unit] deploy/cloudflare-resident/runtimeUnreachable.test.ts::the escape hatches start on a fresh container and leave the object able to serve again::*
64: the template's SANDBOX_PORT_TIMEOUT_MS equals WAKE_PORT_READY_MS[unit] deploy/cloudflare-resident/runtimeUnreachable.test.ts::the wake budget — the first connect after a start waits for the control port through the SDK's knob::*
64 live (recreate-container): on a WARM resident, POST /debug {"op":"recreate-container","resource":…} answers 202 {recreated:true, restoreStartedAt}; /debug info then shows restoring (reason rehydrating) and, within a few minutes, warm with lastRestore.at after restoreStartedAt, the same snapshot.mirrorBackupId/checkoutBackupId as before, runtimeUnreachable: null; the tail carries runtime-unreachable: destroying the container — snapshots kept then the restore's mirror restore: … in … s / checkout restore: … lines[agent] after the release deploys, on a fixture resident (or a production one at a quiet hour: the restore takes it out of warm for the restore's duration); the ladder itself is not reproducible live — nothing wedges port 3000 from outside a healthy container — so its receipts are the unit rows above and, at the next occurrence, the log lines runtime-unreachable: the control port did not answer within 30 s — attempt N of 6 and … rung <r> at attempt N of 6 climbing on /debug info
64 live (the reset of last resort): POST /offboard then POST /onboard for the same resource, back to back from the same isolate, answers 200 then 200 {state:"onboarding"} — never onboard failed arming the resident: no such table: container_schedules — and /debug info shows schedules.provisionRun: 1 right after[agent] on a fixture resident (repo:jshttp/vary), after the release deploys
62: residentText strips ANSI, redacts and caps, and is the identity on runtime-replaced, the op-refused message and the disk-pressure: prefix; sanitizeResidentBody rewrites only error/reason/stateReason/summary and a mirrored stderr — a token in stateReason is scrubbed on the streamed /exec document, the JSON routes' body and the nested /residents live view alike — passes non-objects through and leaves the input untouched; residentState admits the closed table only[unit] src/execution/residentText.test.ts::residentText::*, ::sanitizeResidentBody::*, ::residentState::*
62: an error keeps a whole refusal — the item 35 message with its admin action and 422 tail — up to RESIDENT_ERROR_CAP, still stripped and redacted; reason and summary stay at the card-sized cap[unit] src/execution/residentText.test.ts::sanitizeResidentBody — an error is a reply, a reason is a card note::*; src/core/commands/repo.test.ts::repo onboard::the not-in-installation refusal (403) reaches chat whole — the admin action and the 422 detail after it are not clipped (item 35, item 62)
62: the bot's parse — a poisoned /status reason and an off-table state, a poisoned non-2xx probe body, a poisoned /op refusal/summary with a credential in stdout, a poisoned /exec error — never reach a probe result, an op result or a thrown message raw[unit] src/execution/resident.test.ts::resident text is made safe at the parse (item 62)::*; src/core/residentAdmin.test.ts::makeResidentAdminClient sanitizes resident text at the parse (item 62)::*
62: the note a poisoned probe reason produces is one redacted line; a setup failure carrying remote text closes the card with one redacted line and redacts the reply[unit] src/execution/factory.test.ts::makeExecutor resident selection::a poisoned probe reason reaches the note as one redacted line (item 62); src/core/dispatcher.test.ts::*::a setup failure carrying remote text closes the card with one redacted line and redacts the reply (resident-repos item 62)
7/11/34: two snapshot generations — a rotation retires the replaced pair and deletes the one retired before it, the first rotation and a re-fired one delete nothing; offboard, rebuild and their dry runs name every recorded generation's ids; a fresh onboard clears old facts and both snapshot generations[unit] src/execution/snapshotRetention.test.ts::rotateSnapshots::*; src/execution/snapshotRetention.test.ts::backupIdsOf::*; src/execution/snapshotRetention.test.ts::the resident Worker's wiring::*; deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet lifecycle::keeps a retired address closed to a stale watchdog after cleanup
7 live: after a refresh that moved the sha, /debug info shows the new snapshot ids and the bucket still holds the previous pair under backups/<id>/; after the next such refresh that pair is gone[agent] two consecutive default-branch moves on one resident, backupObjects counted between them
62: the disk-pressure refusal names counts and keep tokens, never a thread key or a free-text detail; a step-report tail is stripped and redacted before the cut[unit] src/execution/residentDiskBudget.test.ts::diskPressureReason — the refusal names free, reserve, projected, what was evicted and what was kept::full shape; src/execution/residentStepReport.test.ts::describeStepFailure::strips terminal control sequences and redacts credentials before keeping the tail (item 62)
62: oneLine keeps the first non-empty line with collapsed whitespace and composes with redactAndCap for a title[unit] src/core/redact.test.ts::oneLine::*
52: detectCommands — pnpm workspace → pnpm install --frozen-lockfile / no-op build / pnpm test; no root package.json → no install + no-op build/test (a lockfile alone is still none); packageManager beats the lockfile, unknown field falls through; npm with/without lockfile; yarn classic vs berry; bun lockfiles; unparseable package.json = no scripts; every note text[unit] src/core/repoToolchain.test.ts::detectCommands::*
52: githubRepoInspector — tree + package.json reads with the bearer and the raw accept, deciding fields only; no package.json → one call; 404 names both causes, other statuses carry the code; no credential → no request; throwing fetch named; unparseable → null; ref URL-encoded[unit] src/execution/githubRepoInspect.test.ts::githubRepoInspector::*
52: repo onboard — inspects <slug>@<ref>, posts the detected table, replies with toolchain + reasons + — none cells; no-package.json table lacks install; explicit flags win per key; uninspectable → npm table + the ⚠️ not-inspected line (also with no inspector wired) naming only the defaulted keys, or "nothing was assumed" when all three are explicit; an explicit --build true renders verbatim, not as "none"[unit] src/core/commands/repo.test.ts::repo onboard::item 52*
52: settle — onboard + real rebuild settle, dry-run/list/offboard/reconfigure/test/build do not; warm after N polls at SETTLE_POLL_MS; down at install/build/test → reason + the fix commands, down elsewhere (snapshot/clone/timeout/no reason) → reason + a retry hint and never repo reconfigure; 404 / non-200 / throw / other state / admin unavailable each a sentence; still onboarding at SETTLE_MAX_MS → bounded ⏳ report[unit] src/core/commands/repo.test.ts::provisioning follow-up (item 52)*::*
52: an accepted onboard has a bounded settled outcome; dry runs do not settle[unit] src/core/commands/repo.test.ts::provisioning follow-up (item 52)*::*, src/core/commandRegistry.test.ts::settle — the deferred outcome*::*
52: admin client status(resource) → GET /status?resource= with the bearer[unit] src/core/residentAdmin.test.ts::*status*
52: Live: repo onboard <owner>/<pnpm workspace> → reply names Toolchain: pnpm, table pnpm install --frozen-lockfile / none / pnpm test; the thread gets ✅ … is warm (or a ❌ with the resident's own install/build error) within SETTLE_MAX_MS; repo onboard <owner>/<repo with no package.json> → Toolchain: none, no install, and reaches warm[agent] Run both onboards in a channel the bot is in after the bot deploy; compare the replies to the wording above and repo list to the follow-up.
Typecheck passes[agent] cd deploy/cloudflare-resident && npm install && npm run typecheck — exit 0.
Every route 401s without a bearer[agent] For each of /onboard /offboard /reconfigure /rebuild /residents /debug /status /attach /exec /read /write /op: curl -s -X POST https://<resident hostname>/<route> → 401 {"error":"unauthorized"}; /rebuild is admin-scope, so it also 401s with the operator token.
Operator token refused on admin routes[agent] curl -s -X POST -H "Authorization: Bearer $OPERATOR" .../onboard (and /offboard, /reconfigure, /debug, plus GET /residents) → 401.
Admin token valid on operator routes (superset)[agent] curl -s -H "Authorization: Bearer $ADMIN" ".../status?resource=repo:<owner>/<name>" of an onboarded resource → 200.
Onboard → provisioning → warm with recorded sha[agent] curl -s -X POST -H "Authorization: Bearer $ADMIN" -H "content-type: application/json" -d '{"resource":"repo:jshttp/vary","commands":{"test":"node -e \"require('"'"'./index.js'"'"')\"","build":"npm pack --dry-run","install":"npm install --no-audit --no-fund"},"defaultRef":"master","provisioningTimeoutMs":600000}' .../onboard → 202 {"state":"onboarding"}; poll /status → warm; /debug {"op":"info"} shows sha equal to GitHub's master HEAD, a stamped snapshot {ref, sha, lockfileHash} with two backup ids, schedules {provisionRun: 0, provisionDeadline: 0}, and after the next cron firing refresh.instance set.
Refresh with stable sha does not reinstall or re-snapshot[agent] POST /debug {"op":"refresh-now","resource":…} (admin) → action:"created"; when /status refresh.instance.lastStep reads measure: done, /debug {"op":"info"} → lastRefreshAt bumped, state warm, snapshot.createdAt and both backup ids UNCHANGED.
Natural (unforced) refresh cadence[agent] With no debug calls for >10 min, /debug {"op":"info"} shows lastRefreshAt advancing on its own every ~600s and state warm.
Wake path: restoring → warm by restore, not re-clone[agent] POST /debug {"op":"stop-container"} then {"op":"refresh-now"}; poll /status every ~4s → a visible {"state":"restoring","reason":"rehydrating"} window, then warm; /debug {"op":"info"} → lastRestore.ms well under provisioning time, provisionedAt and snapshot ids unchanged.
5: a restore the runtime replacement interrupts is interrupted with a restore-interrupted: reason — every SDK wording for a replaced runtime, on the message or one cause down; a stalled, capped or timed-out restore and a crashed container stay down with the r2-restore-failed: reason[unit] src/execution/residentRefresh.test.ts::restoreFailureDisposition …::*
5 live: a resident Worker deploy that rolls the container mid-restore leaves the resident degraded(restore-interrupted: …), the instance step's retry restores onto the new container → warm within about two minutes, no down, no rebuild, provisionedAt and snapshot ids unchanged[agent] deploy the resident Worker (or {"op":"stop-container"}) while /status reads restoring; poll /status: degraded with the reason above, then restoring, then warm; /debug {"op":"info"} shows lastRestore refreshed and autoRebuilds unchanged
Stamp mismatch is refused → down(snapshot-stamp-mismatch)[agent] A restored disk that fails the stamp verification → /status {"state":"down","reason":"snapshot-stamp-mismatch: restored disk {…} != stamp {…}"} and no further cycle runs (a down resident is given no instance). The check verifies the mirror's rev-parse sha and the ls-tree lockfile key against the stamp on every restore; to trigger deliberately, corrupt/replace one of the two backups/<id>/ R2 objects and force a wake.
No lifecycle timer: the resident sources arm no alarm slot (the one exception is the registry DO's one-shot drain-lift alarm at the drain record's own ends — until, or the hold's earlier cycle bound, item 69 — record 0064, never on ResidentDO or the refresh cycle), carry no refresh alarm handler and no alarm-missed, and schedule a callback only for provisioning (initResident); live, /debug {"op":"schedules"} on a settled resident answers {"provisionRun":[],"provisionDeadline":[]} and never a refresh or sweep key, and a watchdog pass never answers action:"rearmed"[unit] deploy/cloudflare-resident/lifecycle.test.ts::* (red-verified against the alarm chain: setAlarm in a comment, onRefreshAlarm, alarm-missed and a schedule outside initResident); [agent] POST /debug {"op":"schedules"} then {"op":"run-watchdog"} after the deploy → every action is none, provision-timed-out or auto-rebuilt
The retired alarm chain's rows cannot outlive the flip: worker.ts names exactly the three retired callbacks (onRefreshAlarm, onWorktreeSweep, onDiskMeasure), declares no method by those names, and ResidentDO's constructor deletes each one's rows synchronously after super() and before its asynchronous UID-index initialization; live, a resident carrying the chain's pending rows before the deploy answers /debug {"op":"schedules"} with no refresh or sweep row after it, and the resident Worker logs no Callback onRefreshAlarm not found line[unit] deploy/cloudflare-resident/lifecycle.test.ts::worker.ts declares the retired alarm chain's callbacks — the refresh alarm, the worktree sweep, the disk measure — and no method by those names, deploy/cloudflare-resident/lifecycle.test.ts::ResidentDO's constructor deletes every retired callback's rows before the first event — the SDK's alarm loop skips a row whose callback is gone without deleting it (red-verified: both fail against the flip's first commit, which declared neither); [agent] the four production residents' {"op":"schedules"} before and after the release that carries this, plus a Worker-log query for Callback .* not found over the deploy window → none
Provisioning failure fails closed with a named step[agent] Onboard a nonexistent repo (e.g. repo:acme/no-such-repo) → within ~60s /status → down with provision-failed at clone: … terminal prompts disabled (anonymous git fails fast; no hang). The registry slot is KEPT (status is 200, not 404) until offboard. A too-small provisioningTimeoutMs (10s) on a real repo → down provision-failed at install: … (timed out) — per-step budgets fail closed.
Stuck onboarding → down(provision-timeout) + slot released[agent] Fault injection: on a resident whose recorded deadline has passed, POST /debug {"op":"force-onboarding"} then {"op":"run-watchdog"} → result action:"provision-timed-out", and /status → 404 (cap slot released; /residents count drops). The organic path — kill a mid-provision DO and wait out the deadline — is equivalent but needs a mid-flight redeploy; the injected check exercises the same provisionTimedOut().
Mint without secrets is command-level, never a lifecycle flip[agent] With GITHUB_APP_* secrets unset: POST /debug {"op":"mint-token","resource":…} → {"ok":false,"error":"github-app-not-configured: …"}; /status immediately after → still {"state":"warm","reason":""} and lastRefreshError untouched by the debug mint.
A watchdog-stamped degraded(stale-mid-flight…) always runs the next cycle — never parks on a streak, and never held to the idle cadence: a row that parked warm and whose wake died before a cycle could clear idleSince is due at the next bucket once it reads restoring or degraded, the idle interval applying to a warm row alone[unit] src/execution/residentInstanceId.test.ts::shouldCreateRefreshInstance …::an idle-marked row that is degraded (the watchdog's stamp on a wake that died before its cycle could clear idleSince) or restoring is due at the awake cadence: one bucket after the last instance, never the idle interval, ::an idle-marked warm resident gets an instance only at the idle cadence the row records, counted in whole buckets since the last instance; [agent] On a resident whose /status shows a stale refreshing or restoring (older than 30 min, nothing running) and whose /debug {"op":"info"} shows idleSince set (the row parked before the marker; the earlier run of this procedure was on a row without idleSince, which is why it passed against the cadence bug): POST /debug {"op":"run-watchdog"} → degraded(stale-mid-flight…) naming the last instance and, in the same pass, instance.action:"created"; the resident returns to warm (or a refresh-produced degraded reason) within one cycle (the failure shape this rules out: still degraded(stale-mid-flight) after the pass, refresh.instance unchanged, lastRefreshAt untouched).
A mint FAILURE (App configured, repo outside the installation) still refreshes: the cycle records token-mint-failed (… fetching anonymously) and returns to warm with lastRefreshAt advanced[agent] On repo:jshttp/vary (public, not in the installation): POST /debug {"op":"refresh-now"} → action:"created"; within ~60s /status → warm and /debug info shows lastRefreshAt within the last minute and lastRefreshError naming the mint (token-mint-failed (command-level, fetching anonymously): … HTTP 422 …); the failure shape this rules out: state stays degraded after the cycle ran and lastRefreshAt does not advance.
Repo-scoped mirror mint: only contents:read, pull_requests:read, metadata:read for root-owned clone/fetch and PR lookup; the returned permissions, repository_selection: selected and sole repository are verified before cache[unit] src/execution/githubMintScope.test.ts::GitHub installation token scope::keeps a resident mirror token read-only, including its pull request lookup, ::checks the returned repository and effective permissions without including the token in an error; deploy/cloudflare-resident/credentialBoundary.test.ts::resident model credential boundary::mints only mirror-read permissions and checks the effective GitHub scope before caching; [agent] an authorized POST /debug {"op":"mint-token","resource":"repo:<owner>/<name>"} returns ok:true only after the Worker's scope check, without returning bearer bytes; private-repo onboard and refresh still clone/fetch successfully.
Offboard removes everything and reports it[agent] POST /offboard {"resource":…} → 200 with registryRemoved:true, schedulesCancelled:true, containerStopped:true, storageCleared:true, backupObjectsDeleted:4, r2ObjectsDeleted:<n>, errors:[] — backupObjectsDeleted is 4 for a provisioned resident (2 snapshots × archive+metadata under backups/<uuid>/, deleted via the stored handles). A second offboard and a /status probe both → 404.
Duplicate onboard refused / cap enforced / invalid input 400s / wrong method 405s[agent] Duplicate onboard → 409; an over-cap onboard → 429 naming the cap; "resource":"Repo:BAD" → 400 naming the format; "resource":"repo:probe" (no owner/name slash) → 400 naming the slug form; GET on /onboard → 405.
Deploy wake ping — the runner GETs the resident's /healthz once after its deploy[unit] src/deploy/plan.test.ts::planDeploy::each step spawns… (the resident step's wakeUrl); [agent] npm run cli -- deploy all --only resident logs resident: awake — GET https://<resident hostname>/healthz → HTTP 200 after Current Version ID.
Deploy preflight: idle fleet → allow[unit] deploy/cloudflare-resident/preflight.test.mjs::…decide()…::idle everywhere → allow, not forced, ::no residents onboarded → allow, ::force on an idle fleet is a plain allow, not flagged.
Deploy preflight: busy → refuse, naming residents and counts[unit] preflight.test.mjs::…decide()…::busy → refuse, naming every busy resident with its count (idle residents are NOT named; hint names RESIDENT_DEPLOY_FORCE=1). [agent] Start a long /exec (e.g. sleep 120) on an attached thread of an onboarded resident, then RESIDENT_DRAIN_TOKEN=… npm run preflight → exit 1, message lists that resident (1 in flight); after the exec ends → exit 0.
Deploy preflight: fail closed on no token / unreachable / unknown resident[unit] preflight.test.mjs::…decide()…::unreachable / no token → refuse with the operator hint, ::a resident whose live view failed is unknown → refuse (fail closed), ::a payload without per-resident inFlight (old Worker) is unknown → refuse, ::a negative or non-integer inFlight (counter bug) is unknown, never idle → refuse, ::malformed payload → refuse; readToken()::uses the drain bearer; read, operator, and admin bearers cannot replace the runner's drain. [agent] (with RESIDENT_BASE_URL=https://<resident hostname> in the env — deploy all sets it from the profile; the script alone needs it) RESIDENT_DRAIN_TOKEN= npm run preflight → exit 1 naming the drain bearer; RESIDENT_DRAIN_TOKEN=bogus npm run preflight → exit 1 quoting HTTP 401; RESIDENT_DRAIN_TOKEN=x RESIDENT_BASE_URL=https://127.0.0.1:9 npm run preflight → exit 1 fetch failed; RESIDENT_BASE_URL= npm run preflight → exit 2 naming RESIDENT_BASE_URL (release-and-deploy.md item 17).
Deploy preflight: a resident provisioning (onboarding, 0 in flight) → refuse naming the state; refreshing/restoring with 0 in flight → allow with a WARNING naming the state; settled states allow; force overrides a provisioning with the state named[unit] preflight.test.mjs::…decide()…::a resident provisioning (onboarding) → refuse…, ::a resident refreshing or restoring with 0 in flight → allow with a WARNING…, ::warm / degraded / down with 0 in flight are not mid-cycle → allow, ::busy AND mid-cycle are both reported…, ::an unrecognized lifecycle state is unknown → refuse…, ::force overrides a provisioning…. [agent] npm run preflight during a refresh window (/debug refresh-now first) → exit 0 with WARNING: mid-cycle: repo:<owner>/<name> (refreshing); during an onboard → exit 1 quoting provisioning: repo:… (onboarding).
Resident retention count: a known durable run owner counts until exact preservation permits release; deploy activity is separately verified through the fenced owner read; only a positively legacy registration uses its deadline plus grace. Attach covers setup plus the full lease; lease start and resume set the actual remainder, re-attach uses that remainder, and an older run, generation, or attachment fence cannot update a newer registration. The ledger mints monotonically increasing fences for the current owner; attach accepts only a larger fence or the exact same owner, including when generations share a timestamp, and retains its high-water row after eviction and binding purge. Drain, memory, and stale-image gates exempt only a live registration owned by this run and fence or a newer ledger claimant. Detach checks the exact owner under the attach lock and retains pushed-branch evidence after an early eviction, so an old setup finalizer or fenced run cannot remove a successor’s tree. Legacy rows use the clean-idle boundary; a lost release cannot block until the next refresh, visible in /debug info[unit] deploy/cloudflare-resident/runRegistration.test.ts::a run's registration is held from attach to release::registrations are durable rows under their own prefix — a fresh isolate still counts the process that survived in the container, ::a successful attach registers the run; a refused attach does not, ::the binding's eviction clears the registration — detach, sweep and disk pressure all end there, ::only the current run may release its workspace, including after its deadline, ::a larger ledger fence claims across runs or generations; delayed older attaches cannot replace it, ::the preflight-facing counts see registered runs the op counters miss::registeredRunsBeyondOps retains durable owners beyond the execution deadline and bounds only legacy registrations, ::an abandoned registration stops blocking at the sweep's idle boundary, while an active or unknown one remains protected, ::the idle sweep protects a registered run through its budget even when its last command was over an hour ago, ::GET /status adds the registrations (getInFlightCount), so a person sees the protected registration count, ::the live view's inFlight and runsInFlight (GET /residents, /debug info) both carry the registrations, ::isIdle protects a registered run beyond an hour of inactivity but permits sleep after its bounded deadline, src/core/reviewRound.test.ts::attachRoundWorkspace (explicit AgentDef → attach + paired release)::passes the ledger claim, run and generation through attach and release, src/core/runLedger/inMemory.test.ts::resident attachment claims::mints increasing fences only for the current ledger owner, including same-second reclaims, src/execution/resident.test.ts::ResidentExecutor.release — return the thread's pool user when a run ends::sends the owning run ID and generation so an old finalizer cannot detach a reclaimed successor, src/execution/factory.test.ts::makeExecutor resident selection::a read-identity agent attaches with readonly:true; a write-identity agent's body has no readonly field, src/execution/resident.test.ts::ResidentExecutor.open (attach-on-open)::sends a run budget only when the caller provides one, src/execution/resident.test.ts::ResidentExecutor.open (attach-on-open)::protects setup through the later lease, then uses the remaining lease on reattach, src/execution/resident.test.ts::ResidentExecutor.open (attach-on-open)::sends the lease deadline to the registration owned by this run, src/core/dispatch/runLoop.test.ts::runLoop — the model turn and everything that rides on it::sets the resident registration deadline from the harness lease, src/core/dispatch/runLoop.test.ts::a resume with the answer in hand…::a resumed run sets its resident deadline from the saved lease remainder without a new lease event, deploy/cloudflare-resident/runRegistration.test.ts::a run's registration is held from attach to release::the lease update changes only its own durable registration, deploy/cloudflare-resident/runRegistration.test.ts::a run's registration is held from attach to release::only the owning run with a live deadline may reattach through drain, memory, and image gates, src/core/dispatcher.test.ts::run ledger write-through (docs/reference/specs/run-history.md item 35)::a resume re-attaches its recorded resident worktree in reuse-only mode and continues the transcript; the row's binding is refreshed (item 54), src/core/dispatch/relaunch.test.ts::prepareRelaunch — the relaunch decided and prepared::an in-process relaunch carries its run ID and generation through reattach while the fleet gate requires ownership. [agent] A fenced preflight reports a terminal registration as retained while preserving its workspace; a live ledger owner refuses until it settles.
An owned repo-resident run claims its ledger fence before probing, rechecks ownership after asynchronous selection and sandbox seeding, and checks the resident attachment fence under the thread lock before drain, memory, image, or recreate gates. A stale generation never enters a cold fallback or starts from a completed warm attach; its registration is released with the attached fence. A current owner may.[unit] src/execution/factory.test.ts::makeExecutor resident selection::a fenced owner cannot provision cold when the resident is down, ::a current owner rechecks before and after taking a cold fallback, ::a generation reclaimed during the resident probe cannot provision cold, ::a generation reclaimed during the resident probe cannot attach warm, ::a warm attach uses the fence claimed after its probe, ::a generation reclaimed during warm attach releases its registration before returning, ::a resident registration mismatch never becomes a cold fallback; src/execution/factory.test.ts::makeExecutor seeded sandbox::a generation reclaimed during a seed wait cannot start from that sandbox; deploy/cloudflare-resident/runRegistration.test.ts::a run's registration is held from attach to release::checks the attachment fence under the thread lock before drain, memory, and image gates
Deploy preflight: force bypasses with a warning[unit] preflight.test.mjs::…decide()…::force overrides busy — allowed, flagged, and the warning still names the busy residents, ::force overrides unreachable — allowed and flagged. [agent] RESIDENT_DRAIN_TOKEN= RESIDENT_DEPLOY_FORCE=1 npm run preflight → exit 0 with preflight WARNING on stderr.
/status and /residents expose in-flight counts[agent] curl -s -H "Authorization: Bearer $ADMIN" ".../status?resource=repo:<owner>/<name>" → {"state":"warm","reason":"","inFlight":0} at rest; GET /residents → top-level inFlight (a number when every resident answered, null with inFlightUnknown ≥ 1 otherwise) and residents[].live.inFlight; during a running /exec both read ≥ 1.
Engine state survives a redeploy[agent] env -u CLOUDFLARE_API_TOKEN npm run deploy while a resident is warm → immediately after, /status → warm and /debug {"op":"info"} shows the same facts/snapshot and refresh.instance (DO SQLite persists; the container disk does not need to — the next wake restores).
14: the exec env carries the pairs — the resident executor sends a write run's four commit identity variables in each /exec body's env, resolved per exec, a caller's variables under them and no x-env-* header, and the Worker reads them through the one validated reader under its own injected variables[unit] src/execution/residentExecEnv.test.ts::ResidentExecutor — the commit identity rides each /exec body's env::*, src/execution/residentExecEnv.test.ts::resident Worker /exec env::*
x-env-* headers are inert[agent] Black-box through /exec: `curl -s -X POST -H "Authorization: Bearer $OPERATOR" -H "content-type: application/json" -H "x-env-EVIL: pwned" -H "x-env-GH_TOKEN: sneaky" .../exec -d '{"resource":"repo:jshttp/vary","threadKey":"slack:u4a","command":"env
Timing-safe comparison is constant-time[agent] Not provable over the network; held by code review — read timingSafeEqual in deploy/cloudflare-resident/worker.ts and confirm it is still a byte-XOR accumulate over equal-length buffers with no early return other than the length check (the length is what any comparison leaks).
Unprivileged worker pool + no root escalation in the image[agent] Through the engine: provisioning/refresh run install/build via su -s /bin/bash worker1 -c … (a root-owned command table executing as uid 2001) — a failing su would fail provisioning. Direct /exec proof: command:"id" on an attached thread → uid=2002(worker2) gid=2002(worker2) groups=2002(worker2) — every thread exec is privilege-dropped to its own pool user.
writable attach requires a Git door bearer; older thread and staging credential files are removed before reused-tree Git probes and dependency hooks; cleanup refuses symlinked parents without deleting outside files[unit] deploy/cloudflare-resident/credentialBoundary.test.ts::*; deploy/cloudflare-resident/legacyCredentials.test.ts::*; [agent] After deployment, inspect a reattached authorized writable run: the origin names the Git door, .git/github-credentials is absent, and a missing-door attach returns a named refusal.
failed thread-directory removal retains its binding and pool user, so a later run cannot inherit an old credential file under the same UID[unit] deploy/cloudflare-resident/credentialBoundary.test.ts::resident model credential boundary::keeps a pool user reserved until its old credential-bearing thread directory is removed; [agent] Fault-inject a resident eviction rm failure on a disposable thread and confirm detach reports thread-cleanup-failed, the binding still names its user, and a second thread receives another UID or a pool-exhausted refusal.
a pool user with an untracked older thread or op directory, or old private staging content, cannot be assigned to a fresh attach or op; both scans fail closed and a candidate remains reserved through inspection; a failed write attempts safe stage cleanup, and residual content still refuses UID reuse[unit] deploy/cloudflare-resident/credentialBoundary.test.ts::resident model credential boundary::refuses an old thread directory owned by a candidate UID, deploy/cloudflare-resident/legacyCredentials.test.ts::legacy resident credential cleanup::refuses reuse of a UID with a credential or staged write without reading either, ::fails closed on a linked staging directory, ::cleans a failed write's stage file without following a linked stage; [agent] On an isolated resident fixture, leave a pool-owned thread directory and staged credential after its binding is absent, then confirm a new attach and op receive another user or a named refusal and cannot read the old content.
a failed resident VM destroy leaves a durable refusal before runtime identity is forgotten; rebuild, recovery and offboard cannot provision or run on an unconfirmed old disk, and offboard can retry without deleting backup state[unit] src/execution/residentDestroyGate.test.ts::resident container destruction fence::*, deploy/cloudflare-resident/destroyFence.test.ts::resident VM destroy uncertainty::*; [gap] On an isolated resident fixture, fault-inject a rejected SDK destroy, verify status reports resident-destroy-unconfirmed, exec and onboard refuse, offboard preserves state, then retry destroy and verify normal operation only after confirmed fresh VM placement.
A generated UID and its exact owner are reserved durably before native account creation; every UID-scoped command, cleanup and cancellation checks that owner and the sole claimant. Missing or malformed generation state refuses admission. Historical usage never triggers a VM reset. Operator recreation retains its durable admission pause and independent native, activity and preservation guards; generation rotation occurs only after confirmed destruction.[unit] deploy/cloudflare-resident/uidOwnership.test.ts::indexed resident UID ownership::*; deploy/cloudflare-resident/runRegistration.test.ts::resident workload admission::*; deploy/cloudflare-resident/poolSpends.test.ts::resident pool UID generation fence::*; deploy/cloudflare-resident/destroyFence.test.ts::resident VM destroy uncertainty::*; [gap] Verify exact ownership and generation rotation on an isolated physical VM; local tests are not production reset proof.
a UID-scoped shell guard reads its one UID claimant row rather than all retained thread rows; each DO start rebuilds the row from primary bindings before serving events, and every primary create, eviction, rollback or deletion updates its claimant row transactionally; a missing or malformed row and two claimants refuse shell access[unit] src/execution/residentPoolSpends.test.ts::resident pool UID binding index::*, deploy/cloudflare-resident/poolSpends.test.ts::resident pool UID binding index::*; [gap] Measure command latency on an isolated resident with the observed 1,830 retained bindings, including startup rebuild, and verify that a duplicate live claimant still refuses before shell execution.
distinct worktree + user per thread on the same ref[agent] Same attach body with "threadKey":"slack:u4b" → {"workspace":"/workspace/threads/slack-u4b-aa489653/master","user":"worker3",…} — different path, different OS user, same sha.
OS-layer cross-thread isolation[agent] threadA /exec echo … > notes-a.txt && chmod 600 notes-a.txt (file lands worker2 600); threadB /exec cat /workspace/threads/slack-u4a-27aa257c/master/notes-a.txt → Permission denied exit 1; ls /workspace/threads/slack-u4a-27aa257c/ → Permission denied exit 2 (700 thread dir denies traversal, not just the file).
engine plumbing unreadable from threads[agent] threadB /exec cat /workspace/mirror/config → Permission denied (mirror is root:worker1 750; worker1 keeps group read for the checkout's fetch-from-mirror); cat /workspace/.resident/git-credentials; ls /workspace/.resident/ → both Permission denied (700 root).
shared dep inodes are tamper-proof[agent] threadA /exec echo poison >> node_modules/.package-lock.json → Permission denied while stat shows the inode worker1 worker1 664 — hardlink materialization shares file inodes read-only; a thread can replace entries in its own dirs but cannot mutate bytes the warm checkout (or another thread) sees.
no App token is visible in thread process arguments, environment or credential files[unit] deploy/cloudflare-resident/credentialBoundary.test.ts::*; [agent] On an authorized run, inspect key presence and file absence without printing bearer values.
20: the resident no longer refreshes an App token into a thread worktree; writable /exec requires a door binding and run bearer; named-ref replacement scrubs its superseded sibling before the same UID may enter the new tree, and own-PR fallback scrubs the retained tree before a user-scoped branch probe; failed scrubs refuse attach[unit] deploy/cloudflare-resident/credentialBoundary.test.ts::*, deploy/cloudflare-resident/legacyCredentials.test.ts::*
20 live: a reused pre-fix worktree loses its App credential file before the next model command; a natural authorized Git fetch/push uses the door and obeys its repo/ref policy[agent] After deployment, use an existing authorized writable run on an onboarded repository; record file absence, accepted bound operation and refusal of a foreign ref without exposing token values.
re-attach is idempotent; untracked scratch survives[agent] Re-attach slack:u4a (no refHint — the binding is sticky) → same workspace, recreated:false, reconciled:false, deps:"none"; /exec shows notes-a.txt intact and git status --porcelain -uno empty.
18: build outputs are rebuildable in a thread tree[agent] On a fresh-thread attach of a repo with a build step, /exec npm run build exits 0 and stat -c '%U %h' dist/index.js shows the thread user with link count 1. The failure shape this rules out: npm run build → EACCES … dist/… with dist/index.js worker1 644, link count 3 (a hardlinked build output).
18: tool-managed paths inside node_modules are rebuildable[agent] on a fresh attach, stat -c '%U %h' node_modules/.vite (or .cache) shows the thread user with link count 1 (a real copy) while stat -c '%U %h' node_modules/react/package.json shows worker1 with link count > 1 (still hardlinked); /exec npx vitest run rewrites node_modules/.vite/vitest/results.json and exits 0; echo poison >> node_modules/react/package.json is still Permission denied.
dirty tracked state → wipe + recreate[agent] /exec echo "// dirt" >> index.js (status M index.js), then re-attach → recreated:true, deps:"hardlink" (rematerialized); /exec git status --porcelain empty and notes-a.txt gone — a fresh clean tree.
hostile inputs are refused before anything derives[agent] refHint:"../evil" → 400 naming the ref pattern; threadKey:"foo;rm" → 400 naming the threadKey pattern; /debug {"op":"threads"} immediately after shows no new binding — nothing created.
nonexistent ref → named error, nothing bound[agent] refHint:"no-such-branch-again" on a fresh threadKey → 400 {"error":"unknown-ref: ref \"no-such-branch-again\" does not resolve in the mirror (even after a fetch)"} and /debug threads shows the fresh allocation rolled back (no binding, no leaked pool user).
needs-ref contract names the default branch[agent] Attach on a fresh threadKey with no refHint → 409 {"error":"needs-ref: this thread has no ref binding yet — supply refHint","needs":"ref","defaultRef":"<default branch>"} (the bot-side bind-by-default flow is item 30).
mirror mutex serializes refresh vs attach[agent] POST /debug {"op":"refresh-now"} then an immediate attach on a fresh thread → attach completes with mutexWaitMs > 0 (it queued behind the refresh's mirror fetch); /debug info right after shows the refresh also completed (lastRefreshAt bumped, state warm). Both finish; neither fails.
/read /write confined to the worktree, as the thread user[agent] /write {"path":"docs/u4-note.txt","content":"…"} → {"ok":true,"bytes":25}; /read returns the exact content; /exec stat shows the file worker2-owned (written privilege-dropped, content staged via file — never argv). Escapes: path:"../../mirror/config", path:"/workspace/.resident/git-credentials", write to "../../checkout/pwned.txt" → all 400 path-escape.
unattached/evicted/recycled threads get named errors[agent] /exec on a never-attached threadKey → in-body {"error":"not-attached: …","needs":"attach","exitCode":127}; on an evicted thread → {"error":"evicted: …","needs":"attach"}; after a container stop+wake (disk recycled, worktrees gone) → {"error":"worktree-missing: …","needs":"attach"}, and the follow-up attach recreates the same workspace (recreated:true, deps:"hardlink").
Hot path: repeated /exec on a warm attached thread is materially faster than before the memo round, and correctness held — a mid-run resident deploy still surfaces as runtime-replaced (item 43) and the next call re-probes[agent] On a warm resident thread, time 20 × POST /exec {command:"true"} back-to-back before and after the deploy carrying this change; expect a clear median drop (the removed work: 1 registry hop + 3 storage gets + 2 container forks per call). Then POST /debug {"op":"stop-container"} (or a resident deploy) mid-thread → the next /exec answers runtime-replaced/worktree-missing per items 21/43, never a stale success.
Exec/op/read streams are capped at the source: exit code preserved (0 and non-zero), stdout/stderr separate, each stream cut at capBytes, cd failure surfaces as before, arbitrary command text (quotes/&&/subshell/trailing comment) unchanged, BOTH temp files cleaned on exit[unit] src/execution/residentExecWrap.test.ts::capWrappedCommand (run under real bash)::* — the generated script runs under a real bash in the test
Timeout salvage: a SIGKILL mid-command leaves the fixed files; the recovery script returns the capped pre-kill output and removes them; recovery over missing files is a clean no-op[unit] same suite, ::fixed-file mode + recovery… (a real SIGKILL from the test harness) and ::recovery over missing files…
/exec command bound: a command over 64 000 chars → pre-validation 400 naming the limit (a normal client Error bot-side, never infra); a long-but-legitimate command (e.g. a ~20 KB heredoc) runs[unit] src/execution/resident.test.ts::…command-too-long rejection…, ::two consecutive command-too-long rejections are plain errors, never infra (healthy resident) (fixtures at the 64 000 wording). [agent] on a warm thread: /exec a 65 001-char command → 400 at most 64000 chars; a 20 KB heredoc writing a file → exit 0 and the file exists.
Live: a thread /exec emitting ~10 MB (e.g. yes … | head -c 10000000) answers with the usual 100k-char truncated streams and a clean exit — and the resident stays up (no runtime-replaced from DO memory pressure)[agent] Run it on a warm resident thread; expect truncated: true, exit 0, and /status still warm.
Hot path: /exec on a never-onboarded resource answers a named refusal (503 not-serviceable: resident is not provisioned yet…), not a hang and not a 404 — the registry hop is gone from the data plane by design[agent] POST /exec {resource:"repo:acme/nonexistent", threadKey:"t", command:"true"} with the operator bearer → in-body error naming not-provisioned; /attach on the same resource still answers 404 … is not onboarded.
Sweep idle release: one hour of inactivity or an expired run registration selects a candidate; a live or uncertain owner stays, while an exact terminal owner with matching saved or clean evidence may be released[unit] deploy/cloudflare-resident/pausedWorkspace.test.ts::paused unpublished work at resident removal::*
Image reconcile: after a deploy that changed the pool, checked image replacement stops an eligible idle container within one refresh cycle (never by an attach, which refuses instead — issue 2101) and comes back on the new image[agent] After a deploy that changed the pool, with verified preservation and no active or uncertain owner, wrangler tail switchboard-resident shows image-stale (refresh): worker33 missing … stopping (an attach in the window logs image-stale (attach): … the attach is refused and falls back to the sandbox); next /status window restoring → warm; then a review attaches warm (no cold-fallback note).
Idle sleep: a resident with a known run owner remains awake until item 71 permits VM loss; inactive or unverified private files do not count as clean[unit] deploy/cloudflare-resident/pausedWorkspace.test.ts::one preservation guard covers automatic loss paths::*
Dash shows idle mode[unit] web/src/pages/residents.test.ts::ResidentDetailPage::shows idle mode when the resident parked its refresh, 'awake' otherwise
Read token: GET /residents → 200; POST /debug {"op":"info"} / schedules / threads → 200; POST /debug {"op":"stop-container"} (and sweep-now, refresh-now, …) → 403; POST /onboard, /rebuild, /attach, /status → 401[agent] curl -s -H "Authorization: Bearer $READ" … per route/op.
48: refresh plan — unchanged sha → unchanged; key unchanged → rebuild without install on a keep-deps clean; key changed or no deps marker → full clean + install, unless the deps-installing marker names the same key — then rebuild WITH install on a keep-deps clean (a resumed install, item 57), a marker for another key being stale evidence (full clean), and a landed deps key always winning over a leftover installing marker; HEAD + built + deps all matching → reuse; a matching built marker with a different HEAD, or a matching HEAD with an unfinished build, never reuses; keep-deps clean is exactly git clean -fdx -e node_modules, full is git clean -fdx[unit] src/execution/residentRefresh.test.ts::planRefresh …::*, ::checkoutUpdateCommand::* (red-verified: module absent)
47: the rebuild holds the mirror lock only to stage (the hardlink seed plus the mirror fetch into the staging tree) and to swap (two renames, the markers written under the same lease); the reset/clean, the deps-view re-link and the build run off-lock in the staging tree and never name the warm checkout; no lock lease budgets the build; the retired tree is removed off-lock; materializeThreadDeps' torn-cache cp -al stays under the mutex[unit] deploy/cloudflare-resident/stagedBuild.test.ts::the refresh rebuild holds the mirror lock only to stage and to swap — never for the build::*, deploy/cloudflare-resident/stagedBuild.test.ts::materializeThreadDeps keeps its torn-cache guarantee::*
47: the staging seed self-heals a swap the container died inside and hardlink-copies the checkout; the swap restores the retired tree on a failed rename and removes nothing under the lock; checkoutUpdateCommand no longer fetches (the mirror read is CHECKOUT_FETCH_COMMAND, under the stage lock)[unit] src/execution/residentRefresh.test.ts::stageCheckoutScript…::*, src/execution/residentRefresh.test.ts::swapCheckoutScript…::*, src/execution/residentRefresh.test.ts::checkoutUpdateCommand::never fetches…
47: Live: a rebuild in flight (a slow build) while an attach lands → the attach completes with mutexWaitMs in the git-network range (sub-second to seconds), never mirror-busy; the cycle still completes warm[agent] POST /debug {"op":"refresh-now"} on a repo whose build takes minutes, then an attach on a fresh thread mid-build; read mutexWaitMs on the attach answer and /debug info after. [gap] Not yet run live — needs a resident whose build outlasts the attach.
48: Live: a default-branch merge that does not touch package-lock.json → wrangler tail switchboard-resident shows refresh: … rebuild (lockfile unchanged — deps kept, build only) and … done in <ms> in the seconds range (well under 60 s), /residents lastRefreshAt advances with state warm; a merge that changes the lockfile → (lockfile changed — full install)[agent] wrangler tail switchboard-resident across a default-branch merge, then /debug info → warm, lastRefreshAt advanced, inFlight 0. [gap] The lockfile-changed half needs a default-branch merge that touches package-lock.json.
48: Live: the first refresh after a resident-Worker deploy holds/reaches warm fast — /status warm within one cycle, no full reinstall in the tail; a deploy landing mid-rebuild is followed by reuse (checkout, deps and build already materialized …) or a keep-deps rebuild, never a second full install[agent] wrangler tail switchboard-resident across a resident-Worker deploy (npm run deploy in deploy/cloudflare-resident), then poll /status through the next instance: restoring(rehydrating) → refreshing → warm within one cycle with no reinstall in the tail. [gap] The reuse plan needs a deploy to land between build completion and snapshot.
44 (the interruptible revision): the preflight decision — onboarding (even with 0 in flight) refuses by name with the force hint; refreshing/restoring with 0 executing runs and no active admission allow with preflight ok: … WARNING: mid-cycle: <resource> (<state>) and never REFUSED, listed under interrupting; a run in flight refuses whatever the state; force over a provisioning is allowed and flagged; busy is judged on executingRuns under an active deploy fence (a refreshing resident whose only in-flight work is its own cycle allows with the warning; missing or impossible activity counts refuse)[unit] deploy/cloudflare-resident/preflight.test.mjs::resident deploy preflight — decide()::a resident provisioning (onboarding) → refuse, naming the state, even with 0 in flight: an isolate swap fails the provision and only a rebuild recovers it + ::a resident refreshing or restoring with 0 in flight → allow with a WARNING naming the state: the refresh instance retries the step after the swap (item 44), a restore is retried by the next hydrate (item 61) + ::busy AND mid-cycle are both reported — neither shadows the other; runs in flight refuse even when the cycle alone would only warn + ::force overrides a provisioning — allowed, flagged, and the warning names the state + ::\executingRuns` decides busy under the fence: a refreshing resident whose only in-flight work is its cycle allows with a warning; a run refuses(red-verified:interruptingabsent, refreshing refused; thenrunsInFlight` ignored)
44 (deploy fence): a normal upload first requires a live fleet drain and a drain-scoped /deploy-fence read; the fence refuses even the registered owner’s reattach while activity is checked and survives the isolate swap until undrain. An exact live owner or in-memory operation refuses; an exact terminal run or acknowledged owner with matching registration, thread and owner fence remains protected but does not count as executing. An ACK needs a positive safe revision; it proves neither runtime ending nor source confirmation nor workspace removal. Retained owners still need the exact UID ledger and sole claimant. Missing owner, stale generation/fence, invalid count, a deploy fence with under 20 minutes left for upload and readiness, and unknown lifecycle refuse. A refused activity read clears only its own reattach fence and leaves the drain. A Worker without /deploy-fence refuses rather than guessing from runsInFlight.[unit] deploy/cloudflare-resident/preflight.test.mjs::resident deploy preflight — decide()::retained terminal registrations do not count as executing under a closed deploy fence, ::refuses retained work without a live fence, an executing owner, or unknown ownership, ::refuses unclassified protected activity but permits an operation to overlap a retained registration; deploy/cloudflare-resident/runRegistration.test.ts::deploy registration activity::*; deploy/cloudflare-resident/drain.test.ts::deploy fence lifetime::*; resident npm run typecheck. [gap] First deployment from a Worker that predates /deploy-fence needs a separate reviewed bootstrap; no live safety receipt yet.
44: Memory retains exact acknowledged owner revisions after terminal settlement. Deploy activity, UID-ledger classification and owner reconciliation read the same validated ACK; malformed/foreign ACKs, conflicting claimants and missing spends stay unknown. Live ownership wins over earlier ACKs. Workspace deletion and image replacement retain their independent custody guards.[unit] deploy/cloudflare-resident/runRegistration.test.ts::deploy registration activity::retains an exact acknowledged owner without granting workspace removal; ::keeps acknowledged retention subject to the exact UID ledger and sole claimant; ::reconciles an acknowledged owner through the same retained observation; ::refuses foreign or malformed acknowledged evidence without changing live-owner protection; deploy/cloudflare-memory/runLedger.test.ts::exact owner evidence for resident preservation::retains the exact terminal workspace obligation through*
44 (ownerless legacy deploy read): a registration with matching thread key, no raw run ID/generation/fence, no high-water fence or binding last-run owner (missing or explicitly null; named or malformed non-null owners fail closed), a valid legacy timestamp/deadline, no active operation and an elapsed existing deadline-plus-grace or clean-idle protection boundary is nonexecuting but its binding and workspace remain retained. Before that boundary it is unknown; an active operation is executing. Partial, malformed, mismatched, fenced, and named-owner rows remain unknown/executing. The fenced live view also requires an exact spent UID and sole binding claimant for a retained row; missing or conflicting ledger evidence is unknown until exact-thread reconciliation. It reports only fixed category/reason counts, never raw run IDs, thread keys or customer names; preflight still refuses unknown activity.[unit] deploy/cloudflare-resident/runRegistration.test.ts::deploy registration activity::classifies only an elapsed, ownerless legacy registration as nonexecuting while retaining its workspace; ::retains an elapsed ownerless legacy registration with a persisted null last-run owner; ::fails closed on partial, malformed, fenced and named-owner legacy lookalikes without leaking identifiers; deploy/cloudflare-resident/preflight.test.mjs::resident deploy preflight — decide()::allows only a categorized elapsed legacy row behind a ready fence and refuses malformed or protected rows
Resident registration ownership: every new attach supplies a complete, valid run ID, generation and positive fence before materializing a workspace; the registration and high-water fence commit together before success is returned. An admin may reconcile only one exact retained thread with an elapsed ownerless row or a verified terminal owner: a single UID claimant, no conflicting spend, active operation or user process, and an existing expected user-owned directory at the stored canonical or deterministic collision-safe -ref-<hash> replacement path are required. Unreadable or partial evidence (including an own owner field valued undefined) refuses without mutation. The transaction adds only the missing UID spend and/or an explicit retained-legacy marker; it never removes a binding or private bytes. A marked legacy row still needs a fresh fenced claimant for new work, and remains protected from automatic removal. After reconciliation, that exact thread may attach with a fresh owner; detachment refuses a conflicting owner and retains unsaved bytes even for the exact owner. Fenced deploy readback reports only fixed category/reason counts, including reconciled legacy; no raw identifiers.[unit] deploy/cloudflare-resident/runRegistration.test.ts::exact-thread owner reconciliation::*; deploy/cloudflare-resident/runRegistration.test.ts::a run's registration is held from attach to release::a larger ledger fence claims across runs or generations; delayed older attaches cannot replace it; resident typecheck. [gap] A disposable resident and an independent live fenced readback must prove real DO storage, UID directory and run-history observations before production migration; no live workspace is touched by this unit.
Resident partial-owner release: an own run ID, generation, or fence field on a persisted registration or last-run owner is owner evidence even when its value is undefined. Legacy detach without a full owner cannot evict that binding or delete private worktree bytes; a truly ownerless legacy registration keeps its supported release path.[unit] deploy/cloudflare-resident/runRegistration.test.ts::exact-thread owner reconciliation::keeps private bytes when a partial owner row meets legacy detach…; deploy/cloudflare-resident/runRegistration.test.ts::a run's registration is held from attach to release::only the current run may release its workspace, including after its deadline
Resident retention marker: a persisted legacyRetainedAt field with a malformed value is unknown evidence. Fenced readback and reconciliation refuse it; detach and automatic workspace removal keep the private tree. A valid marker continues to protect an elapsed legacy registration.[unit] deploy/cloudflare-resident/runRegistration.test.ts::exact-thread owner reconciliation::treats a present malformed retention marker as unknown and preserves its workspace; ::keeps private bytes when a malformed retained marker reaches legacy detach; ::the fenced readback refuses a missing or conflicting spend until the exact thread is reconciled
Resident activity count: any persisted run ID, generation or fence field in a registration or last-run owner counts as protected activity even when its value is undefined; only a genuinely ownerless elapsed registration may count zero. The status, idle and image gates share that count.[unit] deploy/cloudflare-resident/runRegistration.test.ts::the preflight-facing counts see registered runs the op counters miss::counts a persisted partial owner field in…; ::GET /status adds the registrations (getInFlightCount), so a person sees the protected registration count; ::isIdle protects a registered run beyond an hour of inactivity but permits sleep after its bounded deadline
44: a step killed from outside is refresh-interrupted: <step> … (exit 143 / the platform's exit code: 143 / Session terminated / SIGTERM, any of install, build, checkout-update); an ordinary failure stays <step>-failed: … verbatim; a step the cycle itself timed out ((timed out)) is never an interruption; a bare "killed" in compiler output is not either[unit] src/execution/residentRefresh.test.ts::classifyRefreshFailure …
44: refresh-interrupted is never recorded as degraded — the instance step throws it to the engine, whose retry re-enters the step, so the park streak never sees it[unit] the classifier answers interrupted: true with the refresh-interrupted: prefix (src/execution/residentRefresh.test.ts::classifyRefreshFailure (a build SIGTERM…::exit 143 during build → refresh-interrupted, the reason prefixed so the instance step retries it…); the throw itself is Worker code without a harness (runInstanceStep) — [agent] the live recipe below: /status never reads degraded(refresh-interrupted…), the instance row's lastStep reads <step>: interrupted (refresh-interrupted: …) — the engine retries, and after recovery /debug {"op":"info"} shows no degraded streak
44: Live: a container replacement landing mid-refresh → the step is retried by the engine (refresh-interrupted: … on the instance row, never on /status) and warm within ~90 s, no streak[agent] What can interrupt: only a CONTAINER replacement — an image-changing deploy or stop-container; a Worker-only wrangler deploy swaps the DO isolates but the container and its processes survive, so it interrupts nothing (attempted live: the refresh completed untouched under one). The classifier runs in runInstanceStep, so the interrupted step must be a REFRESH cycle, not POST /rebuild (that is the provision path: onboarding → warm, never refresh-interrupted), and planRefresh only rebuilds when the default branch has MOVED since the resident's sha (unchanged otherwise). On a repo with no build script the build step is a no-op (~26 ms observed) and the kill lands in install (lockfile-change cycles) or snapshot (~30 s, every cycle) — a repo with a real build/install exercises the build path. Recipe: land a commit on the default branch (any merge), GET /residents → live.sha still the old sha, then POST /debug {"op":"refresh-now","resource":"repo:<owner>/<name>"} (admin) and, as soon as /status leaves warm (refreshing), POST /debug {"op":"stop-container"} (kills the running step; refuse if other execs are in flight — it kills them too); poll /status every 5 s: it stays refreshing (never degraded(refresh-interrupted…)), /debug {"op":"info"} refresh.instance.lastStep reads <step>: interrupted (refresh-interrupted: …) — the engine retries, then warm within ~90 s (the 30 s retry delay + the step), versus waiting for the next bucket; no park streak, and the retried step's refresh: … reuse / keep-deps line in wrangler tail.
44/#216: a step killed from outside is refresh-interrupted: <step> … (exit 143 / the platform's exit code: 143 / Session terminated / SIGTERM, any of install, build, checkout-update); an ordinary failure stays <step>-failed: … verbatim; a step the cycle itself timed out ((timed out)) is never an interruption; a bare "killed" in compiler output is not either[unit] src/execution/residentRefresh.test.ts::classifyRefreshFailure (a build SIGTERM…
44: a container stopped under one of the resident's own short execs (the ready-stamp probe, a marker read) answers the platform's exit-code wording, Container exited with unexpected exit code: 143, not a shell's exit 143 — SIGTERM's status however it is spelled is the same interruption, in the cycle classifier and in the restore disposition alike; any other exit code (1, 137, 139) stays a genuine crash[unit] src/execution/residentRefresh.test.ts::classifyRefreshFailure (a build SIGTERM…::the platform's exit-code wording for a SIGTERMed container …; src/execution/residentRefresh.test.ts::restoreFailureDisposition (a restore the runtime replacement interrupts…::a container that exited with code 143 under the restore's own execs is interrupted too…
Race fixes (16c) hold: wake-fetch never waits past ATTACH_MUTEX_WAIT_MS; facts written after the wake-fetch carry the concurrent cycle's lastRefreshAt; a re-attach during a sweep clean check keeps its tree[code review] typecheck-only Worker; behaviors are timing-dependent and reviewed by construction (single re-read before every write/evict; bounded lock waits).
16c: the refresh instance counts its cycle in flight only past the entry gates — runInstanceStep hands the step its count instead of counting before it runs, the fetch step calls it only once refreshGate answered go, the other steps (install, build, snapshot, sweep, measure) first thing — so isIdle, reconcileImage("refresh") and the gate's disk-full recycle never see the probing cycle as an operation in flight; and a failed step is excluded from the disk-full recovery's in-flight count only when it had counted itself — a failure inside the gates contributed nothing, and excluding it anyway would read one real run as none[unit] deploy/cloudflare-resident/instanceStep.test.ts::* (red-verified against the instance path that counted itself before its gates: 5 failed)
Stuck mid-flight marker normalizes: refreshing older than 30 min with nothing running → watchdog degraded(stale-mid-flight: … the last instance <id> is <engine status> …) and, in the same pass, the next instance → warm; the idle gate parks only a settled row — warm, or degraded past item 16b's streak (never stale-mid-flight, which never settles) — and the idle cadence applies to a warm row alone, so a parked degraded row stays due every bucket until a rehydrate flips it warm[agent] POST /debug {"op":"run-watchdog"} on a resident showing stale refreshing → action:"none", reason stale-mid-flight… naming the instance id and the engine's word on it, instance.action:"created"; /status within a cycle → warm.
Sweep releases idle bindings on a SLEEPING resident (tree already gone)[agent] Dash for a slept resident with idle bindings after the next refresh instance's sweep step (the idle cadence) → the bindings show evicted; /residents live.threads users "".
Detach: release("always") POSTs /detach force:true; "if-idle" force:false, a tree kept for an op in flight reports released:false + reason, a release that discarded work reports leftBehind when well-formed and nothing otherwise; non-2xx / transport failure → released:false with a legible reason, never a throw[unit] src/execution/resident.test.ts::ResidentExecutor.release …::*
16a: force-detach decision — nothing in flight → proceed; non-force + in flight → the unchanged busy refusal; force + in flight → kill as the binding's pool user; force never kills as a non-pool user (root, empty, worker1, out-of-range) and names why[unit] src/execution/residentDetach.test.ts::planForceDetach …::*
Live hard stop on a registered resident run blocks late commands, confirms termination of the original UID processes, and records stopped_hard; resident binding, files and UID spend history remain available, while the run leaves the active list. Unbound native submissions retain pending cancellation even after a kernel replacement; historical effects remain unknown.[gap] On an isolated resident, start a run with a detached child, hard stop it from /runs, verify the original boot and UID have no remaining processes and the durable run is terminal; verify a delayed original request is refused and another thread remains unaffected. Inspect the retained workspace and original ownership without reattaching.
Dispatcher releases the workspace in the run finally: if-idle for coding, always for the read-only review agent; a throwing release never fails the run[unit] src/core/dispatcher.test.ts::releases the executor's workspace when the run ends …, ::a release that throws never fails the run …
Live: after a review run on an onboarded repo, that thread's binding is evicted but its UID remains spent until confirmed VM destruction; a coding run that leaves uncommitted work has its tree evicted too, the bot log reading [release] … released — N uncommitted change(s) and M unpushed commit(s) were left in the worktree; … and the run's record a work_left_behind note; wrangler tail switchboard-resident shows the /detach and the counts[agent] Post agent:review on a PR of an onboarded repo; when the card completes, the resident detail shows that thread's row evicted within seconds of its last attach. Then agent:coding that leaves an uncommitted edit → its row is evicted too and the bot log names what went.
inactivity eviction keeps the UID reserved and the binding[agent] POST /debug {"op":"backdate-thread","threadKey":"slack:u4d","days":10} then {"op":"sweep-now"} → {"evicted":["slack:u4c","slack:u4d"],"kept":2}; /debug threads shows the evicted bindings RETAINED with evicted:true, user:"", ref intact; /exec ls /workspace/threads/ from a live thread shows the evicted dirs gone. Confirm the evicted UIDs remain unavailable to another owner until VM destruction. Re-attach of slack:u4d (no refHint) → recreated on the sticky ref; it may reclaim its own clean UID. (The unforced path is the same onWorktreeSweep on its self-rescheduled hourly schedule.)
wake path unaffected by the perms hardening[agent] {"op":"stop-container"} + {"op":"refresh-now"} → restore completes (lastRestore {at, ms}), state warm, and /exec stat /workspace/mirror from a re-attached thread → root worker1 750 (the post-restore ensureGitSetup re-locks the restored mirror).
75: dynamic allocation admits a fresh owner beyond 250 past owners without resetting or reassigning their VM identities; active or unknown owners and processes still enforce the separate work limit[unit] deploy/cloudflare-resident/runRegistration.test.ts::resident workload admission::*; deploy/cloudflare-resident/uidOwnership.test.ts::indexed resident UID ownership::*; [gap] Repeat through the deployed attach endpoint on an isolated resident after image qualification.
warm probe → ResidentExecutor; no ctx.repo → per-thread with ZERO probes[unit] src/execution/factory.test.ts::makeExecutor resident selection::warm probe → ResidentExecutor, attached on open…, ::ctx.repo undefined → per-thread path with ZERO probe calls (total input contract)
named fallback carries state+reason verbatim; not-warm never cached[unit] src/execution/factory.test.ts::…::not-warm probe → fallback carrying state and reason verbatim; no attach…, ::not-warm states are NOT cached — the next dispatch probes again
serviceable non-warm states attach — refreshing, and degraded for intact-checkout reasons → ResidentExecutor with the informational note; degraded by an in-rebuild failure stays cold; a mirror-busy attach during a refresh falls back with the named attach-failed note; onboarding/restoring/down fall back with the verbatim state/reason note and no attach[unit] src/execution/residentState.test.ts::isServiceable::* (reason-class matrix), src/execution/factory.test.ts::…::refreshing probe → ResidentExecutor WITH an informational note…, ::degraded probe → ResidentExecutor, note carries the reason, ::degraded by an in-rebuild failure (install-failed) → per-thread fallback, no attach, ::refreshing probe then a mirror-busy attach (503) → per-thread fallback…, ::%s probe → per-thread fallback with the verbatim state/reason note; no attach (×3)
Live: a run dispatched while the resident is refreshing attaches to the last snapshot instead of going cold[agent] With the residents dash showing state refreshing (a default-branch advance within the last minutes, or POST /debug {"op":"refresh-now"} as admin), post a review run in a channel the bot is in → the status card reads resident refreshing — attached to the last snapshot (not — using fresh sandbox) and the resident detail shows a new thread worktree for that thread.
outage circuit breaker — one probe per window, armed by a failure to reach the host alone[unit] src/execution/factory.test.ts::…::probe transport failure → named fallback + negative cache (second dispatch makes no fetch); src/execution/resident.test.ts::ResidentExecutor.probeStatus (transport flag: network=true, HTTP-level=false)
25: a probe whose deadline passes is marked timedOut beside transport and names the deadline; a network failure is not; the selection falls cold for that dispatch alone and the next dispatch probes again (no outage window); the default deadline is 8 s, so a resident answering at 7.9 s is selected warm[unit] src/execution/resident.test.ts::ResidentExecutor.probeStatus::a probe whose deadline passes is an unreachable marker flagged transport-level AND \timedOut`, naming the deadline; src/execution/resident.test.ts::ResidentExecutor.probeStatus::a network failure is an unreachable marker flagged as transport-level, and NOT as a deadline miss; src/execution/factory.test.ts::makeExecutor resident selection::a probe that misses its deadline falls cold for THIS dispatch alone and arms NO outage window: the next dispatch probes again; src/execution/factory.test.ts::makeExecutor resident selection::the default probe deadline is 8 s: a resident that answers at 7.9 s is selected warm`
not-onboarded repo runs on the per-thread path with a named cold-fallback note pointing at onboarding (the fall-through is visible)[unit] src/execution/factory.test.ts::…::404 not-onboarded → per-thread path with a named cold-fallback note pointing at onboarding
exec parses streamed heartbeats + in-body errors; re-attach exactly once[unit] src/execution/resident.test.ts::ResidentExecutor.exec (streamed parse, non-zero-exit-as-result, needs:attach recovery, second-failure legibility, no blind retry)
read/write round-trip, 409 re-attach, path-escape legible[unit] src/execution/resident.test.ts::ResidentExecutor.readFile / writeFile
43: one runtime-replaced on /exec → the typed ExecSandboxRestartedError at once carrying the resident's words, waitedMs 0, no re-attach, the command is NOT re-run[unit] src/execution/resident.test.ts::ResidentExecutor.exec::runtime-replaced (a deploy mid-command) on /exec is the typed ExecSandboxRestartedError at once — no re-attach, the command is NEVER re-run (asserts the call sequence is exactly /exec)
43, the control-reset carve-out: a DO reset over a live container answers control-reset, distinct from runtime-replaced and never mistaken for it — the resident source scan proves the two error classes and the two reason builders are distinct and that run() routes a DO reset to ControlResetError before the runtime-replacement handling in both phases, so the reset path never says the word; the pure saysControlReset is exactly the resident's control-reset reason and never a replacement, a worktree-missing or a plain result[unit] deploy/cloudflare-resident/controlReset.test.ts::a DO code-update reset answers control-reset, distinct from runtime-replaced::*, src/execution/residentWake.test.ts::saysControlReset: a DO reset over a live container, distinct from a replacement::*
43, the client on a control reset: on /exec reason:"control-reset" is the typed ExecControlResetError at once — never a restart, never infra, the container unchanged and the outcome unknown; on /read and /write — the routes CONTROL_RESET_REISSUE_ROUTES names, /write a full-content put whose re-issue carries the whole content again — it re-attaches once and re-issues, a second reset being the unknown outcome (ExecControlResetError), never infra, any other route's reset the unknown outcome at once, and a needs:"attach" on the re-issued call the precise worktree-unavailable infra error — the op's one re-attach spent, never a generic status error[unit] src/execution/resident.test.ts::ResidentExecutor.exec::control-reset (a DO reset mid-command) on /exec is the typed ExecControlResetError at once — no re-attach, NOT a restart, NOT infra: the container is unchanged and the outcome is unknown, src/execution/resident.test.ts::ResidentExecutor.readFile / writeFile::a 409 control-reset on read (idempotent) re-attaches once and re-issues; a second one is the unknown outcome, ExecControlResetError — never infra, ::control-reset then needs:"attach" on the re-issued read…, ::a 409 control-reset on /write re-attaches once and re-issues the same full-content put…, ::the control-reset re-issue names its routes…
43, live: a control reset over a live pi across a resident WORKER-only deploy[agent] With a coding run in flight on a resident, deploy the resident Worker code alone (no image change) so its Durable Object resets under the live pi → the /exec in flight answers reason:"control-reset" (not runtime-replaced), the run keeps its card on the SAME pi (its page shows a resumed note for the control-plane reset, no sandbox_restarted note, relaunches: 0), and no second pi is orphaned (ps shows one pi for the run); owed after the next release
43: two runtime-replaced on /exec in a row are two typed restarts, never the streak's infra; on the idempotent routes a success between two resets the streak[unit] src/execution/resident.test.ts::ResidentExecutor.exec::two runtime-replaced /exec answers in a row are two typed restarts — the streak's infra error is for the idempotent routes, never for the word the harness keys on, ::a successful op between two runtime-replaced reads resets the streak (each is a one-off deploy)
43: one deploy on /exec is the typed restart, never infra[unit] src/execution/resident.test.ts::ResidentExecutor infra classification::a runtime-replaced on /exec (one deploy) is the typed restart, never infra
43, 27: the pure decision — runtime-replaced and the preflight's worktree-missing say the container under the thread is gone; evicted, not-attached, a rolling refusal, a bare mention of the word and a success do not; the message carries the resident's words and that the command was not run again[unit] src/execution/residentWake.test.ts::saysContainerGone: the answers that say the container under the thread is gone::*
27: worktree-missing on /exec is the typed restart at once — no re-attach, the command is never re-issued — while an evicted worktree keeps the one re-attach + retry[unit] src/execution/resident.test.ts::ResidentExecutor.exec::needs:"attach" saying the container disk was recycled (\worktree-missing`) on /exec is the typed ExecSandboxRestartedError at once — no re-attach, the command is never re-issued, ::needs:"attach" for an evicted worktree re-attaches once and retries the command`
43: idempotent routes retry once after re-attach; a second runtime-replaced is infra[unit] src/execution/resident.test.ts::ResidentExecutor.readFile / writeFile::a 409 runtime-replaced on read (idempotent) re-attaches once and retries; a second one is infra, ::runtime-replaced then needs:"attach" on the retried read (deploy + evicted worktree) is the precise worktree-unavailable infra error
43: the resident answers reason:"runtime-replaced" for a stale process handle / runtime_replaced interruption on exec, read, and write, and never re-issues a command the SDK does not vouch as unstarted[gap] resident-side classification (isRuntimeReplacement, two-phase run()) is typechecked (cd deploy/cloudflare-resident && npm run typecheck) but the resident Worker has no test harness; live proof: wrangler deploy (Worker code only, same image) while a /exec sleep 60 is in flight on repo:jshttp/vary → in-body {error:"runtime-replaced: …", reason:"runtime-replaced", exitCode:127}, the bot's tool result shows the re-check text, and the run continues (no ⚠️ abort)
43: a CONTAINER restart under a running command (not just an isolate swap) on /exec — the RuntimeIdentityInactiveError and container_stopped shapes vouch the runtime moved and are named runtime-replaced at once; the RPCTransportError loss kinds say only that the transport died, so they are the word only once the resident knows the container gone (the exit the monitor recorded, or a restore under way) and otherwise the SDK's own words with no reason, which the harness's one more command then decides (the third failure shape)[gap] live proof, distinct from the row above because a Worker-only deploy never exercises these: with a pi coding run's /exec sleep 120 in flight on repo:jshttp/vary, deploy the resident with a CHANGED container image (touch the Dockerfile, e.g. bump a comment, then wrangler deploy — the platform replaces the container, not only the isolate). Expected: the first failing /exec answers either reason:"runtime-replaced" with the SDK shape in its parenthetical (Runtime identity is no longer active, container_stopped, or RPCTransportError … peer_closed once the exit was recorded) or the SDK's bare words (Peer closed WebSocket: 1006 …, The container is not running, consider calling start()) with NO reason — never a bare 500; in the second case the run's page shows the one more command's harness_error notes (the wait through the restore window) followed by the sandbox_restarted verdict once the container answers the word or worktree-missing; either way pi is relaunched from the record and the run continues. Repeat once with the /read route mid-restart: the resident's re-attach + retry answers the file content on the second attempt (the idempotent routes keep the word).
43: the raw stopped-container spawn refusal (The container is not running, consider calling start()) — the container ROLL a deploy causes, distinct from the container_stopped interruption — classifies as runtime-replaced through the shared wording (on the message and its cause chain), and does NOT broaden to a genuine crash (container exited with unexpected exit code) or the readiness probe (the container is not listening)[unit] src/execution/residentRefresh.test.ts::RUNTIME_REPLACEMENT_WORDING …::classifies the stopped-container spawn refusal as a runtime replacement, ::matches on the cause chain too (the resident walks selfAndCauses), and is case-insensitive, ::does NOT broaden to genuinely-fatal container failures — a crash or the readiness probe stay ordinary failures
43, the /exec gate: every runtime replacement the exec route meets — the preflight's worktree probe and the command alike — goes through replacedExecAnswer and never straight to the word; the gate says the word when the SDK vouched the runtime moved or the resident knows the container it held is gone, and otherwise what the SDK said with no reason; what the resident knows is the container stop it saw (onStop, recorded with its exit code and reason, logged, the SDK's own reconciliation still run, cleared by a command the container answers) and a restore under way, never the platform's running flag; the SDK's vouching is the typed classes and RUNTIME_MOVED_WORDING, never the down words or a transport kind, while isRuntimeReplacement keeps the union for the resident's own steps; /read and /write keep the word[unit] deploy/cloudflare-resident/execReplacedWord.test.ts::*
43: the replacement wording is the union of two lists a reader tells apart — the SDK's words that vouch the runtime moved (RUNTIME_MOVED_WORDING) and the platform's words for a container that is merely down (STOPPED_CONTAINER_WORDING) — and the union's source, flags and reach are unchanged[unit] src/execution/residentRefresh.test.ts::RUNTIME_REPLACEMENT_WORDING (a deploy that ROLLS the container, not just swaps the isolate)::is the union of two lists a reader tells apart…
43: at the boundary the bug tripped — a container roll under /exec is the typed restart carrying the resident's words (no infra error, no re-attach); the bound still holds on the idempotent routes — a second roll with no success between is an ExecInfraError, a container that cannot come back[unit] src/execution/resident.test.ts::ResidentExecutor infra classification::a container roll under /exec is the typed restart carrying the resident's words — no infra error, no re-attach, ::bound: a container that cannot come back is infra on the idempotent routes — a second roll with no success between
attach-on-open surfaces needs-ref (name a branch) and not-onboarded legibly[unit] src/execution/resident.test.ts::ResidentExecutor.open (attach-on-open)
a successful attach records binding {ref, sha}; needs-ref carries the Worker's defaultRef (undefined from an older Worker)[unit] src/execution/resident.test.ts::records the attach result's ref@sha as the thread binding …, ::409 needs:"ref" carries the resident's defaultRef …
the warm path is named positively on the card (a debug note, routing-and-config.md item 28) — resident · <owner/name> · <ref>@<sha7>; bind-by-default re-attaches once on the resident's defaultRef and says so; a 409 without defaultRef still propagates ResidentNeedsRefError[unit] src/execution/factory.test.ts::warm probe → ResidentExecutor … (note asserted), ::needs-ref WITH the resident's defaultRef → re-attach once on that ref …, ::warm probe then a needs-ref attach failure → ResidentNeedsRefError propagates …
per-repo gate (item 26) — human open-when-absent, credentials explicitly scoped; configured allowlist refuses BY NAME with no executor[unit] src/config.test.ts::per-repo access (canUseRepo); src/core/dispatcher.test.ts::resident repo dispatch::a canUseRepo refusal is a named reply and no executor is created
fallback note reaches the status card AND the run's stream as a cold_sandbox note — after the attach, before the loop, head material[unit] src/core/dispatcher.test.ts::resident repo dispatch::a resident fallback note appears in the status frames and on the run's stream as a cold_sandbox note (named, never silent)
CLI stable thread key (ask --thread <key> > ephemeral)[unit] src/cli.test.ts::parseCliArgv — the \ask` built-in…; [agent]npx tsx src/cli.ts ask --thread cli:demo "help"` answers inline (flag parsed, no crash).
live executor round-trip against the deployed resident[agent] tsx script driving ResidentExecutor at repo:jshttp/vary: probeStatus → {state:"warm"}; open attaches; exec node -e → hi from resident as worker2 on master; write/read round-trip byte-identical; ls /no/such/dir → exit 2: result (not an error); /debug backdate-thread + sweep-now eviction then exec → transparent auto-re-attach (recreated worktree, clean git status); fresh-thread open without refHint → legible "name the branch" error.
resolver extracts explicit signals, degrades conservatively[unit] src/core/repoContext.test.ts — slug + head-of-ask on branch X, branch:X, Slack <url|label> unwrap, /tree/<ref>, PR URL / owner/name#N → head ref via mocked REST (auth header from resolveGithubToken when configured; fetch/HTTP failure and cross-fork heads degrade to repo-only), lowercasing, no-signal → {} with zero fetches, metacharacter/path tokens ignored whole, hostile ref tokens pattern-refused, the 20:05Z issue-2161 narrative fixture and prose on <word> bind no ref.
thread-established repo inherited; ask-once answer binds[unit] src/core/repoContext.test.ts::…thread history inheritance… — follow-up inherits the repo, branch:main/branch:fix/x bind against it while prose “on main”/“on fix/x” does not, current-message repo beats the thread's, assistant turns never establish a repo, PR URLs in history contribute repo with NO fetch.
the thread's own pull request binds the follow-up (item 29): the record's PR binds the ref to its head branch, the pr, the head sha and the base; a ref in the message wins over the head; a closed or unreachable PR → prUnpostable, no ref, no pr — and the record's own PR, merged or closed, also closedRecordPr with the merge-time head GitHub recorded for it — never the branch's later tip — and its head branch as headRef when the head lives on this repository (recordPrOf reads it as merged or closed with that headBranch; none for a headless one or a turn's PR); another repository's PR is not the thread's; the newer of a turn's PR and the run's wins, the run's when undated; a PR the message names beats both; the dispatcher hands the newest finished run's PR from its one thread read[unit] src/core/repoContext.test.ts::resolveRepoContext: the pull request the thread's own run opened binds the follow-up (resident-repos item 29)::*, src/core/dispatch/thread.test.ts::threadPrOf — the pull request the thread's work lives on (docs/reference/specs/resident-repos.md item 29)::*, src/core/dispatcher.test.ts::a follow-up seeds from its session (docs/reference/specs/session-log.md item 9)::the thread's pull request rides the target resolution…
A current PR #N in a thread with a recorded own PR selects N in that repository, ahead of older cited PRs. It is marked as the current message's target and as thread-owned only when N matches the durable record; a different N cannot silently adopt the old PR.[unit] src/core/repoContext.test.ts::resolveRepoContext: PR-source flags for ship::a bare PR number in a completed unit's thread selects its published PR over an older cited PR
A review request with a current bare PR #N and a vetted request, thread or channel repository pairs that number with the resolved repository and fetches its current PR head; an explicitly addressed slug refused by the probe stops resolution before the number can pair with an inherited repository or contextual URL; other agents retain the durable-thread or routed-task rule. A vetted addressed target and its bare PR number outrank a contextual repository or PR URL, including on follow-up; a refused address cannot fall back to that citation. A PR or plain repository URL introduced as reference material by the nearest action in its own clause, before or after a bare review PR, cannot replace its inherited target or supply a new one; an explicit repository URL remains the target even when a contextual PR link appears before or after it; a direct PR URL before a later contextual bare example remains the target even when that example has its own repository address; a separately requested later bare PR conflicts, while a preceding contextual PR URL cannot replace a later addressed or inherited bare review target, while two different PR targets without a context cue, including two direct PR links, require clarification even when the bare PR has an addressed repository; two different requested bare PR numbers require clarification, while an earlier or later contextual bare PR is ignored; an address attached to either contextual citation cannot supply or replace the bare review's repository, while a later explicit target clause still binds. A release review that says it packages merged component PRs keeps the directly requested PR; a later explicit review request for another PR still conflicts. A thread target or matching durable PR record outranks a different channel default for a bare review. A merely mentioned current-message slug or quoted PR URL/shorthand supplies no review target, even when the probe would accept it; a quoted example cannot replace the bare PR or its inherited repository.[unit] src/core/repoContext.test.ts::resolveRepoContext: explicit signals in the current message::a review pairs a bare PR number with a vetted repository target, without changing other agents' rule, ::a review never pairs a bare PR with an incidental current-message slug…, ::a review pairs a bare PR with an explicit address or a vetted inherited repository despite incidental context, ::an addressed review target outranks a contextual repository URL…, ::an addressed review PR outranks a contextual PR citation, ::a follow-up keeps the addressed PR instead of the contextual citation, ::a refused addressed review target cannot fall through to a contextual URL, ::a refused addressed review PR cannot fall through to a contextual PR citation, ::a contextual PR citation cannot replace an inherited review target, ::a contextual repository URL cannot supply a bare review PR's repository, ::a contextual repository URL cannot displace an inherited review target, ::a contextual repository URL in history cannot bind a later bare review PR, ::a contextual repository URL in history preserves a later bare review's inherited target, ::a preceding contextual repository URL cannot supply a bare review PR's repository, ::a preceding contextual repository URL cannot displace an inherited bare review target, ::a preceding direct repository URL still addresses a bare review PR, ::a direct repository URL before a bare PR remains the target, ::a contextual PR citation without an inherited target asks for the repository, ::a direct PR URL before a later bare reference remains the target, ::a release review keeps its direct PR when the description names merged component PRs, ::a later review request still conflicts after a packaged PR citation, ::a direct PR URL outranks a later addressed bare example, ::a later separately requested bare PR conflicts with an earlier direct target, ::a preceding contextual PR URL cannot override an addressed review target, ::a preceding contextual PR URL cannot override an inherited bare review, ::a preceding contextual PR address cannot supply a bare review PR's repository, ::a preceding contextual PR address cannot displace an inherited bare review target, ::an explicit repository URL survives a later contextual PR link, ::an explicit repository URL after contextual material addresses the bare review PR, ::two distinct direct PR links require clarification before a head fetch, ::two distinct direct PR shorthands require clarification, ::a later direct target conflicts with a bare target despite an earlier contextual link, ::a contextual PR link before a direct review link cannot replace it, ::a contextual PR link after a direct review link cannot create a conflict, ::an earlier see cue cannot hide a PR URL introduced by review, ::a later review action still makes a second direct PR a conflict, ::a contextual PR shorthand after a bare review target is not the target, ::a PR URL marked as context after the link is not the bare review target, ::a cue in an earlier clause cannot hide a later explicit PR URL, ::a cue after a clause break cannot excuse a conflicting PR URL, ::two different PRs with no context cue require clarification, ::an addressed bare PR does not bypass conflicting-PR clarification, ::an addressed bare PR conflicts with a different repository's same-number PR citation, ::two different bare review PR numbers require clarification before a head fetch, ::a second bare PR explicitly marked as context leaves the first target, ::compare with marks a second bare PR as contextual, ::a preceding contextual bare PR leaves the later requested PR as the target, ::an address on a preceding contextual bare PR cannot redirect the requested PR, ::an address attached to a contextual bare PR cannot supply the review target, ::an address attached to a contextual bare PR cannot replace an inherited target, ::an explicit thread repository outranks a different channel default for a bare review, ::a target address after a contextual PR clause wins over a channel default, ::a target address before a contextual bare PR still wins, ::an address in a contextual citation cannot redirect an inherited review…, ::a quoted example cannot replace a review's bare PR target…, ::a quoted PR URL alone never supplies a review target, ::addressed repos: … bind and rebind once the registry vets them::a refused explicit repository stops resolution before a bare PR can bind to the thread's old repository
A current PR #N can recover the thread's own completed publication even when a newer stopped pipeline or publication in the same thread belongs to another PR; only a unit with matching publication owner, branch, PR, and expected head supplies that record, and owner resolution retains the explicitly selected publication.[unit] src/core/dispatch/thread.test.ts::ownerOf and instanceOf — the thread's owner (record 0051's owner rule)::finds an older completed publication when a newer stopped pipeline shadows its thread, src/core/dispatcher.test.ts::a unit-owned thread (record 0051's reply-as-event and gone-instance rules)::an explicit PR number reaches target resolution through an older completed unit despite a newer stopped pipeline, ::an explicitly named older publication stays selected after a newer publication is resolved
A coordinator child with no run record selects PR #N when its routed task opens with that PR in an explicitly addressed repository. Every bindable bare number, shorthand, and PR URL in that request must agree with the routed target, including explicit references inside code spans; a conflict is refused before a workspace or model turn, never inherited from thread history. The child's resolved PR remains independent of its durable publication binding.[unit] src/core/repoContext.test.ts::resolveRepoContext: PR-source flags for ship::a ship child whose task opens with the existing PR binds that PR independently of its durable publication, ::a routed task citing two PRs refuses the conflicting target before inheriting a prior PR, ::a routed task refuses a conflicting PR URL even when the URL would otherwise be the strongest signal, ::a quoted PR URL cannot override a routed PR without triggering the conflict gate, ::a routed task refuses a PR URL in another repo and accepts its own PR URL; src/core/dispatch/authorize.test.ts::authorizeRepo — the repository gates, once the target has landed::refuses a routed task with conflicting PR references before starting a coding run
the not-onboarded gate's best guess (record 0054): one bounded registry listing — the probe's timeout, skipped inside an outage window and by a cold profile — and a rejected slug near ONE resident answers the corrected line to type and the evidence; a tie or an unanswered listing leaves the question without a guess[unit] src/core/dispatch/authorize.test.ts::authorizeRepo — the repository gates, once the target has landed::the not-onboarded gate guesses the one near resident…, ::a registry that does not answer leaves the question without a guess…; src/core/dispatcher.test.ts::resident repo dispatch::fresh thread + a rejected bare slug near ONE resident → one question…, ::two residents tie: the question stands without a guess…
24 (record 0054): the cold fall-through note names the resident the typed repo is near, when the registry listing answers with exactly one near match — silence or a tie leaves the note as the fix alone[unit] src/execution/factory.test.ts::makeExecutor resident selection::404 not-onboarded with the registry answering names the resident the typed repo is near (record 0054)
32 (record 0054): repo onboard reads the installation's repository list before the mint — a name the list does not hold with one near match carries a guess (the corrected line + evidence) on the refusal so renderRefusal can show Yes/No, instead of spending the mint; a name the list HOLDS that the mint still refuses reads the Worker's cause: policy and keeps the admin's way forward; a Worker body without cause renders as today[unit] src/core/commands/repo.test.ts::repo onboard::a name the installation does not hold with one near match carries a guess (record 0054): the refusal's text is the base sentence; the guess holds the corrected line and the evidence so renderRefusal shows Yes/No, ::a name the installation HOLDS that the mint still refuses…
a refused bare slug is REPORTED, not silently dropped — only when nothing bound[unit] src/core/repoContext.test.ts::bare prose slugs never hijack a thread…::*rejectedRepo* — fresh thread + rejected slug → { rejectedRepo } (no repo); thread weakly bound to a rejected repo → that repo on a signal-less follow-up; rejected on <slug> reported; an accepted candidate clears it; no probe → nothing rejected; the bound-thread payload tests still answer { repo } with no rejectedRepo. src/core/dispatcher.test.ts::resident repo dispatch::fresh thread + rejected bare slug + a repo-needing agent → one not-onboarded reply, no run… — one 📦 reply naming the slug, repo onboard <slug> and the URL alternative; zero model calls; makeExecutor never called; the ack card closes not started (repo not onboarded). ::the same rejected slug with a no-repo agent (general) runs unchanged… (resolver never called). ::a bound thread with a prose slug in the follow-up stays silent…. [agent] live: a fresh agent:coding in <owner>/<not-onboarded> thread gets the 📦 reply and no run/sandbox; a prose try/catch follow-up in a bound thread produces no message.
addressed repos — in <owner/name> vetted by the probe is STRONG (binds a fresh thread, rebinds a URL-bound one); a refused address changes nothing except a bare-PR review, which stops before fallback; a merely-mentioned onboarded slug and a code-spanned one stay weak; in <name> resolves through the registry listing only when exactly one onboarded repo carries the name, compared exactly and case-insensitively — never by substring, so an unrelated onboarded repo cannot shadow the match (ambiguous/unknown = prose; failed or absent listing = no answer; listed at most once per resolution); an address in history binds the follow-up; last strong wins across turns, while a current vetted address and bare PR outrank a contextual repository or PR URL; without a probe an addressed slug stays weak; repoFromThread honors vetted addressed slugs and ignores names[unit] src/core/repoContext.test.ts::addressed repos: … bind and rebind once the registry vets them::* (22), ::resolveRepoContext: explicit signals in the current message::an addressed review target outranks a contextual repository URL…
bare names only in the directive position (agent:coding in api: … binds; "the crash is in api" is prose); a slug is addressed anywhere[unit] src/core/repoContext.test.ts::addressed repos…::a bare name counts only in the directive position…
a determiner between in and the target is skipped and never itself addressed (in the atlas repo and in our atlas repository: address atlas; in the acme/web repo addresses the slug; in the logs addresses logs, an unknown name, so binds nothing; a determined name deep in prose is still prose)[unit] src/core/repoContext.test.ts::addressed repos…::a determiner between in and the target is skipped…
the start of a line is a compose position, and on [the] <name> repo (or repository) addresses too (*On the atlas repo:* heading a paragraph after a line of prose binds atlas on a fresh thread and a follow-up in that thread inherits it; in the atlas repo: at a line start binds; the same words mid-line are prose on a fresh and a bound thread; on <name> without the repo word and on the main branch bind no repo)[unit] src/core/repoContext.test.ts::addressed repos…::a repository named at the start of a line is addressed…
silence is not a refusal — an addressed slug the registry did not answer for → unverifiedRepo (bound or fresh thread; a throwing probe too; a URL beside it still wins except in a bare-PR review); an unanswered address in history is no answer; a fresh thread's unanswered weak slug → unverifiedRepo, not rejectedRepo; the dispatcher replies "couldn't verify", no model turn, no executor[unit] src/core/repoContext.test.ts::addressed repos…::the registry did not ANSWER for an addressed slug…, ::an unanswered address in HISTORY…; src/core/dispatcher.test.ts::…::unverified repo (registry unreachable) + a repo-needing agent → one could-not-verify reply, no run
the onboarded probe — true for any lifecycle state, false for 404 / non-transport HTTP errors, "unreachable" for transport failures and the outage window they open; undefined without config/bearer[unit] src/execution/factory.test.ts::residentOnboardedProbe::*
the registry listing for names — admin bearer, GET /residents, repo: prefixes stripped and lowercased; non-2xx, malformed body, a failure to reach the host (→ outage window), a deadline miss (→ no window; the next call fetches), no bearer/config → undefined[unit] src/execution/factory.test.ts::residentSlugsLister::*
the card names the bound repo — resident · <owner/name> · <ref>@<sha7> on the warm and non-warm attach notes, a debug note on the label[unit] src/execution/factory.test.ts::makeExecutor resident selection::warm probe → ResidentExecutor…, ::refreshing probe…, ::degraded probe…; src/core/dispatcher.test.ts::…::needs-ref WITH a defaultRef…
production default resolver wired into dispatch (no injection)[unit] src/core/dispatcher.test.ts::repo/ref resolution + resident prompt selection …::the production default resolver (no injection) extracts repo/ref from the head-of-ask routing clause
needs-ref WITH defaultRef → bound to the repo default, no question, the run proceeds, the ✅ title carries resident · <owner/name> · <ref>@<sha7> (repo default — no branch named) at debug (a debug note, routing-and-config.md item 28)[unit] src/core/dispatcher.test.ts::…::needs-ref WITH a defaultRef → bound to the repo default with a loud note, no question, the run proceeds. [agent] agent:coding on an onboarded repo in a fresh thread naming no branch → no 🌿 question; the card names the default branch.
needs-ref WITHOUT defaultRef → ONE question, zero model calls; the ack card closes as not started (no run card)[unit] src/core/dispatcher.test.ts::…::needs-ref from attach → ONE clarifying question; no model turn; the ack card closes as not started; typed error: src/execution/resident.test.ts::409 needs:"ref" is a TYPED error…
the thread answer rebinds via re-attach and runs[unit] src/core/dispatcher.test.ts::…::the thread answer "branch:main" rebinds via re-attach and runs (attach body carries refHint:"main" from the typed follow-up + the history-inherited repo)
resident prompt variant reaches the provider; fallback prompt unchanged[unit] src/core/dispatcher.test.ts::…::a resident run gets the agent's resident system variant naming the repo, ::the per-thread fallback path keeps the agent's own system prompt (regression); content pinned by src/agents/registry.test.ts::resident prompt variants (ready worktree, no clone/install/gh instructions, push-then-submit — ::coding variant pushes the branch and submits the description; PR creation is not its job — fallback prompts untouched)
canManageRepos is fail-closed[unit] src/config.test.ts::repo management gate (canManageRepos) — no repo:write grant → non-admins refused, admins pass; a repo:write grant admits its holder + admins; no admin at all still closed.
non-admin repo management is refused by the registry gate; repo list stays open and mutating verbs use inline run records[unit] src/core/commands/repo.test.ts::gates … and scopes, src/core/commands/repo.test.ts::repo.list::*, src/core/dispatcher.test.ts::repo management commands …::*, src/core/dispatcher.test.ts::inline command runs + run receipts …::*
onboard — Node defaults + main, --ref / quoted --test/--build/--install overrides (the shared grammar; the tokenizer normalizes smart quotes), slug lowercased, invalid slug / hostile ref / unknown option each refused BEFORE any client call without echoing the value, resident 4xx relayed as conflict/not_found/invalid_input with HTTP <n>: <error>, the onboard warning surfaced[unit] src/core/commands/repo.test.ts::repo onboard::*
dry-run replies render the itemized plans[unit] src/core/commands/repo.test.ts::repo offboard / rebuild (--dry-run)::* — dryRun:true reaches the client; replies carry counts/ids and "Nothing was changed"; unknown flags are usage errors; a 404 is not_found.
reconfigure merges onto the live command table[unit] src/core/commands/repo.test.ts::repo reconfigure::* — partial patch merged onto /residents' current table (whole-table replacement semantics preserved); ref-only patch skips the fetch; nothing to change → invalid_input; not onboarded → not_found.
offboard dry-run — itemized plan, resident fully intact[agent] POST /offboard {"resource":"repo:jshttp/fresh","dryRun":true} → 200 {"dryRun":true,"wouldRemove":{"registryRecord":true,"schedules":0,"snapshotBackupIds":[2 ids],"backupObjects":4,"r2Objects":0,"threadBindings":0,"container":"warm"}} (schedules is 0 on a settled resident: provisioning's two rows are its only timer); immediately after, /status → warm and refresh.instance unchanged.
rebuild dry-run — plan only, still warm[agent] POST /rebuild {"resource":"repo:jshttp/vary","dryRun":true} → 200 plan: from {state:"warm"}, discards = the current snapshot stamp + ids + backupObjects:4, reprovision {defaultRef:"master", provisioningTimeoutMs:600000}, keeps {registryRecord:true, threadBindings:7}; /status after → warm, snapshot ids unchanged.
real rebuild — down→onboarding→warm with fresh snapshots[agent] POST /rebuild {"resource":"repo:jshttp/vary"} → 202 {…, backupObjectsDeleted:4, state:"onboarding"}; /status polled → onboarding → warm; /debug info → NEW backup ids, fresh provisionedAt, lastRestore:null (a true re-provision, not a restore), the next cron firing creates its first refresh instance, thread bindings retained.
rebuild refused mid-engine[agent] POST /rebuild on a resource in state onboarding → 409 {"error":"rebuild-refused: the engine is mid-flight (state onboarding) — …"} (e.g. right after its onboard).
onboard installation check skips HONESTLY when the App is unconfigured[agent] With GITHUB_APP_* unset: onboard → 202 carrying warning:"github-app-not-configured: installation membership was NOT verified …"; provisioning proceeds to warm. With the secrets set: onboard of a private repo IN the installation → 202 with NO warning and reaches warm; onboard of a repo NOT in the installation (e.g. repo:jshttp/fresh, public, App installed on the org only) → 403 {"error":"not-in-installation: the GitHub App cannot mint a token scoped to repo:jshttp/fresh — install the App on the repository first (github-token-mint-failed: HTTP 422 …)"}; /status → not onboarded, /residents count unchanged (no slot consumed).
36: the down transition rebuilds at once with budget left — the instant recorded, the count named in the reason; every rehydration reason is eligible, a provision failure never[unit] src/execution/residentAutoRebuild.test.ts::autoRebuildDecision (the down transition decides; the watchdog is the backstop)::a rehydration down with an empty history is rebuilt at once… + ::…::every reason only a rebuild can escape is eligible…
36: the budget — spent at the second rebuild inside the window, the reason stamped with the count and the one way back (never a reopen instant); entries outside the window pruned before judging and before writing back; a stamped reason never judged again; malformed history dropped[unit] src/execution/residentAutoRebuild.test.ts::autoRebuildDecision (the down transition decides; the watchdog is the backstop)::the budget is spent at AUTO_REBUILD_BUDGET rebuilds… + ::…::a rebuild older than the window no longer counts… + ::…::a reason already stamped budget-spent is never judged again… + ::…::malformed history entries are dropped…
36: the Worker wiring — goDown judges after the down write and before the error; one reader and writer of the history row, deciding through the pure module and rebuilding as auto; a person's rebuild clears the history, the automatic keeps it; the watchdog's down branch is the same decision and reports auto-rebuilt; no strike counter survives and a serving pass keeps the history; force-down fires the transition on request; /debug info carries autoRebuilds[unit] deploy/cloudflare-resident/autoRebuild.test.ts::the down transition is the trigger + ::the watchdog is the backstop, never a strike counter + ::the fault injection and the read view
36: the firing record counts the residents a pass rebuilt, only when there were any[unit] src/core/schedules.test.ts::watchdogFiring (the resident's firing record)::item 36: the residents a pass auto-rebuilt…
36 live (the transition and the budget): POST /debug {"op":"force-down","resource":"repo:jshttp/fresh","transition":true} on a warm resident answers onboarding already (the log: auto-rebuild (transition): auto-rebuild (1 of 2 in 24 h): r2-restore-failed: injected …); polled to warm with fresh snapshot ids; {"op":"info"} autoRebuilds holds one instant. A second → onboarding, two instants. A third → stays down, reason ending — auto-rebuild budget spent (2 in 24 h) — repo rebuild resets it; {"op":"run-watchdog"} leaves it (action:"none"); POST /rebuild → onboarding and autoRebuilds empty[agent] after the deploy, on a small public resident (repo:jshttp/fresh)
36 live (the backstop): bare force-down → down (r2-restore-failed: injected …); one {"op":"run-watchdog"} → action:"auto-rebuilt", reason auto-rebuild (watchdog): …, state onboarding; polled to warm[agent] same resident, the budget reset by a /rebuild first
61: a killed extract — exit 143 / SIGTERM wording → restore-interrupted; a probe that found the runtime gone → restore-interrupted: … — the container stopped under the restore whatever the exit; a live runtime or no probe → down; (timed out) wins[unit] src/execution/residentRefresh.test.ts::restoreFailureDisposition (a restore the runtime replacement interrupts is retried, never down)::an extract killed by SIGTERM… + ::…::a runtime the Worker probed and found stopped…
61: the instance step reads the wake path's restore-interrupted: verdict as an interruption — at the start of the message only[unit] src/execution/residentRefresh.test.ts::classifyRefreshFailure (a build SIGTERM'd by a deploy is an interruption, not evidence)::a wake path's … verdict (the probe found the container gone) is an interruption for the instance step too…
61: the Worker wiring — the runtime probed after any failure that was not a typed replacement, both facts handed to the disposition, the verdict thrown as a StepError carrying the reason[unit] deploy/cloudflare-resident/autoRebuild.test.ts::the wake path reads a killed extract the way the instance step does
43: The container just exited is a runtime replacement — a roll mid-cycle is refresh-interrupted (retried), a roll under a restore is restore-interrupted, never a counted step failure[unit] src/execution/residentRefresh.test.ts::RUNTIME_REPLACEMENT_WORDING (a deploy that ROLLS the container, not just swaps the isolate)::classifies … as a runtime replacement too, so a roll mid-cycle is an interruption the engine retries, never a counted step failure
67: which step is whose — the command-table steps are the repository's, the stale-process sweep and every other step the resident's; the step is read off a <step>-failed: reason; a reason parks on repeat only when it is evidence about the repo or GitHub[unit] src/execution/residentInfraStreak.test.ts::which step is whose
67: the ladder — count at 1–2, recreate at 3, one cycle for the recreated container, down at ≥ 5; the reasons name the count, the step, the span and what happens next[unit] src/execution/residentInfraStreak.test.ts::the ladder over consecutive resident-step failures
67: the Worker wiring — one writer of the row and its erasers; refreshFailed parks a repo step and climbs for a resident step (count → degraded, recreate → recreateContainer, down → destroy, forget the identity, clear the row, goDown); disk-full first; infra-streak: is non-evidence and the gate parks only what parksOnRepeat allows; the fault injection is admin-only; the read view carries the row with its rung[unit] deploy/cloudflare-resident/infraStreak.test.ts::the count lives in storage under one key + ::refreshFailed climbs the ladder for resident steps and parks for repo steps + ::the reasons are read as the resident's, never the repo's + ::the fault injection and the read view
67: a fetch that failed on the mirror is the resident's (fetch-failed, counted); GitHub's failures — DNS, timeouts, HTTP errors, credentials, a repository GitHub says is not there — stay github-unreachable[unit] src/execution/residentRefresh.test.ts::fetchFailureIsMirrors (a fetch that fails on the mirror is the resident's, not GitHub's) + deploy/cloudflare-resident/infraStreak.test.ts::the fetch step names GitHub only for GitHub's failures
67: infra-streak is a rehydration-flavored down, so item 36's transition rebuilds it[unit] src/execution/residentAutoRebuild.test.ts::autoRebuildDecision (the down transition decides; the watchdog is the backstop)::every reason only a rebuild can escape is eligible…
67: the classifier carries the failing step on every outcome[unit] src/execution/residentRefresh.test.ts::classifyRefreshFailure (a build SIGTERM'd by a deploy is an interruption, not evidence)::an ordinary build failure stays <step>-failed with the message verbatim
67 live (the recreate rung): on a throwaway resident KEPT AWAKE by a live attach (an operator POST /attach on a thread, not detached — an idle resident's cycle parks at the gate before the fetch, item 16b, and the fault is never reached), {"op":"infra-streak","count":2}, then {"op":"break-mirror"} → {broken:true} (the mirror's config gone; objects/ and the ready stamp intact, so the wake check passes and the fetch fails), then a cycle ({"op":"refresh-now"} answering created, or the next cron bucket) → the fetch fails fetch-failed: exit 128: … 'origin' does not appear to be a git repository (never github-unreachable), the log reads infra-streak: cycle 3 of 5 failed in the resident's own step fetch then recreate: destroying the container …; /status reads degraded (infra-streak: 3 consecutive cycles … the container was destroyed …); the following cycle restores from the snapshot and completes → warm, /debug info infraStreak: null[agent] after the deploy
67 live (the down rung): same resident, still attached, {"op":"infra-streak","count":4}, break-mirror, a cycle (wait for the previous instance to finish — refresh-now beside a live cycle answers skipped) → the transition sees down (infra-streak: 5 … rebuilding) and /status reads onboarding at once, then warm with fresh snapshot ids; autoRebuilds gained an instant; infraStreak: null[agent] same resident
67 live (the repository side unchanged): needs a head that moves — an unchanged default branch skips the install and build steps (planRefresh → unchanged), so a reconfigured command table proves nothing until the repository's head advances. The failing command must be build: it runs on every rebuild, while install runs only when the committed lockfile's key changes (planRefresh → keep-deps), so install: "false" never executes on a repository whose lockfile did not move. On a throwaway resident kept awake by a live attach: reconfigure to build: "false", move the head (a genuine commit on the throwaway repository), a cycle → degraded (build-failed …) with the last good snapshot kept and infraStreak: null; reconfigure back, a cycle → warm at the new head[agent] when the throwaway repository's head moves
43/#566: the raw stopped-container spawn refusal (The container is not running, consider calling start()) — the container ROLL a deploy causes, distinct from the container_stopped interruption — classifies as runtime-replaced through the shared wording (on the message and its cause chain), and does NOT broaden to a genuine crash (container exited with unexpected exit code) or the readiness probe (the container is not listening)[unit] src/execution/residentRefresh.test.ts::RUNTIME_REPLACEMENT_WORDING (a deploy that ROLLS…::classifies the stopped-container spawn refusal as a runtime replacement, ::matches on the cause chain too (the resident walks selfAndCauses), and is case-insensitive, ::does NOT broaden to genuinely-fatal container failures — a crash or the readiness probe stay ordinary failures
43/#566: at the boundary the bug tripped — a container roll under /exec is the typed restart carrying the resident's words (no infra error, no re-attach); the bound still holds on the idempotent routes — a second roll with no success between is an ExecInfraError, a container that cannot come back[unit] src/execution/resident.test.ts::ResidentExecutor infra classification::a container roll under /exec is the typed restart carrying the resident's words — no infra error, no re-attach, ::bound: a container that cannot come back is infra on the idempotent routes — a second roll with no success between
Residents dash routes: /residents[/] index; /residents/<owner>/<name>[/] detail (lowercased); non-slug / traversal / extra segments / other paths → null (fall through)[unit] src/channels/residentsView.test.ts::parseResidentsRoute::*
Residents index: one fold per resident with state dot/reason/ref/short sha, a detail ↗ link to its page and the count of runs on it; the toolbar counts the runs on the listed residents; state→tone mapping for every lifecycle state; empty state names repo onboard; hostile slug/reason render as text and a non-slug resource never becomes a link; shared nav marks Residents current; the page opens /residents?stream=1[unit] web/src/pages/residents.test.ts::ResidentsIndexPage::*, src/channels/residentsView.test.ts::residentStateTone (escaping red-verified via the seed-island test)
42 fold: a resident's fold is one row per tree, each reading run · tree · clock — a row per live run (agent chip, the request, who started it and the thread link under it; a ticking running for with the start on hover and open ↗ with the run's token at the right) joined by thread key to its worktree binding as a monospace block beside it (ref and sha linked to its commit when hex over deps mechanism · the tree's own bytes from the disk sample), and a dimmed row per live binding no run is using (idle, its last use, no clock, no link) — evicted bindings never; a run whose attach has not completed reads no worktree bound yet; a run without an http(s) thread URL names its surface; the summary's facts line keeps its separators and carries the refresh and disk sample stamps on hover; the disk line carries free space (capped under diskBudgetMb when set), the room in more trees and the sample age; an unmeasured resident says so in the size cell and the disk line; ?open=<slug> opens a fold on first paint; hostile thread fields render as text and a non-hex sha never links[unit] web/src/pages/residents.test.ts::ResidentsIndexPage — the fold (item 42: what is on this resident)::*
42 live: the index keeps itself current from the feed — connection state (connecting/live/disconnected), an upsert adds a run to its resident's fold, a finished upsert or a removed takes it out, a residents frame replaces the listing (state, bindings, disk, count), the tab title carries the running count and the favicon the fleet's tone, a reconnect reloads, unmount closes the stream[unit] web/src/pages/residents.test.ts::ResidentsIndexPage — live over the feed::*
42 model: residentThreads normalizes the bindings newest-attach first; runsOnResident picks the live rows naming the slug, oldest first; bindingFor joins a run to its live (never evicted) binding by thread key; threadTreeKiB reads a tree's bytes from the sample; diskHeadroom is the admission's arithmetic (free under the cap, reserve, room per tree kind); applyResidentsFrame is the page's reducer and ignores a malformed frame[unit] src/channels/residentsModel.test.ts
42 feed: serveResidentsFeed opens with the SSE head and replays the viewer's repo runs; a run without a repo never reaches the page; a repo run's create/finish/eviction are forwarded as the runs index's upsert/removed; the listing is re-read and pushed whole when a repo run's dispatch.workspace.attach span ends and when its stream seals — never on an ordinary event or a finish alone; a periodic listing read also carries drain and image-report changes when no run event fires; reads are coalesced to one in flight plus one queued; a failed read sends nothing; a disconnect releases the index subscription and every per-run watch[unit] src/channels/residentsFeed.test.ts::serveResidentsFeed::*, ::isAttachEnd::*
Residents index tab favicon = the fleet's worst tone: red over amber over green; green only when every resident is warm; grey for no residents and for an unknown/unreachable resident with nothing worse to show (a non-record entry counts as unknown); rendered by the shell from the seed with four distinct dot URIs[unit] src/channels/residentsView.test.ts::residentsFleetTone::*, src/channels/webShell.test.ts::renderShell::the residents index wears the fleet's worst dot from the seed: red over amber over green, grey when empty or unknown
Resident detail: thread worktrees newest-attach first with sha linked to its commit (hex only), deps mechanism, times, evicted marker + N live · M evicted summary; "no thread worktrees" when none; hostile thread fields render as text[unit] web/src/pages/residents.test.ts::ResidentDetailPage::lists thread worktrees (ref, sha linked to its commit, deps mechanism, attach times), newest first, marking evicted ones, ::says so when a resident has no thread worktrees, and never links a non-hex sha
Attach records the sha on the binding; /residents live view carries threads[agent] After a run attaches, the resident detail's row for that thread shows its short sha linked to the commit and the deps mechanism (e.g. hardlink).
Resident detail: state, ref, sha (linked to the GitHub commit only when a sha exists), lockfile hash, provisioned/refreshed, snapshot stamp + backup ids (or "no snapshot"), schedules, command table + effects, worktree TTL; failure reason + last refresh error for a down resident; {error} live view → "unreachable" grey; hostile command strings render as text; back link to /residents[unit] web/src/pages/residents.test.ts::ResidentDetailPage::*
Residents handler: ignores non-/residents paths; reads the admin registry LIVE on every request (call count asserted); serves index/detail with the /runs CSP + X-Frame-Options: DENY + no-store; unknown slug → 404 "not onboarded"; non-GET → 405 allow: GET; no client → 503 naming execution.resident; upstream non-200 or throw → 502 with the upstream status/reason[unit] src/channels/residentsView.test.ts::createResidentsViewHandler::*
42 image readiness: the index and detail keep lifecycle state distinct from the container image report, show pending/current/unknown without treating warm as image proof, and the index names a drain and its held repos from the live listing even if no resident row is present; the index seed and listing feed frames carry both[unit] web/src/pages/residents.test.ts::ResidentsIndexPage::shows the fleet drain and image reports beside warm state, ::ResidentsIndexPage::shows a deploy drain even when the resident listing is empty, ::ResidentDetailPage::shows the container image report separately from lifecycle, src/channels/residentsModel.test.ts::residents image readiness::*, src/channels/residentsView.test.ts::createResidentsViewHandler::carries the live deploy drain into the index seed
42 handler, the runs and the feed: the index seed carries the server clock and the registry's live runs that name a repo under the viewer's runs:read predicate, each with its token — never a finished run, one without a repo, or one the viewer may not read (a viewer admitted to nothing sees the residents and no runs); ?stream=1 on the index is the feed: an SSE head, the viewer's repo runs replayed, live upserts, and a fresh listing when a run's attach span ends, opened without a listing read of its own; each of the feed's listing reads is a dashboard.residents root with route feed[unit] src/channels/residentsView.test.ts::createResidentsViewHandler — the runs on residents and the live feed (resident-repos item 42)::*
Residents dash behind Access: with SSO configured, /residents without a valid Cf-Access-Jwt-Assertion → 403; with it → the live list of onboarded repos; a detail page links to the GitHub commit[agent] Open https://<PUBLIC_BASE_URL>/residents signed in → rows match repo list; curl -i https://<PUBLIC_BASE_URL>/residents (no cookie/JWT) → 403.
end-to-end chat commands against the LIVE resident[agent] tsx script driving dispatch() (real config-command path, provider poisoned to throw) with execution.resident at the real base URL and RESIDENT_ADMIN_TOKEN in env: repo list (non-admin) → *Resident repos* (1/6): • jshttp/vary — warm · ref master · sha 1220b9c4 …; repo offboard jshttp/vary --dry-run (admin) → the itemized plan reply ending "Nothing was changed"; repo onboard acme/api (non-admin) → 🚫 naming the admin; repo rebuild jshttp/vary --dry-run (admin) → the rebuild plan reply. Zero provider calls, zero status cards; /status after → jshttp/vary still warm.
typed `repo testbuild` binds optional refs through the registry schema, applies authorization, and refuses hostile input before any backend
/op runs the table command in a disposable checkout with warm deps[agent] POST /op {"resource":"repo:jshttp/vary","op":"test","ref":"master"} (operator bearer) → streamed {ok:true, …, sha:"1220b9c4…", summary:"test passed on repo:jshttp/vary @ master (1220b9c4) in 5s", deps:"hardlink", reconciled:false, durationMs:<ms>} — deps from the shared lockfile-keyed cache, no install (durationMs a fraction of provisioning).
status op = no checkout, lifecycle + sha + lastRefresh[agent] POST /op {"op":"status"} → {ok:true, state:"warm", ref:"master", sha:"1220b9c4…", lastRefreshAt:"<iso>", summary:"repo:jshttp/vary is warm on master @ 1220b9c4, …"} — instant, no container exec.
op on a concurrently-attached ref leaves the thread worktree untouched; checkout deleted after[agent] Attached slack:u6iso on master (worker3), planted u6-marker.txt; ran /op test on the same ref; after: thread /exec shows the marker intact, git status --porcelain -uno empty, and ls -A /workspace/ops/ | wc -l → 0 (per-op checkout deleted). Evict the binding afterwards via backdate+sweep to remove the tree; its UID remains spent until VM destruction.
38: the server-side /op budget is the exec ceiling, matching the client's /op wait — never the 5-minute per-command default[unit] deploy/cloudflare-resident/opExecTimeout.test.ts::op exec budget (resident-repos item 38)::bounds an /op run at the exec ceiling, matching the client's /op wait — never the 5-minute per-command default and ::the op run itself is capped by OP_EXEC_TIMEOUT_MS; src/execution/resident.test.ts::ResidentOperations.run::bounds the /op request with an AbortSignal.timeout at the exec ceiling…
a failing op is a result, not an infrastructure error; non-onboarded and invalid refs are named outcomes[unit] src/execution/resident.test.ts::ResidentOperations.run::a failing op is a RESULT…, src/execution/executor.test.ts::LocalOperations::a failing test run is a RESULT…, src/core/commands/repo.test.ts::repo test / repo build …::*
a mutating command-table entry is refused by the resident[agent] Live via reconfigure: mark test mutating, observe op-refused, then restore and verify the op passes
op on a ref whose lockfile differs reconciles via the shared cache[agent] Mechanism row: /op materializes deps through the EXACT materializeThreadDeps used by attach (same-key cp -al proven live above via deps:"hardlink", reconciled:false; the differing-key scoped-install branch proven live in the attach checks above — deps:"install", reconciled:true), and the op response carries deps/reconciled as evidence. Not separately exercisable on jshttp/vary (single branch, no committed lockfile → every ref shares one key).
LocalOperations (dev/CLI) is a real second implementation[unit] src/execution/executor.test.ts::LocalOperations — status/existence, npm test pass+fail, --if-present build, ref-ignored honesty; wired as the local-execution default in the dispatcher (defaultOperations).
live zero-setup dispatch against the deployed resident[agent] tsx script calling dispatch() with a scripted provider (one bash tool_use; no ANTHROPIC_API_KEY in the check environment) and the real resident config. (1) agent:coding on <default branch> in <onboarded repo>: run node -e "console.log(1)" and report on a fresh thread → the production default resolver extracts {repo, ref} from the text, the executor is the ResidentExecutor, the provider receives the resident system variant (ends Target repository: <repo>. The worktree is already on this thread's bound branch …), and the ONLY [tool] log line is the scripted command itself — zero setup commands; the exec runs in /workspace/threads/<thread>/<ref> as a pool user on that branch. (2) Fresh thread, repo but no ref (agent:coding in <repo>: say hello): with a defaultRef configured for the repo the run binds to it with no question and the ✅ title carries (repo default — no branch named) (item 30); with no defaultRef → the 🌿 ask-once question naming the repo, ZERO provider calls, no status card, and the thread answer on <branch> (history carrying the original ask) inherits the repo, /attach binds that branch, and the run executes in a fresh worktree as another pool user. Evict the bindings afterwards via backdate-thread + sweep-now to remove the trees; their UIDs remain spent until VM destruction.
45: ref fate from the pulls list — open wins, merged beats closed, none → no-pr; the REST body is parsed defensively (non-array → null, malformed elements dropped)[unit] deploy/cloudflare-resident/gc.test.ts::pullsFate*, ::parsePullsBody* (red-verified: module absent)
45: merged, closed or gone refs select candidates; item 71 still checks the exact owner and private tree before removal[unit] gc.test.ts::reclaimDecision*, deploy/cloudflare-resident/pausedWorkspace.test.ts::one preservation guard covers automatic loss paths::*
45: live — a thread whose run is attached and not yet released keeps its tree through a reclaim pass whatever its branch's fate: POST /attach a thread on a branch the mirror does not hold (a new unpushed one), then POST /debug {"op":"reclaim-now"} → the thread is in kept with why:"run-held", its tree still on disk; after POST /detach the next pass reclaims it as gone[agent] the reclaim-now and threads debug ops of item 13 on a throwaway resident
45: live — a review thread bound to a PR branch is reclaimed within one refresh cycle of the PR merging (or its branch being deleted), its live pool-user allocation released while the UID stays spent until confirmed VM destruction, the binding kept with evictedWhy[agent] On repo:<owner>/<name>: attach a thread on an open PR's head branch (POST /attach {"threadKey":"cli:gc50","refHint":"<branch>"}), merge/close the PR or delete the branch, then POST /debug {"op":"reclaim-now","resource":"repo:<owner>/<name>"} → reclaimed:[{"threadKey":"cli:gc50","ref":"<branch>","why":"gone"|"merged (pr merged)"}]; /debug threads shows the binding evicted:true, user:"", evictedWhy set; the dash row reads evicted <ts> · <why>. Negative control: a thread bound to main (default ref) or to a branch with an OPEN PR is in kept with why:"default-ref" / "pr-open". The unforced path is the same function at the end of every onRefreshAlarm (≤ 10 min when awake).
45: a finished ref with unsaved or uncertain private work remains bound; exact terminal saved or clean evidence allows reclaim[unit] deploy/cloudflare-resident/pausedWorkspace.test.ts::paused unpublished work at resident removal::*
46: LRU pick — coldest by last activity (attach or provisioning, evicted bindings count); floor keeps a just-used resident; live worktree / non-warm / busy / unknown in-flight / failed view are each rejected by name; ties deterministic; empty fleet total[unit] gc.test.ts::pickEvictionCandidate*
46: chat — --evict-coldest sends evictColdest:true; the 202 evicted block renders as a ♻️ line naming the resident + last use; a 429 with rejected renders per-resident bullets; the flag is refused on repo reconfigure[unit] src/core/commands/repo.test.ts::repo onboard::--evict-coldest opts the onboard into LRU eviction …, ::an over-cap onboard with no eligible resident relays the per-resident reasons, ::--evict-coldest is an onboard-only flag: reconfigure refuses it as an unknown option
46: dash shows the eviction reason on an evicted thread row, as text[unit] web/src/pages/residents.test.ts::ResidentDetailPage::lists thread worktrees … marking evicted ones (asserts evicted <ts> · <why> with a <b> payload staying text)
46: live — over the cap WITHOUT the flag → 429 naming the opt-in and nothing offboarded; WITH the flag → the coldest eligible resident is offboarded and the new one onboards; no eligible candidate → 429 with rejected reasons[agent] (with the effective cap lowered to the current fleet size via item 49's set-test-overrides, so the existing fleet IS the full cap; fillers must be slugs of the installation's owner because the installation-scoped mint refuses others, and each is a billable container; the example fleet below is <owner>/<repo> + jshttp/vary) repo onboard <owner>/<other> → ⚠️ … HTTP 429 … resident cap reached (2/2) … evictColdest:true …, repo list unchanged; repo onboard <owner>/<other> --evict-coldest while <repo> is active and vary is degraded → 429 with rejected[] naming both reasons (active Nm ago (floor 60m) / N live worktree(s), state degraded); then repo rebuild jshttp/vary, leave it untouched for the 1 h floor, and repo onboard <owner>/<other> --evict-coldest → 🏗️ Onboarding … ♻️ Made room: evicted \jshttp/vary` …, repo list2/2 with the newcomer,vary's /status→ 404.evictColdest:"yes"→400 evictColdest must be a booleanbefore any registry write; a duplicate onboard WITHevictColdest:true→409 already onboarded`, nothing evicted (the flag is inert off the 429 path). The floor BOUNDARY itself is item 49's live row.
49: override parsing — lower-or-equal accepted, neither field = clear, raising above the constant / non-integers / zero cap / negatives → named 400 text[unit] deploy/cloudflare-resident/gc.test.ts::parseTestOverrides* (red-verified)
49: effective limits — defaults with no override; only the named fields lowered; a record from another build ignored (stale-build); stored values above the constant clamped[unit] gc.test.ts::effectiveLimits* (red-verified)
49: repo list names an active override (enforced vs default cap, floor, when set, how to clear); no warning without one[unit] src/core/commands/repo.test.ts::repo.list::an empty registry and an active test override keep their wording
49: live — set cap:1 on a 1-resident fleet → the next onboard 429s at (1/1); GET /residents shows cap:1, capDefault:5, testOverrides{…}; read token on the op → 403; cap:4 is accepted as a lowering override; cap:6 → 400; clear → /residents cap:5, no testOverrides; and with floorS:600 + a fresh filler the boundary is provable in minutes: --evict-coldest at ~8 min → rejected … active 8m ago (floor 10m), at ~11 min → the filler is evicted[agent] curl -X POST -H "Authorization: Bearer $ADMIN" .../debug -d '{"op":"set-test-overrides","cap":1,"floorS":600}' then the probes above with repo:<owner>/<small repo> (test:"true", build:"true") as the filler; finish with -d '{"op":"set-test-overrides"}' to clear and verify /residents (cap:5, testOverrides:null). Expected shapes: read token → 403 forbidden: admin scope required; {"cap":4} → 200 {cap:4, capDefault:5, …}; {"cap":6} → 400 cap must be an integer between 1 and 5 …; a set → 200 {cap, capDefault:5, floorS, floorDefaultS:3600, override:{…, build:"<marker>"}}; under the floor → 429 rejected:[{<filler>, "active Nm ago (floor Mm)"}, …], past it → 202 evicted:{resource:<filler>, lastActivityAt, …backupObjectsDeleted:4, errors:[]}.
46: the flag alone never changes the atomic cap semantics — without 429 it is inert; the slot swap is one input-gated registry section and nothing is destroyed before the newcomer holds the slot[code review] handleOnboard: the eviction branch is entered only on result.status === 429 && evictColdest; its only registry mutation is registry.replace(victim, record) (delete + put under the DO input gate, cap re-checked); teardownResident runs strictly after a successful swap.
50: readonly body field parses absent→false, boolean→itself, anything else→named 400[unit] src/execution/residentReadonly.test.ts::parseReadonly…
50: the pure plan's read-only mode requests scrub and mirror origin; its writable credential flag and GitHub origin are legacy fallbacks overridden by the Worker's Git door; mode switch on a live tree (both directions, and pre-field bindings count as writable); evicted prior never switches; reused read-only tree still scrubbed[unit] src/execution/residentReadonly.test.ts::planReadonlyAttach
50: the bot sends readonly: true iff the run's identity is read (review), omits the field otherwise (coding); the release mode pairs with the same identity[unit] src/execution/factory.test.ts::a read-identity agent attaches with readonly:true…, src/execution/resident.test.ts::sends readonly:true in the attach body…, src/core/reviewRound.test.ts::releaseModeFor (per-agent attach/release pairing)::*
50: Worker wiring compiles against the pure plan (attachThread(threadKey, refHint, readonly), ensureThreadWorktree(binding, sha, originUrl, modeSwitch), scrubThreadCredentials)[unit] resident npm run typecheck
50: live — agent:review on the resident: /debug threads shows the binding with readonly: true; in the worktree test -f .git/github-credentials → absent, git config credential.helper → unset, git remote get-url origin → /workspace/mirror, git fetch origin → permission denied / not a repository, git diff origin/main...HEAD --stat → works; then agent:coding in the same thread → recreated: true, credentials: "ok", origin = the Git door, .git/github-credentials absent and credential.helper unset[agent] Run agent:review then agent:coding on an onboarded repo in one thread; inspect via /debug threads and /exec in that thread's worktree.
51: sha body field parses absent→null, full 40-hex lowercase→itself, anything else (7-char abbreviation, uppercase, HEAD, main, ../evil, 39/41 chars, non-strings)→named 400[unit] src/execution/residentHead.test.ts::parseWantSha…
51: the sha applies to the ref it was resolved for — refHint = bound ref or no refHint → kept; refHint ≠ sticky bound ref → dropped; no sha → null[unit] src/execution/residentHead.test.ts::wantShaForBinding
51: fetch decision, naming why — missing ref → missing-ref (with or without a wanted sha, returnable or not); no wanted sha + ref present, on a ref the binding could not return from → no fetch; ref present at an OLDER tip than the wanted sha → stale-tip; tip already at the wanted sha → no fetch; unreadable tip + wanted sha → stale-tip; a returnable ref the mirror still holds → returnable-ref, even at the wanted tip; a returnable ref at a stale tip is stale-tip, not the verification[unit] src/execution/residentHead.test.ts::mirrorFetchReason
51: the client sends sha in the attach body when an expected head is set, omits the field otherwise[unit] src/execution/resident.test.ts::sends sha in the attach body when an expected head is known…
51: attach target after the fetch — the ref when it exists; the expected commit, detached, when the ref is gone but the mirror holds the commit; unknown-ref when no commit was named or the mirror lacks it[unit] src/execution/residentHead.test.ts::attachTarget::*
51: executor selection forwards a resolved headSha as the attach body's sha (alongside refHint + readonly), no field when unresolved[unit] src/execution/factory.test.ts::a resolved headSha is sent as the attach body's sha…
51: a ref whose tip is not the commit the caller named is stale-tip, in both directions and for an unreadable tip; a ref at that commit, or a caller naming none, clones the tip[unit] src/execution/residentHead.test.ts::attachTarget::the ref exists but its tip is not the commit the caller named… + ::…::the ref exists and the caller named no commit, or its tip IS the commit…
51: Worker wiring — the tip is read once and handed to attachTarget, the worktree is provisioned at that same tip (a tip the read could not answer is a named failure, never a null sha), stale-tip answers 409 with the state[unit] deploy/cloudflare-resident/staleTip.test.ts::the attach refuses a stale tip instead of running it
51: the bot sends the sha on the attach that binds the run and on a move, and on no recovery re-attach — the run re-attaches the branch as it left it[unit] src/execution/resident.test.ts::…::a recovery re-attach after an eviction omits the sha the run started with… + ::ResidentExecutor.moveTo::re-attaches with the new sha and answers the sha the worktree is now at
51 live (stale-tip): on a throwaway resident, POST /attach {resource, threadKey, refHint:"main", sha:"<40 hex of a commit that is not main's tip>"} (operator scope) → 409 {error:"stale-tip: ref \"main\" is at <tip7> in the mirror even after the fetch; the caller expects <want7> — …", state:"warm", reason:"stale-tip"}; the same attach with sha = the tip → 200[agent] after the deploy
51: the dispatcher passes RepoContext.headSha into executor selection[unit] src/core/dispatcher.test.ts::passes the resolved repo, ref and PR head to executor selection
51: Worker wiring compiles against the pure decision (attachThread(threadKey, refHint, readonly, wantSha), wantShaForBinding, mirrorFetchReasonFor)[unit] resident npm run typecheck
51: live — push a commit to an open PR's branch, then within the same refresh cycle (< 10 min) agent:review <PR url> in a thread already bound to that branch → the status card's resident · <branch>@<sha7> names the NEW head, the review posts to the PR (no ℹ️ Review not posted … reviewed head X is not the PR head Y), and /debug threads shows the binding's sha = the new head[agent] (post-deploy of resident + bot)
16b: clean-check decisions are unchanged in one spawn — missing tree → clean ("worktree missing"); a failed git probe → NOT clean with the first error line named; changes/unpushed counted with the exact pre-existing wording; PAM banners cannot shift a field; the git probes run inside su, never as root[unit] src/execution/residentCleanliness.test.ts
16b: an expired registration does not make a known owner safe for idle sleep; the resident keeps its runtime until item 71 authorizes loss[unit] deploy/cloudflare-resident/pausedWorkspace.test.ts::one preservation guard covers automatic loss paths::*
53: describeStepFailure — the pnpm capture (real bytes: ERR_PNPM_LOCKFILE_CONFIG_MISMATCH on stdout, the [WARN] on stderr) keeps the error AND the warning; both streams labelled; an empty stream drops its label; no output at all says no output; a timeout says (timed out); a long stream keeps its END and marks the cut; the whole reason stays bounded; exit 0 throws[unit] src/execution/residentStepReport.test.ts
53: stepFailureLog — names the step, carries both streams at the log budget, stays under 20 kB for a runaway step[unit] src/execution/residentStepReport.test.ts
53: imagePins — exact global install → pinned; @latest, a range and a bare name → floating; a FROM with no tag or :latest → floating while node:22-slim is not; reads through a line continuation past glued shell separators; a comment naming @latest is not an instruction. Fence: all three Dockerfiles (resident, sandbox, bot) carry zero floating pins[unit] src/deploy/imagePins.test.ts
53: bun — one exact global install per execution image, the same version in both, the base's binary removed in the same layer before the install, and the layer's own bun --version | grep -qx proving the bun on PATH is the pin; the resident proves it once more as worker1 after the pool exists[unit] src/deploy/imagePins.test.ts::the execution images' bun::*
53: the reported reason was never a cause — pnpm install --frozen-lockfile on a pnpm workspace's default branch inside cloudflare/sandbox:0.13.0-next.751.1 with an unpinned pnpm@latest (11.x), as worker1 via su -s /bin/bash: exit 0, stderr byte-identical to the stored down reason[agent] container repro: run the install inside the base image as worker1 and compare its stderr with the recorded reason
53: the pin removes the noise at its source — the same repro on the image built from the pinned Dockerfile: pnpm 10.34.5 / yarn 1.22.22 asserted at build, install exit 0 with stderr empty, still delegating to the repo's own packageManager pin[agent] container repro on the pinned image: assert the tool versions at build, run the install as worker1, expect exit 0 and an empty stderr
53 live: after the resident deploy, repo rebuild <owner>/<name> either reaches warm or goes down with a reason that names the failing command's own error (both streams present)[agent] post-deploy: rebuild a resident whose install had failed and read the reason on /status
54: isDiskFullMessage — the SDK's ENOSPC writeFile error (the refresh reason item 54 describes), git/cp No space left on device, npm ENOSPC → true; git config's errno-less exit-4 message, the stale-lock File exists message, near-miss words → false[unit] src/execution/residentDisk.test.ts::isDiskFullMessage …::*
54: parseDfFreeKiB — the Available column of df -Pk; header-only, non-numeric, or an error line → null, never 0; DF_FREE_ARGV is df -Pk /workspace[unit] src/execution/residentDisk.test.ts::parseDfFreeKiB …::*
54: diskFullReason names the step, keeps the message verbatim, appends the probe only when it answered; isDiskFullReason matches the exact disk-full: prefix; the floor is 128 MiB[unit] src/execution/residentDisk.test.ts::diskFullReason / isDiskFullReason …::*
54: planDiskFullRecovery — clear → recycle; inside the 1 h cooldown → wait naming the minutes and "does not fit"; ops in flight → wait naming the count; a live binding used within the floor → wait naming the floor and a run mid-flight, even with every tree clean, never dirt (the plan has no input about the trees); the order is cooldown, in flight, recent use[unit] src/execution/residentDisk.test.ts::planDiskFullRecovery …::*
54: disk-full recovery checks every live binding through item 71 before stopping or destroying the VM; an uncertain owner blocks the recycle with a visible reason[unit] deploy/cloudflare-resident/pausedWorkspace.test.ts::one preservation guard covers automatic loss paths::*
54: a recycle is refused while a binding was used within the floor even when every tree is clean — the why names the floor and a run mid-flight, never dirt[unit] src/execution/residentDisk.test.ts::planDiskFullRecovery …::a live binding used within the floor keeps the container even with every tree clean…
54: classifyRefreshFailure — ENOSPC wording → disk-full: outright (over a kill signature too); the errno-less git-config exit 4 → disk-full only with a probe below the floor, else git-setup-failed; a kill signature with a low probe stays refresh-interrupted; no probe → the step's own failure[unit] src/execution/residentRefresh.test.ts::classifyRefreshFailure …::* (red-verified: fails on a classifier without the disk-full class)
54: a recycle decided at the cycle's entry gate throws CycleRestartError, so the engine's retry restores onto the empty disk within the minute; a recycle decided by a failed fetch or step ends that cycle and the next bucket's instance restores[agent] the live row below: wrangler tail switchboard-resident shows disk-full: recycling the container … then refresh instance <id>: fetch disk-full-restart: the container is restarting — the step is retried onto it, and restoring · rehydrating → warm inside the same instance
54: disk-full: … is never a serviceable degraded reason (the bot goes cold without attaching)[unit] src/execution/residentState.test.ts::isServiceable::degraded stays cold … (fence: the allow-list is exact — the test names the case so a widened list fails loudly)
54: the two git messages are two different failures — a full disk with a creatable inode is exit 4 failed to write new configuration file <cfg>.lock; a pre-existing .lock is exit 255 could not lock config file <cfg>: File exists[agent] Linux tmpfs, no resident needed: docker run --rm --tmpfs /t:size=64k alpine:3.20 sh -c 'apk add -q git; printf "[a]\n\tb = c\n" > /t/gitconfig; dd if=/dev/zero of=/t/fill bs=4k 2>/dev/null; git config --file /t/gitconfig safe.directory /x; echo exit=$?; rm -f /t/fill; : > /t/gitconfig.lock; git config --file /t/gitconfig safe.directory /x; echo exit=$?' → error: failed to write new configuration file /t/gitconfig.lock / exit=4, then error: could not lock config file /t/gitconfig: File exists / exit=255.
54 live: a full disk is named, the card goes cold without an attach, and the resident recycles itself[agent] On a warm resident with ONE attached thread and nothing else live: from that thread /exec fallocate -l <free minus 64M> /tmp/fill (as the thread user; df -Pk /workspace afterwards shows Available under 131072), then leave the thread attached with an uncommitted edit (/exec echo x >> README.md) and backdate it past the idle floor (POST /debug {"op":"backdate-thread","threadKey":"<key>","days":1} — the fill's own exec bumped lastAttachAt, and a binding used within the hour keeps the container), then POST /debug {"op":"refresh-now"}. Expect: /status → {"state":"degraded","reason":"disk-full: fetch … ENOSPC … (/workspace: <n> KiB free)"}; a Slack request naming the repo shows resident degraded (disk-full: …) — using fresh sandbox on its card and wrangler tail switchboard-resident shows NO attachThread for it; the tail shows disk-full: <threadKey> tree discarded with the disk — left behind 1 uncommitted change(s) and 0 unpushed commit(s), discarded with the tree (or … the tree could not be measured before its removal (…) when the full disk refused the probe's mktemp) then disk-full: recycling the container …; /debug threads shows the binding still live (evicted:false, its user kept) with recycledLeftBehind: {uncommittedChanges: 1, unpushedCommits: 0} (or recycledUnmeasured); then within ~2 min /status restoring · rehydrating → warm, /debug info → lastRestore fresh and lastRefreshError cleared; the fill file and the tree are gone with the disk — the thread's next /exec answers worktree-missing and its attach recreates the tree (recreated:true). Before the backdate, refresh-now instead leaves lastRefreshError ending — container kept: a live worktree was attached to or used within the last 60 min …. Repeat the fill within the hour → degraded(disk-full: …) stays, /debug info lastRefreshError ends — container kept: recycled <n> min ago and the disk filled again …. Then free the space (rm the fill from a re-attached thread, detach) → the next cycle goes warm without a recycle.
18: pnpm's .pnpm store stays hardlinked — never in the mutable-cache swap — while a .cache nested inside it, .modules.yaml and .bin still are (a pnpm-workspace attach ~190 s → tens of seconds, thread tree 2.6 GB → ~0.45 GB)[unit] src/execution/residentDepCache.test.ts::mutableCachePaths…::pnpm's \.pnpm` store is package content, never a cache…; [agent]fresh-thread attach on a pnpm-workspace resident: card duration anddu -xshof the thread'snode_modules/.pnpm` (hardlinked: link count > 1 on its files)
18: the one-fork materialization script gates each dir on src/dst exactly like the old spawns, combines the chown + harden walks into one traversal with the identical -perm -g+w -o -perm -o+w test, falls back from cp -al to plain copy, tags mechanisms/mutable listing/failed steps; the real-shell proof uses the host primary group while resident calls retain the same-named user/group default; the swap script replays rm -rf/cp -R/chown -Rh per path[unit] src/execution/residentDepCache.test.ts::depCacheScript… + ::parseDepCacheScriptOutput + ::mutableCacheSwapScript…
18: the swap script gates each path on its store counterpart — a tree-private dot entry with no counterpart is left in place (no rm/cp/chmod/chown) and named on a skipped=<relative path> line, every path with one keeps the exact sequence inside its guard; run for real over such a tree the script exits 0 with the entries still there; the parser collects the skipped= lines[unit] src/execution/residentDepCache.test.ts::mutableCacheSwapScript…::a path whose store counterpart does not exist is tree-private… + ::mutableCacheSwapScript…::run for real over a tree whose dot entries have no store counterpart… + ::parseDepCacheScriptOutput::collects the skipped= lines…; [agent] on a resident whose checkout carries a top-level node_modules dot entry the store entry lacks, the next refresh reads warm and the Worker log names the entry on a deps: deps-mutable-swap left … in place: line
5/7: an R2 SNAPSHOT upload that outlives its 5-min budget rejects with the transfer NAMED (<what> timed out after <ms>ms), a late loser never surfaces as an unhandled rejection; restores are judged by bytes instead (item 61)[unit] src/execution/residentRefresh.test.ts::withTimeout…
45: the for-each-ref listing parses into the mirror's branch set (trimmed, blanks dropped, empty listing = empty set)[unit] deploy/cloudflare-resident/gc.test.ts::parseRefListing…
Perf live: cold-wake restore and refresh-cycle timings improve (concurrent restore/snapshot pairs; reclamation fate lookups no longer serial) — compare lastRestore.ms and the tail's refresh: … done in <ms> before/after[agent] post-deploy of the resident: compare both timings across the deploy
55: measurement — parseDfKiB reads total/used/free (header-only or non-numeric → null); duArgv orders mirror, checkout/node_modules, checkout, threads, homes; parseDu keeps <KiB>\t<path> lines only; assembleDiskSample itemizes by key, unread parts are null (never 0), other = used − Σ and never negative[unit] src/execution/residentDiskBudget.test.ts::measurement …::*
55: reserve — staging is 0.6 × (mirror + checkout), never the deps; a deps-store entry backup in flight adds 0.6 × deps and only then; the floor the larger of 1 GiB and 5 % of the disk; unmeasured parts count 0 in staging while the floor stands; a resident with 6.9 GiB of deps admits a hardlinked tree with 6.7 GiB free[unit] src/execution/residentDiskBudget.test.ts::the reserve …::*
55: under pressure the deps store keeps no spare — the plan with the under-pressure limit removes every unreferenced complete entry coldest first, names each with its size, and never a protected one; the refusal names the spares evicted and their bytes, and evicted no idle tree when none followed[unit] src/execution/residentDepsStore.test.ts::store listing and eviction (LRU among unreferenced entries, never a protected key)::under disk pressure the plan keeps no spare (DEPS_STORE_MAX_UNREFERENCED_UNDER_PRESSURE = 0), coldest first, protected entries untouched, src/execution/residentDiskBudget.test.ts::diskPressureReason — the refusal names free, reserve, projected, what was evicted and what was kept::names the deps-store spares evicted first and their bytes, and says no tree followed when none did
55: the resident wires spares before trees — the admission evicts spares with the under-pressure limit, re-probes df, then orders the trees; the pressure path and the sweep share one protected-key rule and one removal[unit] src/execution/residentDiskBudget.test.ts::resident Worker wiring (static) — spares before trees under disk pressure::the admission evicts deps-store spares first, with DEPS_STORE_MAX_UNREFERENCED_UNDER_PRESSURE, re-probes df, and only then orders the trees, ::the pressure path and the sweep share one protected-key rule and one removal (entries and their backups)
55: projection — hardlink → parts.checkout; reconcile (lockfile diverged, install command present) → + RECONCILE_DEPS_RATIO × parts.deps (strictly between a hardlink and a full copy); reuse → 0; a missing part → null, never a guess[unit] src/execution/residentDiskBudget.test.ts::projecting …::*
55: diskBudgetMb caps free at budget − used (never negative; a budget above the disk is the disk; capped says whether it decided)[unit] src/execution/residentDiskBudget.test.ts::diskBudgetMb …::*
55: admission — fits when free − reserve ≥ projected with the exact shortfall otherwise; a fresher df overrides the sample's free; in-flight commitments are deducted ONCE from the raw reading (a re-check after an eviction feeds rawFreeAfterEviction's raw number, never a net one); the cap refuses; an unmeasured projection admits only while free clears the reserve; reuse is never refused[unit] src/execution/residentDiskBudget.test.ts::checkDiskAdmission …::* (red-verified: with the reserve zeroed the install-thread case admits)
55: eviction order — coldest attach first, ties on key; the requesting thread, a busy tree, the default branch and a tree attached under 10 min are kept by name; an unparsable lastAttachAt is kept as recent[unit] src/execution/residentDiskBudget.test.ts::orderEvictionCandidates …::*
55: disk pressure considers cold trees in normal order and removes only those item 71 authorizes; a protected tree is reported as kept[unit] deploy/cloudflare-resident/pausedWorkspace.test.ts::one preservation guard covers automatic loss paths::*
55: the refusal text names projected (or UNMEASURED), free, the cap when one decided, the reserve and both terms, the shortfall, what was evicted with the bytes back, and every keep with its why; isDiskPressureReason matches the exact prefix[unit] src/execution/residentDiskBudget.test.ts::diskPressureReason …::*
55: one unit everywhere — formatGiB (two decimals under 10 GiB, one above, ? unknown) and formatDiskGauge (used/total (pct))[unit] src/execution/residentDiskBudget.test.ts::formatting …
55: eviction removes the pool user's pnpm store, .cache, .npm, .yarn, .bun in one rm[unit] src/execution/residentDiskBudget.test.ts::threadUserCacheCleanArgv …
55: a disk-pressure 503 from /attach falls back to the cold sandbox with the whole refusal in the card note[unit] src/execution/factory.test.ts::makeExecutor resident selection::item 55: a warm probe then a disk-pressure attach (503…
55: repo list appends · disk <used>/<total> (<pct>%) from live.disk; a malformed sample adds nothing[unit] src/core/commands/repo.test.ts::repo.list::item 55 …
55: the watchdog line carries · disk max <pct>% (<owner/name>) for the fullest measured resident; absent/malformed gauges are skipped[unit] src/core/schedules.test.ts::watchdogFiring …::item 55 …
55: /residents index row shows the gauge with the sample time on hover (none for an unmeasured resident); the detail page's Disk section shows gauge, free (under the cap when set), reserve + terms, headroom in trees, sample time, every component (thread trees at unique bytes, homes above 1 MiB), "not measured yet" before the first sample, and a malformed sample never renders NaN or markup[unit] web/src/pages/residents.test.ts::ResidentsIndexPage::item 55 …, ::ResidentDetailPage::item 55 … (×2)
55: the instance type is 4 vCPU / 12 GiB / 20 GB — the CPU is the decision, memory is what the vCPUs require, the disk is what the platform allows for that memory; it fits 16 hardlinked + 1 installing (and 12 + 2) trees with the reserve, is not over-provisioned, neither 32 hardlinked trees nor the mixed workload of 16 hardlinked + 2 installing trees fits this ceiling, the previous 16 GB fit 10 + 1 — over the SAME reserve constants the admission uses[unit] deploy/cloudflare-resident/instanceSizing.test.ts::*
60: purge-bindings selects only evicted bindings under a whole non-production namespace (or longer); a bare/partial/production prefix is refused by name; live bindings are listed as kept, never deleted[unit] src/execution/bindingPurge.test.ts::selectBindingsToPurge::*
60 live: a load run leaves no rows behind[agent] After npm run load -- resident --resource repo:<slug> --threads 4 --hold 30 --cpu-seconds 5, POST /debug {"op":"threads","resource":"repo:<slug>"} lists no load: thread keys; POST /debug {"op":"purge-bindings","resource":"repo:<slug>","prefix":"slack:"} answers 400 naming the production namespace.
55 live: the gauge appears and moves — after the resident deploy, POST /debug {"op":"measure-disk","resource":"repo:<owner>/<name>"} ($ADMIN) answers a DiskSample whose parts.deps is the deps store's size, parts.checkout the tree without it, parts.homes.worker1 < 1 MiB; GET /residents shows the same under live.disk; /residents/<owner>/<name> shows the Disk section with "room for N more"; repo list in Slack shows · disk x/y (pct%); attach a thread (any coding request naming the repo), wait ~5 s, re-read /residents: parts.threads gains the thread at its unique bytes and used grew by about that; after the run's /detach the entry disappears and used falls back[agent] post-deploy: the probes in the criterion, in order
55 live: admission refuses legibly — on a resident with ONE attached thread, from that thread /exec fallocate -l <free − reserve − 0.3G> /tmp/fill (leave room for less than one hardlinked tree above the reserve; df -Pk /workspace confirms), then a coding request on a NEW thread naming the repo: the card reads resident attach failed (… disk-pressure: need 0.5x GiB for a new tree (hardlink), but … free minus the … reserve (snapshot staging … + floor 1.00 GiB) leaves … — short by …; evicted nothing; kept 1: <the filled thread> (attached <n>m ago (floor 10m))) — using fresh sandbox, wrangler tail switchboard-resident shows the same line, /status stays warm (no lifecycle flip); backdate the filled thread (POST /debug {"op":"backdate-thread", threadKey, days:1}) and repeat: the refusal now reads evicted 1 idle tree(s) … and the attach SUCCEEDS on the space it freed (the fill file went with the tree); rm nothing by hand — the eviction did it[agent] post-deploy: the recipe in the criterion
56: the pre-step sweep is a root sh -c (never su), scoped by cwd to the tree (readlink /proc/$p/cwd under /workspace/checkout — a parallel store install elsewhere is never matched), names the survivors (ps -o pid=,args=) BEFORE kill -KILL (SIGKILL, never SIGTERM), waits bounded until EVERY matched pid is gone (exit 1 after 5 s), and exits 0 silently when nothing matches[unit] src/execution/residentRefresh.test.ts::killStaleBuildProcessesCommand …::* (red-verified: export absent)
56: an abandoned wait becomes the step's OWN timeout — timedOut: true, the observed exit (or -1 with "no exit status observed"), "killed" in the report — and classifyRefreshFailure files it <step>-failed: exit <n> (timed out) …, never an interruption[unit] src/execution/residentStepReport.test.ts::abandonedWaitStepResult …::* (red-verified: export absent)
56 live: on a busy resident, wrangler tail switchboard-resident (or the observability log) shows install: killing stale worker1 processes: + the npm command line at most once after the deploy (the orphans the incident left), then a cycle that ends refresh: … rebuild done in <ms> with /residents state: warm, lastRefreshError: null; no later cycle logs checkout-update failed … Directory not empty[agent] post-deploy: watch the tail across the first cycles
57: an installing marker for the target lockfile with no deps key → rebuild with install on a keep-deps clean, why names the resume; an installing marker for another key → the conservative full clean; a landed deps key wins over a leftover installing marker[unit] src/execution/residentRefresh.test.ts::planRefresh: a timed-out install resumes instead of starting over (3 tests; the resume case is red without the planner branch)
57 live: after a cycle whose install timed out, the next cycle logs refresh: … rebuild (install resumes — …) and ends rebuild done in <ms> with /residents state: warm; every failed cycle logs refresh: cycle failed — <reason> in wrangler tail switchboard-resident[agent] watch the tail across the next lockfile-moving merge on a repo whose cold install nears the budget; the Worker wiring (INSTALLING_MARKER, readRefreshDisk, the log line) is deploy/* code covered by npm run typecheck in deploy/cloudflare-resident
58: planThreadDeps — reused tree → nothing; same key → seed only; different key + install command → seed AND install (why names the reconcile); different key, no install command → nothing; threadDepsMechanism → reconcile whenever the install ran, else the seed's mechanism or none[unit] src/execution/residentDepCache.test.ts::planThreadDeps …::* (5 tests; red-verified: exports absent)
58: the residents page counts lockfile-diverged trees at the reconcile cost (N more (0.98 GiB each) lockfile-diverged (reconciling) on the fixture), never at the full deps[unit] web/src/pages/residents.test.ts::ResidentDetailPage::item 55: the Disk section …
58 live: an attach on a branch whose lockfile differs from main answers deps: "reconcile", reconciled: true; the resident log shows deps /workspace/threads/…: lockfile differs from the warm checkout — seeded from the shared cache, install reconciles the delta and attach …: disk admitted — reconcile <GiB> projected; du -xsk of the tree is a fraction of the checkout's deps and the attach completes well inside the bot's wait (the baseline it replaces: install 1.94 GiB projected, 5+ min, /attach fetch failed at ~272 s)[agent] after deploy, attach a read-only probe thread (operator token) with refHint set to a lockfile-diverged branch; record deps, attachMs, the log lines and du in the receipt
61: judgeRestoreProgress — bytes still growing → wait however long (past the old 300 s); no growth for RESTORE_STALL_MS → stalled naming the idle span and the bytes so far; no first byte within the window → stalled (the clock runs from the start); a null sample neither counts as growth nor resets the clock; past the hydrate-wide deadline (RESTORE_MAX_MS, shared by the pending wait and both restores; an explicit deadlineMs caps a late-starting restore by the hydrate's clock, not its own) → capped even while bytes arrive; the cap sits under the 30-min stale-mid-flight window; no samples yet → wait[unit] src/execution/residentRefresh.test.ts::judgeRestoreProgress …::* (6 tests; red-verified: export absent)
61 live: a container swap (an image-changing deploy) is followed by mirror restore: <GiB> in <s> (<MiB/s>) and checkout restore: <GiB> in <s> (<MiB/s>) in the tail and /residents warm with lastRestore {at, ms} — including when the checkout restore runs past 300 s (the shape that forced it: 481 s); a stall reads down(r2-restore-failed: checkout restore stalled: no bytes written for <s> (<GiB> after <s>))[agent] after the next image-changing deploy, read the tail around the first attach; the Worker wiring (restoreWithProgress, pendingRestores, dirKiB) is deploy/* code covered by npm run typecheck in deploy/cloudflare-resident
61 live (rebuild after a failed restore): a down(r2-restore-failed: …) reason ends — container stopped so the transfer cannot land on a rebuild; the rebuild that follows provisions on a fresh disk (uptime inside the container restarts) and the deps store's entry files read 444 after it (find /workspace/deps/*/node_modules -type f -perm -u+w | wc -l → 0 from a read-only attach) — the shape this rules out read 644 on 45 290 of 45 291 files[agent] post-deploy, on the next restore that stalls (or POST /debug {"op":"force-down"} + /rebuild on a resident whose restore is mid-flight); the Worker wiring (the two stop() exits, provisioning's await-restores) is deploy/* code covered by npm run typecheck
61: backupTransferMode — the four presigned inputs, exactly the names the SDK reads; all non-blank → presigned / localBucket: false; any absent, empty or non-string → local / localBucket: true with the missing names in env order; unrelated env ignored[unit] src/execution/residentBackupTransfer.test.ts::backupTransferMode …::* (red-verified: module absent)
61 live (presigned transfers): after the R2 token is put and the resident deployed, GET /healthz answers backupTransfer: "presigned" (before: "local" with backupTransferMissing); the next refresh cycle's backup.create events no longer carry provider: "local-binding"; a container swap then restores through the container (mirror restore: …, checkout restore: … logged) — including a checkout over 1 GB with no isolate exceeded its memory limit and no manual rebuild[agent] post-deploy; first a forced stop-container on a small resident, then the largest resident's next swap
61: content-addressed snapshots — the checkout archive excludes exactly the top-level node_modules; an entry backup record is keyed resident:depsBackup:<key> and a non-key is refused; the archive TTL is at least 180 days; planDepsMaterialization restores a recorded backup before installing while a complete entry or a running install still wins; after a sweep the archives to drop are exactly the evicted keys that have one[unit] src/execution/residentDepsStore.test.ts::entry backups: content-addressed snapshots of the store …::* + ::planDepsMaterialization (hit / join / install)::a recorded entry backup is restored before anything is installed; a hit or a running install still wins (red-verified: symbols absent)
61: planWakeDepsBudget — plenty left → the install keeps its budget and the restore is judged against the hydrate deadline; less left than the install budget → the install is bounded by the remainder; under WAKE_DEPS_MIN_MS (≥ 60 s) or past the deadline → skip[unit] src/execution/residentRefresh.test.ts::planWakeDepsBudget …::* (red-verified: function absent)
61: a presigned restore is extracted, never left mounted — the staging mount is a sibling of the target under /workspace; the extract script prefers unsquashfs from the downloaded .sqsh, falls back to cp -a out of the mount, unmounts (overlay, then the <backupId>_* lowers under the SDK's mount root; a non-id is refused as a glob), removes the archive, and renames the tree in LAST; the unmount-all script takes every fuse mount under /workspace and the SDK's mount root deepest first, then the leftovers; on a real filesystem the cp branch reproduces the tree with its modes and leaves no staging dir; the resident image's build asserts unsquashfs with the same command -v probe the script uses (unsquashfs -version exits 1 on squashfs-tools 4.5 with no filesystem named, and failed the 1.2.0 image build)[unit] src/execution/residentRestoreExtract.test.ts::residentRestoreExtract (item 61: the SDK's presigned restore MOUNTS the archive; the resident extracts it onto ext4)::* (red-verified: module absent)
61 live (extraction): after the release, a wake logs checkout restore: … GiB in … s followed by checkout restore: extract: unsquashfs in … s (or extract: cp on the previous image), mount inside the resident shows no fuse mount under /workspace, /var/backups holds no .sqsh, a second hydrate on the same disk passes clean-before-restore, and the disk sample's mirror/checkout parts are the tree sizes again[agent] after the release: read the wake's log lines, then mount, ls /var/backups and the disk sample from inside the resident
61 live (content-addressed snapshots): after the release, a refresh cycle's checkout backup.create is a fraction of the previous size (the tree without its node_modules); POST /debug {"op":"deps-backups"} lists one record per warm key with deps: backed up <key8> logged once and never again for that key; a forced stop-container restores mirror + tree and then logs deps: restored <key8> from backup in <ms> with no install; a rebuild's provisioning restores the warm key's entry instead of installing[agent] after the release: read the cycle's backup.create size, /debug deps-backups, and the log lines across a forced stop-container and a rebuild
59: store layout — entries under /workspace/deps/<key>, .complete inside the entry; a non-hex or wrong-length key is refused as a path segment[unit] src/execution/residentDepsStore.test.ts::deps store layout …::* (red-verified: module absent)
59: planDepsMaterialization — complete → hit; in flight → join; neither → install; complete beats a stale in-flight memo[unit] src/execution/residentDepsStore.test.ts::planDepsMaterialization …::*
59: depsInstallSemaphoreSize is nproc, and 1 for anything unreadable ("", an error line, 0, null)[unit] src/execution/residentDepsStore.test.ts::depsInstallSemaphoreSize …::*
59: the scratch clone is --shared --no-checkout of the mirror + checkout --detach <sha>; the commit script moves node_modules, renames staging → entry, writes .complete LAST, refuses an install that produced no node_modules, keeps the scratch on adoption, and REPLACES a marker-less entry (crash debris between rename and touch) instead of nesting inside it; run on a real filesystem the entry is complete, the scratch is gone, a second racer for the same key leaves the winner intact, and a re-install over debris ends complete with only the fresh tree[unit] src/execution/residentDepsStore.test.ts::the install scratch tree and the store commit::*
59: deps-harden strips owner write from every installed file; a scratch with no node_modules is refused (exit 1, names the scratch) when the commit has a lockfile and gets an empty node_modules owned by the build user when it has none (NO_LOCKFILE_KEY = sha256 of no input), after which the commit script completes the entry; the real-filesystem case skips, saying why, when the test process is root — the mode bits withhold nothing from uid 0, so test -w cannot observe the hardening[unit] src/execution/residentDepsStore.test.ts::deps-harden …::* (red-verified: symbols absent, then chown group on macOS)
59: listing parse (entry/complete/kib/used + leftovers, garbage ignored); eviction order never names a protected key, debris first, then coldest; planDepsEviction keeps DEPS_STORE_MAX_UNREFERENCED (1) warmest spares and removes the rest plus leftovers; nothing to do → empty plan[unit] src/execution/residentDepsStore.test.ts::store listing and eviction …::*
59: depCacheScript takes the store entry as the node_modules source while build dirs still come from the checkout; without the option it is byte-identical to before[unit] src/execution/residentDepCache.test.ts::depCacheScript with a store-backed node_modules source …::*
59: the tool-cache swap makes its copies writable again (chmod -R u+w between cp -R and chown) — store entries are owner-read-only[unit] src/execution/residentDepCache.test.ts::mutableCacheSwapScript …::the copy is made writable again …
59: duArgv measures the deps store BEFORE the checkout; assembleDiskSample reads parts.deps from it[unit] src/execution/residentDiskBudget.test.ts (the layout fixture carries depsStoreDir)
59: ResidentExecutor.attach parses a heartbeat-streamed binding; a streamed refusal's in-body status (503 mirror-busy, 409 needs ref) is handled exactly like the real status; a pre-validation 404 without status is still not-onboarded[unit] src/execution/resident.test.ts::ResidentExecutor.attach over a heartbeat stream …::* (red-verified: status read from HTTP only)
59 live: a lockfile-changing merge on main — the resident log shows deps: installing <key8> at <sha8> in /workspace/deps/.scratch-…, then deps: installed <key8> in <ms>, the cycle's checkout-update + deps-materialize under the lock and refresh: … rebuild done; ls /workspace/deps (via a read-only attach) shows the entry with .complete; the checkout's node_modules/<pkg>/package.json has link count ≥ 2[agent] post-deploy: watch the tail across the next lockfile-changing merge
59 live: two threads attach on the SAME new key while it installs — one deps: installing, both attaches answer deps: hardlink after it, neither logs a second install; a thread on an OLDER branch attaches in seconds via the entry main used to have[agent] (post-deploy)
59 live: item 58's shape (ship on a lockfile-diverged branch while the refresh installs) binds — the card shows the wait, never fetch failed; the /attach response is heartbeat whitespace then JSON[agent] (post-deploy)
59 live: after the deploy onto a pre-store disk, the first attach on main logs deps: adopting the checkout's node_modules as <key8> once, and /residents disk.parts.deps equals du -sk /workspace/deps; a pnpm resident's .pnpm survives adoption hardlinked (link count on a .pnpm file ≥ 2)[agent] (post-deploy)
63: the collector's offsets, bounds, sanitization and skew rule[unit] src/execution/residentStepTrace.test.ts::createStepTrace::*
63: the bot re-validates the trace at the parse and grafts it under its attach / command span[unit] src/execution/residentTrace.test.ts::*, src/execution/resident.test.ts::ResidentExecutor.attach over a heartbeat stream …::carries the resident's step trace on the binding…
The watchdog firing and the refresh cycle are roots of the resident's own log — resident.watchdog with resident.check children, resident.refresh with its resident.<step> children (tracing.md item 25)[agent] live: wrangler tail switchboard-resident across one cron minute
63 live: an attach answer's trace names mutex_wait, clone/worktree-clone, install with plausible durations that sum to about attachMs[agent] after deploy: POST /attach for an onboarded repo with the operator bearer and read trace in the answer
22: the incarnation id comes from the random source alone, and one predicate judges every lease — alive for the current incarnation inside its budget (the last instant included), dead for another incarnation however fresh, dead past its expiry for the current one[unit] src/execution/residentIncarnation.test.ts::mintIncarnationId …::*, src/execution/residentIncarnation.test.ts::leaseIsDead …::* (red-verified: module absent)
22: takeMutex — an empty row is taken and records the current incarnation, the step and the budget as expiry; a live holder of the current incarnation makes the caller wait for what is left of its budget; a holder of another incarnation or an expired one is taken over at once with the reason named and the dead lease handed back (its tree with it); the row carries a tree only when the taker names one[unit] src/execution/residentIncarnation.test.ts::takeMutex …::*
22: releaseMutex releases only the holder's own row; a row another holder took over, or no row, is left alone[unit] src/execution/residentIncarnation.test.ts::releaseMutex …::*
22: the Worker mints the incarnation id at the isolate start and in swapIncarnation alone; setResidentState clears the memos without swapping, so a lifecycle transition never kills the transitioning cycle's own lease[unit] src/execution/residentIncarnation.test.ts::the incarnation boundary in the resident Worker …::*
22: a dead deps holder's paths are its scratch tree and its staging dir, recovered from the scratch path its lease names[unit] src/execution/residentDepsStore.test.ts::the install scratch tree and the store commit::a dead attempt's paths are its scratch tree AND its staging dir, recovered from the scratch path its lease names; any other path names no attempt
22: the in-flight row — a refresh or hydration lease of the current incarnation inside its budget is in flight, a previous incarnation's is dead, a hydration past the stale bound is dead the way the memo was judged, clearing one kind leaves the other; the refresh lease outlasts the stale bound; the mutex, in-flight and deps-lease keys are distinct and a deps lease key is built only from a lockfile key[unit] src/execution/residentIncarnation.test.ts::the in-flight row …::*, src/execution/residentIncarnation.test.ts::storage keys …::*
22: every step plan called twice over the facts its first run leaves behind is done the second time with exactly one command issued — the fetch (per cycle and ref), the install (a complete entry), the build (the markers, through the refresh planner), the snapshot (the record at the stamp), the restore (the ready marker) and the wake's dependency view[unit] src/execution/residentStepPlan.test.ts::*::called twice… (red-verified: module absent)
22: the fetch, install, restore and deps-view plans each decide from their one fact — this cycle's fetch record for this ref (another cycle's or another ref's is not this step's work), the complete store entry, the ready marker naming the stamp's sha, the checkout holding its view (and whether the entry must be produced first)[unit] src/execution/residentStepPlan.test.ts::planFetchMirror …::*, src/execution/residentStepPlan.test.ts::planInstallDeps …::*, src/execution/residentStepPlan.test.ts::planRestore …::*, src/execution/residentStepPlan.test.ts::planMaterializeDeps …::*
22: the build plan over the refresh planner — facts already at the target → done; HEAD, deps and build markers all at the target → done; key unchanged → run without install on a keep-deps clean; key changed → run with install on a full clean, a resumed install keeping its partial tree[unit] src/execution/residentStepPlan.test.ts::planBuild …::*
22: the snapshot step — no record or a record at another stamp → run, the record at this stamp → done; the commit decision commits over an unchanged record (or none on both reads) and answers superseded when the record appeared, vanished or changed since the read, never a throw[unit] src/execution/residentStepPlan.test.ts::planSnapshot + snapshotCommitDecision …::*
22 live: /debug info shows the resident's incarnation, mirrorMutex null while nothing holds the mirror and leases.refresh naming that incarnation with step: "refresh" during a refresh-now, null again once the cycle ends; a Worker deploy mid-cycle leaves a leases.refresh whose incarnation differs from the new incarnation, and the next cycle runs without waiting[agent] POST /debug {"op":"info"} before, during and after POST /debug {"op":"refresh-now"}; then wrangler deploy during a cycle and read info again
7: the lifecycle row — a missing, unknown or alarm stored value reads workflow; workflow is the one word that parses[unit] src/execution/residentInstanceId.test.ts::lifecycleOf / parseLifecycle …::* (red-verified: the flagged default answered alarm)
7: the refresh instance id is the same anywhere inside one ten-minute bucket and new at the boundary; owner and name are lower-cased with every character outside [A-Za-z0-9_-] mapped to -; for the longest owner and repository name the code host allows it matches ^[a-zA-Z0-9_][a-zA-Z0-9-_]*$, stays within 100 characters and stays distinct per resident through the slug's hash[unit] src/execution/residentInstanceId.test.ts::refreshInstanceId …::*
9: the cron's decision — every serving row is eligible, no flag defers it; an onboarding or down row never; a refreshing row younger than STALE_MIDFLIGHT_MS is a live cycle and skipped, an older one (or one with no timestamp) is created for; an idle-marked warm row is created for only once the idle cadence has elapsed in whole buckets since the last instance, an idle-marked degraded or restoring row and an awake row every bucket and never twice in one, an idle-marked refreshing row younger than the stale bound still a live cycle, and a row with no instance yet at once; the same gate (refreshCycleBlocked) lets refresh-now start a cycle inside a bucket the cron already served but never beside a live one or on a resident that is not serving[unit] src/execution/residentInstanceId.test.ts::shouldCreateRefreshInstance …::*, src/execution/residentInstanceId.test.ts::refreshCycleBlocked …::*
7: the step policy is six attempts thirty seconds apart doubling, whose delays sum to 930 s — past ten minutes; a step's timeout is the method's budget capped at the engine's 30 minutes and never zero[unit] src/execution/residentInstanceId.test.ts::the step retry policy and budgets::*
7: the Workflow binding's class_name is declared in refresh.ts and re-exported by worker.ts under that name, and neither refresh.ts nor shared.ts imports worker.ts at runtime (every import of the entry is type-only) — the entry and the instance module form no cycle[unit] deploy/cloudflare-resident/refresh.test.ts::*
7 live (the gate this release is held for): after the deploy every production resident runs its refresh cycle as instances — /status shows refresh.instance.id advancing one bucket at a time with lastStep: "measure: done", /debug {"op":"schedules"} answers {"provisionRun":[],"provisionDeadline":[]} for every resident after one interval, the Workflows dashboard lists the instances by id, the Worker's log shows one resident.refresh root per instance carrying instanceId, alarm-missed is absent from every /status reason for a week, a resident Worker deploy landing mid-instance ends in a completed instance with one retried step and a row that was never degraded, and no resident is parked[gap] human-gated, the program's live receipt on the tracker
68: the Worker names the refused connect at the spawn alone — run()'s spawn catch throws the typed word after the reset and the replacement and before the unreachable count, its collect catch never does, the recognizer is the platform's wording over the cause chain, and the unreachable counter never counts it; every thread method that runs a command answers the typed error as runtimeBusyErr beside the reset and the replaced words, the body converts nothing, and the shared rejection mapper names the word after the stub boundary[unit] deploy/cloudflare-resident/runtimeBusy.test.ts::the refused connect is named at the spawn alone::*; deploy/cloudflare-resident/runtimeBusy.test.ts::the thread data plane answers the word as the 503 with the token::*
68: the refresh cycle yields to the busy container — the classifier reads the token anywhere in the text as busy with the refresh-yielded: reason, before the disk inference and never on the platform's bare words; the instance step answers stopped (runtime-busy) after the classifier and before any failure is recorded, the yield releases the lease and puts back the settled state the fetch step remembered under refreshing, the disk probe is skipped, and the Workflow skips the housekeeping steps[unit] src/execution/residentRefresh.test.ts::classifyRefreshFailure (a busy container is a yield, not evidence)::*; deploy/cloudflare-resident/runtimeBusy.test.ts::the refresh cycle yields to the busy container::*
68: runtimeBusyErr is the 503 with the token and the system cause keeping the DO's words; a rejected stub call carrying the word — typed, or name and message after the boundary — is answered the same on every route; the word is the platform's transient for the catch-all while the platform's bare words alone are not the token[unit] deploy/cloudflare-resident/threadErr.test.ts::a refused connect (runtime-busy)::*
68: the client re-sends the identical /exec at 3/5/10 s and returns the eventual result; the wait is bounded by the command's budget and by the 5-minute cap with ExecCapacityError, never infra and never a re-attach; a /read 503 with the token is the same wait; a hard stop ends the pause with the run's aborted error and nothing is re-sent; an older Worker's bare words are one send as answered[unit] src/execution/resident.test.ts::ResidentExecutor waits out a refused connect (item 68)::*
68 live: a resident-backed coding run whose gate saturates the container while the harness polls its transcript keeps the run — the resident Worker's logs for the thread carry runtime-busy answers followed by the poll's success, never a resident /exec: Container is taking too long to accept that ends the run[agent] a coding run on a warm resident whose gate is the repository's full verify; the run page's exec spans and wrangler tail switchboard-resident -c deploy/cloudflare-resident/wrangler.jsonc filtered on runtime-busy
72: an operator capture of a retained resident thread reads every directory, file and symlink including .git, ignored and untracked files, odd names and files over /read's 10 MiB cap through bounded /exec responses; exact ref/SHA/binding, boot ID, source manifest and every chunk must agree before a receipt is sealed[unit] src/execution/residentArchive.test.ts::resident archive harness::captures every private entry and independently restores bytes, modes, symlinks and empty directories; ::refuses a changed boot ID or binding before sealing
72: small and empty regular files transfer in bounded batches, including a tree with hundreds of entries; large files retain bounded chunks and directory/symlink metadata stays paged within the existing response cap rather than opening one remote execution per entry[unit] src/execution/residentArchive.test.ts::resident archive harness::batches many small and empty files while chunking a large file; ::pages long symlink metadata within the response cap
72: after begin returns a scratch manifest, paging, validation, chunk and source-verification failures attempt cleanup only while the original binding and boot still agree; the original refusal remains primary, no receipt is written, and a cleanup failure is itself a refusal[unit] src/execution/residentArchive.test.ts::resident archive harness::cleans scratch after a failed transfer without sealing; ::keeps the original refusal when cleanup fails; ::refuses a successful capture when scratch cleanup fails; ::skips scratch cleanup after a binding change
72: the offline verifier reconstructs bytes, modes, symlinks and empty directories from the sealed archive without following archive paths outside the restore root; its file rescan reads through an opened no-follow descriptor, not a path checked before opening; truncated, missing, duplicate or changed chunks and unsafe paths refuse without a success receipt[unit] src/execution/residentArchive.test.ts::resident archive harness::refuses a truncated or missing chunk without sealing the archive; ::rejects a changed archive byte during offline restore
72: the legacy offline reader validates the complete receipt identity and counters before creating a restore directory; missing identity, malformed paths, hashes, counters or an unsupported protocol cannot return a typed verified receipt. A valid legacy receipt remains readable and still proves no physical quiescence, complete runtime custody or retirement authority.[unit] src/execution/residentArchive.test.ts::resident archive harness::rejects an incomplete or malformed legacy receipt before writing a restore; ::captures every private entry and independently restores bytes, modes, symlinks and empty directories
72 live: the exact old Worker/image on an isolated disposable resident captures a mixed private tree, restores it independently, and rejects a truncated chunk or replaced VM; this does not prove the original retained trees or authorize a production upgrade[gap] requires a separately identified disposable deployment and reviewed operator gate
76: the attach exit fills code from the refusal's typed fields; the deploy fence, a pending settlement, an unverified workspace or pool-UID owner and a failed attach step set it at their sites, and a preserved workspace refused inside a worktree step keeps its code[unit] deploy/cloudflare-resident/notServiceable.test.ts::an attach refusal names its class as a setup word (record 0097)::the one attach exit fills \code` from the refusal's typed fields, so every refusal the attach answers is classified in one place; deploy/cloudflare-resident/notServiceable.test.ts::an attach refusal names its class as a setup word (record 0097)::the sites whose `reason` two classes share set the code themselves: the deploy fence, a pending settlement beside a preserved workspace, an unverified workspace owner, and a failed attach step; deploy/cloudflare-resident/notServiceable.test.ts::an attach refusal names its class as a setup word (record 0097)::a preserved workspace refused inside a worktree step keeps its code through the step error, so a pending settlement is never relabeled`
76: a refusal's code is read from its typed fields only: a reason that names one class, the drain's record, a site's own code; a shared reason without its code, an uncovered refusal or a prototype key names nothing; a pending settlement is told apart from every other preserved tree[unit] deploy/cloudflare-resident/threadErr.test.ts::attachRefusalCode::names each refusal class whose reason is its own; deploy/cloudflare-resident/threadErr.test.ts::attachRefusalCode::names the fleet drain by its record, not its words; deploy/cloudflare-resident/threadErr.test.ts::attachRefusalCode::keeps the code a site set where a reason is shared by two classes; deploy/cloudflare-resident/threadErr.test.ts::attachRefusalCode::names nothing for a refusal no word covers, a shared reason without its code, or a prototype key; deploy/cloudflare-resident/threadErr.test.ts::preservedWorkspaceCode::separates a pending settlement from every other preserved tree
76: the bot's attach error carries the resident's code for a deterministic refusal, a drain that never reopened and a transient refusal whose wait never cleared; an absent, unknown or unanswered code is fallback_other[unit] src/execution/resident.test.ts::ResidentExecutor.attach carries the resident's refusal code (record 0097)::a deterministic refusal carries the code the resident answered; src/execution/resident.test.ts::ResidentExecutor.attach carries the resident's refusal code (record 0097)::an absent or unknown code is fallback_other, never a guess from the sentence or the reason; src/execution/resident.test.ts::ResidentExecutor.attach carries the resident's refusal code (record 0097)::a drained fleet that never reopens within the wait is fleet_draining; src/execution/resident.test.ts::ResidentExecutor.attach carries the resident's refusal code (record 0097)::a transient refusal whose wait never clears carries the code of the refusal that began it; src/execution/resident.test.ts::ResidentExecutor.attach carries the resident's refusal code (record 0097)::a definitive re-attach refusal names its own class, never the transient refusal that began the wait; src/core/runSetup.test.ts::run setup words::reads an absent, unknown or bot-only code as fallback_other, never a guess

| Accepted operation authority retains repo/ref with a false or unreachable discovery probe, keeps contextual PRs as evidence rather than publication targets, rejects PR authority whose head differs from the accepted ref, and fails invalid authority rather than substituting | [unit] src/core/repoContext.test.ts::resolveRepoContext: accepted operation authority::* |

A failed coding publication uses the Ship preservation settlement: a verified private artifact precedes destructive release, or ordinary release is deferred to the existing retention policy. Retention is not an archive receipt; mandatory teardown remains in force.

Durable branch publication ​

CriterionProof
PR mutations require committed producer intent; acceptance is saved before auxiliary head/assignee calls. Unavailable intent refuses mutation, and unconfirmed mapping emits no accepted PR event.src/core/codingPrPostStep.test.ts::runCodingPrPostStep (callable with explicit inputs)::commits pending publication before mutation and accepted mapping before auxiliary reads; src/core/codingPrPostStep.test.ts::runCodingPrPostStep (callable with explicit inputs)::refuses the API mutation when pending publication cannot be committed; src/core/codingPrPostStep.test.ts::runCodingPrPostStep (callable with explicit inputs)::does not enter auxiliary work or announce a PR when accepted mapping is unconfirmed

PR mutation requires an available durable run owner. A ledger-disabled or untracked run retains its work and refuses the mutation. Historical runs without a valid complete producer projection remain unverified until the final cutover reconciles their original target; display history cannot reconstruct publication authority.

Full resident reset ​

CriterionProof
Admin repo rebuild --full plans or performs complete resident removal, including dependency backups and thread bindings, then reprovisions with the existing registry settings. It retains the registration, closes new work during reset, refuses active requests or engine cycles, and never starts provisioning after unconfirmed destruction or incomplete cleanup. Retiring the runtime leaves the original operation result unknown.[unit] deploy/cloudflare-resident/reset.test.ts::full resident reset::*; src/core/commands/repo.test.ts::repo rebuild --full::*
Failed current, retired or dependency backup deletion retains its original handles across restart, blocks ordinary rebuild, and prevents complete state removal until checked cleanup succeeds.[unit] deploy/cloudflare-resident/reset.test.ts::full rebuild backup cleanup::*
A resource-bound durable cleanup hold survives lifecycle state changes and key clearing. It refuses hydration, runtime execution and ordinary/manual/automatic provisioning until selected backup handles and the deterministic resident prefix are confirmed empty.[unit] deploy/cloudflare-resident/reset.test.ts::durable full cleanup hold::*

Repository registration without a resident ​

CriterionProof
repo onboard --no-resident persists repo metadata immediately, returns a cold registration, skips provisioning follow-up, and refuses eviction flags[unit] src/core/commands/repo.test.ts::repo onboard::--no-resident registers a cold repo and never polls provisioning; ::--no-resident refuses eviction before contacting the backend
Cold registrations use no resident capacity; legacy entries remain resident entries and conflicting registrations cannot overwrite either mode[unit] src/execution/repoRegistration.test.ts::repository registration without a resident::*
A cold registration is discoverable and fresh coding/PR-less review clones and verifies the effective branch through this run's exact Git door on Cloudflare, E2B and local execution. An existing thread checkout stays untouched, including unpublished commits, dirty/untracked/ignored files and another repository's work; preparation clones beside it rather than using its stale ref, tip or Door endpoint. Explicit task refs win, exact PR/Ship targets retain authority, read-profile Cloudflare reviews keep their isolated run keys, readiness and owner fences hold, and recovery observes the recorded backend/workspace/ref without resetting to the remote or consulting changed defaults; no resident attach or seed[unit] src/execution/factory.test.ts::makeExecutor resident selection::cold registration checkout::*; src/execution/factory.test.ts::makeExecutor resident selection::a generation reclaimed during the resident probe cannot provision cold; src/execution/factory.test.ts::makeExecutor pilot ready environment::refuses a %s repository without a warm snapshot instead of provisioning cold
Resident maintenance, deployment barriers and deterministic resident operations enumerate only resident entries; cold status, listing and offboard never call a resident DO[unit] deploy/cloudflare-resident/repoRegistration.test.ts::metadata-only repo routes::*
Cold registration reconfiguration merges command overrides from the combined resident/repository listing without losing untouched keys, with or without a ref override[unit] src/core/commands/repo.test.ts::repo reconfigure::merges a cold registration's command overrides and optional ref (%s)
The cold coding/PR-less review prompt names the observed branch and existing checkout and promises no dependency snapshot[unit] src/core/dispatch/provision.test.ts::composePrompt — the system prompt for the first turn::%s sees the verified cold registration checkout without a dependency promise; src/core/dispatcher.test.ts::resident repo dispatch::a cold registration keeps its verified default ref in the %s run context and prompt
Live coding and review tasks addressed to a cold registration finish on a per-thread sandbox[gap] requires an operator-selected repository and a deployed Worker; local tests do not prove live task completion
Cold registration uses a dedicated admin route; an older Worker refusing it causes no fallback to resident provisioning[unit] src/core/residentAdmin.test.ts::makeResidentAdminClient (real fetch client)::cold registration uses a dedicated route and never retries onboarding on an older Worker

| Hard stop uses an independent per-run control lock and persists its exact cancellation before a hung attachment can block it. Native UID signals remain original-boot and owner bound. A stable replacement kernel can retire only a previously verified original target, without signalling a successor that reused its UID. Old-epoch death alone cannot declare current work of the same original stopped. When its exact registration independently confirms a new kernel with the same owner, UID and workspace, shutdown stops that runtime and verifies both the current UID absence and old-epoch retirement; an unproved move, pending launch authority or unresolved unbound native submission remains unconfirmed. A separately validated retirement receipt binds original owner, ticket, boot transition and complete journal fingerprints; original journal and control bytes and unknown historic effects remain retained. | [unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::resident force detach shutdown::*; ::resident run cancellation::* | | Runtime retirement uses the original run's ownership and native observations while unrelated refresh work continues. Destruction, original launch authority or thread activity, a running original UID, and changed or unverified native identity keep retirement unconfirmed. Hydration, a recreate hold and exact preserved historical journals alone do not negate verified physical retirement; historical effects remain unknown. | [unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::resident cancellation admission::* | | Failed native shutdown retains its closed observation and refusal under the same original cancellation ticket. The initial and final kill are distinguished, a known exit remains recorded, and raw native errors or output are never retained or returned. The runtime caller validates that observation before exposing it. | [unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::resident run cancellation::retains the exact failed kill observation under its original ticket…; src/execution/runStop.test.ts::run runtime stop::keeps a validated native failure observation for the exact cancellation | | A runtime observation exception returns a closed failure under the original fenced cancellation ticket. If recording that failure also fails, the stop remains pending; no shutdown receipt is invented and no raw provider or storage error is returned. | [unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::resident run cancellation::keeps a runtime observation fault under the fenced ticket without leaking its error; ::keeps the stop pending when saving its runtime failure also fails | | A saved cancellation intent and shutdown receipt bind the exact original ref, UID, kernel boot, workspace, generation and attachment fence. A changed target cannot reuse the receipt, and a legacy completed receipt with no recorded target cannot be silently upgraded into physical shutdown proof. Admission rechecks the canonical binding and ownership in the transaction before saving the target and launch fences. | [unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::resident run cancellation::does not reuse a saved shutdown for a different physical target…; ::does not accept corrupt or foreign cached shutdown state; ::binds an unfinished legacy stop only after the original target is verified; ::does not bind an old stop to ownership that changes during admission | | An unconfirmed native UID kill emits one structured warning with its guard, launch or exit phase, a closed cause, thread and user, and a finite integer exit code when known. Exception text, commands and native output never enter that warning. Confirmed shutdown emits no failure warning. | [unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::resident force detach shutdown::*; ::resident run cancellation::* | | A paired run/thread selector on the existing Read-authorized threads report exposes the exact saved retirement refusal, valid redacted target and current owner matches. Missing/corrupt evidence and successor mismatches are unverified; stored completion is labelled stored-only. It exposes no raw rows, worktree paths, private bytes or errors and performs no report-body write or native control; auth scopes and default listing remain unchanged. | [unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::resident cancellation inspection::*; deploy/cloudflare-resident/nativeOperation.test.ts::resident cancellation inspection HTTP::* | | Cancellation binds data requests to their original run, generation and attachment fence. A cancelled original cannot borrow a successor registration or reattach with a higher fence. Commands, writes, credential inspection and publication use one native admission boundary that checks cancellation before native work, rechecks that the request scope remains active after awaited ownership validation, and settles each admission at most once. Admitted native calls remain durable while awaiting their actual result; unknown outcomes remain held across ordinary resets; only exact hash/owner-matching, physically retired scopes leave active-debt counts, with full paging so foreign operations remain visible; explicit full rebuild is a separate destructive action and does not certify their original result. The legacy UID path waits for admitted work, including privileged publication, kills the exact UID again after delayed launches settle, and independently verifies the original native boot identity and no UID processes, or separately proves the original runtime retired on a stable replacement boot with no outstanding unbound native submissions. Neither path clears an unknown operation journal. Terminal workspace registration and settlement remain readable for metadata-only reconciliation. | [unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::resident cancellation admission::*; ::resident run cancellation::*; deploy/cloudflare-resident/runRegistration.test.ts::cancelled owner attachment::*; src/execution/resident.test.ts::ResidentExecutor.exec::binds commands, reads, writes and publication to the caller's original run owner; deploy/cloudflare-memory/runLedger.test.ts::original workspace durability in real SQLite::cancelled SQLite resident ownership remains readable and can be acknowledged without a VM |

Native operation recovery ​

Per-run native containment changes execution admission, not UI authorization or the meaning of an unknown remote result.

CriterionProof
Native execution is frozen at first canonical actor admission. A queued plane placeholder binds its first execution client's protocol before workspace effects. Legacy records cannot gain native birth after metadata or settled journals disappear. Old clients cannot reclaim or re-claim a native-born run.[unit] src/core/runLedger/workspaceDurability.test.ts::native execution first admission::*; deploy/cloudflare-memory/runLedger.test.ts::the plane's admission stage — /plane/admit, reservations, the seal's walk (orchestration-plane; record 0064)::freezes a queued run's protocol at its first actor claim and refuses an old-client takeover ; deploy/cloudflare-resident/nativeScope.test.ts::native run birth::*
A run reserves its immutable native gate before bootstrap and its attempt before launch. Lost bootstrap replies are observed under the same identity. A closed reservation prevents delayed bootstrap and launch. Closure uses the original scope independently of mutable workspace metadata.[unit] deploy/cloudflare-resident/nativeScope.test.ts::native run scope reservations::*; deploy/cloudflare-resident/workspaceSettlement.test.ts::resident cancellation admission::*
An exact cancellation ticket with original native review targets can close Resident admission before scope birth without waking a VM. It persists the existing cancellation tombstone, refuses a delayed birth, and retains any contradictory registration, legacy protocol or operation evidence.[unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::native startup cancellation::closes the original native review admission before scope birth and rejects delayed birth
Native binary output preserves the existing binary read size limit without truncating a valid base64 chunk.[unit] deploy/cloudflare-resident/nativeScope.test.ts::native binary output::*
Versioned Resident execution and stop requests refuse on old receivers before any SDK work. Native submission and collection have a deadline. A lost initial launch reply can only close its original caller-known attempt; no command replay or PID guess occurs. Reset before close submission resumes the same monotonic close after a positive observation.[unit] deploy/cloudflare-resident/nativeScope.test.ts::native scope transport deadlines::*; ::native run scope reservations::resumes an exact close after reset before its submission; src/execution/runStop.test.ts::run runtime stop::*
A confirmed closed, empty scope or a fresh different kernel can retire only its exact execution debt. Original unknown journals remain unchanged; no remote publication or private-write result is invented. Changed gate, hash, owner or ticket evidence remains unresolved.[unit] deploy/cloudflare-resident/nativeOperation.test.ts::native scope execution retirement::*; deploy/cloudflare-resident/nativeScope.test.ts::native run scope reservations::retires only the original scope after a fresh different kernel observation
Shared hydration, dependency producers, cache adoption, Git setup and background backups have independent maintenance ownership. A consumer's cancellation preserves that shared work and its unknown results. Selected thread dependency views, commands, writes, credentials and publication remain run-owned.[unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::shared maintenance ownership::*
The native image helper confines ordinary and privileged payloads before their first instruction, denies the SDK control port, protects writable control roots, preserves build environment limits and keeps original-namespace UID inventory visible. Its private proc view is read-only and its private writer rejects linked targets before truncation. Kernel subtree kill stops detached children while preserving a neighbour.[agent] Run npm run check:native-scope on Linux with Docker; it exercises the actual helper in a disposable private cgroup namespace with explicit local capabilities. This is local source proof. Repeat on the exact Cloudflare Resident image and policy before live acceptance.
Native execution gives each original thread/run a private persistent /var/tmp view. Pi and OpenCode control files remain available across calls and owner generations, outside model /tmp cleanup. A later run on the same thread, even with the same UID, cannot read or change prior harness state. Host /var/tmp, controller paths, kernel controls and unrelated runs remain protected. Scope closure retains harness files.[agent] Run npm run check:native-scope with a qualified local image. It renders writes through ExecHarnessContainer and both real harness path builders, then checks actual filesystem/FIFO contents, reattach, same-thread/same-UID peer isolation, host preservation, scratch cleanup and existing kernel closure assertions in the installed helper. Exact provider/UI acceptance remains separate.
Pre-migration unknown SDK submissions are not adopted into a new gate. An unavailable kernel, missing/corrupt identity or foreign target stays unconfirmed; scoped closure does not authorize destroying the shared resident or backups.[unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::resident cancellation admission::*; deploy/cloudflare-resident/nativeOperation.test.ts::native operation journal::*; deploy/cloudflare-resident/workspaceSettlement.test.ts::legacy cancellation admission::*

The original-owner journal adds a supported exact-result recovery action; record 0101 records the boundary and remaining launch-loss gap.

CriterionProof
The read-scoped native-operation inventory is bounded and reads stored metadata only. It reports legacy records without inventing process identities.[unit] deploy/cloudflare-resident/runRegistration.test.ts::resident workload admission::lists legacy native operations without inspecting a runtime; deploy/cloudflare-resident/nativeOperation.test.ts::native operation journal::metadata inventory is bounded and preserves legacy evidence
Thread and operator requests share a journal of native admissions. A known result settles its exact attempt; closure cannot erase an unknown attempt. Each acknowledged process identity can be recovered from the original SDK process result without executing the command again.[unit] deploy/cloudflare-resident/nativeOperation.test.ts::native operation journal::*; deploy/cloudflare-resident/runRegistration.test.ts::resident workload admission::recovers a lost operator result without replaying its command
Lost storage acknowledgements qualify only through exact original attempt or full-identity readback. Missing, changed or unreadable evidence stays held; a scope closed during admission cannot submit. Lost native launch acknowledgements remain unknown and never authorize replay under the pinned SDK.[unit] deploy/cloudflare-resident/nativeOperation.test.ts::native launch storage acknowledgements::*; deploy/cloudflare-resident/runRegistration.test.ts::resident launch acknowledgement recovery::*
Acknowledged process identity is recorded before unrelated reachability-memo cleanup. Both common execution and credential inspection keep collecting the same handle after a narrow status wire-observation failure; only an original terminal result and committed settlement release its attempt. Runtime, authority, owner/fence, cancellation, attachment, identity parsing and journal-persistence uncertainty remain refusals. The first closed status diagnostic excludes raw error/output/credentials; no exec replay occurs. Lost native launch replies remain unsupported; storage ACK loss qualifies only through the exact readback criterion above.[unit] deploy/cloudflare-resident/runRegistration.test.ts::native process identity collection::*; deploy/cloudflare-resident/credentialInspection.test.ts::resident credential diagnostic::collects the original credential-inspection handle after observational identity failure, ::retains unknown native outcome when credential collection also fails, ::does not collect after an authentication-denied identity observation
Admin recovery compares the exact stored record and original process identity, requires terminal outcomes for every admitted attempt, and atomically archives evidence before removing only that unresolved record. Active work, changed ownership or records, missing results, lost launch replies, opaque writes/publication effects and legacy records remain held. Neither recovery nor a terminal process receipt removes workspaces or changes run verdicts.[unit] deploy/cloudflare-resident/nativeOperation.test.ts::native operation journal::*; deploy/cloudflare-resident/runRegistration.test.ts::resident workload admission::recovers a lost operator result without replaying its command
The HTTP read bearer can inventory stored metadata; only admin may request exact recovery, which never opens ordinary fleet admission.[unit] deploy/cloudflare-resident/nativeOperation.test.ts::native operation HTTP permissions::*
Explicit admin termination signals only the matching process capability from the original runtime after journal and owner checks. Its control request and acknowledged or unknown outcome are durable; an uncertain termination cannot be replayed. Only an original terminal result can settle the journal, and unrelated work remains untouched.[unit] deploy/cloudflare-resident/nativeOperation.test.ts::native operation journal::terminates only the original recorded process and preserves unrelated work; ::refuses termination when the original owner or process identity changes; ::records an unknown termination outcome durably and refuses replay after restart
Global deploy permits use a caller-known UUID and immutable request hash before the first RPC. A lost reply qualifies only through one exact admitted or released readback in the original living caller; no admission, cleanup or native work is replayed. Release atomically saves a receipt and removes the exact permit. At most 128 release receipts are retained; absence, eviction, foreign identity, malformed evidence and caller death stay unknown or held. Legacy callers may release only legacy permits, and expiry never removes debt. Fenced responses report producer identity, withdrawn/retained disposition, and omitted versus failed metadata reads without changing the upload gate.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::*; deploy/cloudflare-resident/preflight.test.mjs::resident deploy admission diagnostic::*. [gap] Real Cloudflare storage/RPC acceptance, original legacy request association and separately qualified recovery, and rollback to a pre-receipt producer remain unproved; no source test or diagnostic establishes native ending.
A Resident call preserves its original result or error when permit release cannot be confirmed. A completed check saves its output and refuses repeat execution. A retained permit still blocks deployment and keeps its separate release diagnostic.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::keeps the original Resident error when permit release remains unverified; ::saves the completed check once while failed permit cleanup still blocks deployment
Deployment diagnostics distinguish operation holds from run owners and include bounded record keys, process identities and recovery reasons without changing the safety gate.[unit] deploy/cloudflare-resident/preflight.test.mjs::resident deploy preflight — decide()::names the held operation and original process instead of guessing that its run owner is unknown
A refused preflight retains a closed 16 KiB projection of the already-returned native-operation page in its existing log/artifact, with original hashes, owners, whole bounded pending maps or explicit unavailable detail, returned controls and page/record omissions. Received-projection completeness never proves complete control storage or native ending. It makes no extra request and diagnostic failure leaves admission, force and exit behavior unchanged.[unit] deploy/cloudflare-resident/preflight.test.mjs::resident deploy preflight — main()::*
A non-successful deploy-fence HTTP response remains a refusal and is reported as a Worker response, not a missing consultation. A sent request whose reply is lost keeps its outcome unknown, including a received HTTP response whose body fails; it requires original-request reconciliation before retry. The client reads at most 8 KiB of error evidence and logs only closed source-backed reason classes, phase, validated producer and admission facts, evidence availability and a safe inspection step. The legacy acquisition error does not identify which fence predicate failed. Missing facts remain unavailable; no response body, credential or private path is copied, and no extra request, recovery or release occurs. Diagnostic failure does not change the gate.[unit] deploy/cloudflare-resident/preflight.test.mjs::resident deploy fence HTTP refusal::*
Live recovery resolves an operator-selected original operation after a lost reply without replay or workspace changes.[gap] requires the reviewed Worker, original scoped admin authority, exact resource/record and retained original native process status; synthetic tests do not prove deployed recovery

| Offboard can retry the exact durable cleanup target after registry and resource rows are removed; dry-run stays read-only and successful checked cleanup releases the hold. A missing or different target returns 404. | [unit] deploy/cloudflare-resident/reset.test.ts::offboard pending cleanup recovery::retries prefix cleanup after registry and resource removal, including restart | | A recorded Resident selection with an incomplete upstream binding resolves only a positive matching native registration, fence and UID target under the same cancellation. Missing or foreign custody returns unavailable without a signal; target discovery is not source promotion or proof of no workspace. | [unit] deploy/cloudflare-resident/workspaceSettlement.test.ts::resident cancellation admission::resolves only a positively owned original target when its upstream binding is missing; src/execution/runStop.test.ts::run runtime stop::asks the recorded Resident backend to resolve an incomplete original target without attaching | | Retired native history is preserved byte-for-byte and excluded from active debt only while its immutable fingerprint, owner, generation and fence match a confirmed original-epoch receipt. Complete paging reveals foreign work beyond retired pages; altered records stay unresolved. | [unit] deploy/cloudflare-resident/nativeOperation.test.ts::native launch storage acknowledgements::keeps original retired history and finds foreign active work beyond a full retired page; ::preserves retired journal bytes across late callbacks and refuses renewed launch or recovery |

Original recorded check inspection ​

/inspect-check and /settle-check require the existing operator credential and accept only run, call and phase. Each receiver resolves Memory's canonical source independently. Inspection never changes an alias. Settlement only consumes a sealed archive and a unique matching admission alias plus an exact stored native-ended receipt; physical retirement, run status and a missing journal do not prove ending. Available completed output stays available. Exit124/137/143 output remains partial and unknown; without native-ended proof it cannot become known or be pruned as completed. With proof, retirement keeps its partial payload. Ending proof takes precedence over an output-only tombstone. Same-owner no-repeat tombstones remain, and other unknown aliases are untouched.

CriterionProof
Stored native ending proof validates exact original key/hash and either original process tuples or a qualified same-attempt native timeout receipt, without lookup, signal or mutation.[unit] deploy/cloudflare-resident/nativeOperation.test.ts::stored original native endings::*; ::retained native timeout ending::*
A native timeout retains its exact scope/boot/attempt empty-group acknowledgment and original launcher completion before removing the sole pending admission. Interrupted output stays unknown; failed writes, uncertain readback, foreign/partial proof, other pending work and later timeouts cannot authorize removal or overwrite the receipt. Missing-proof refusal emits an identity-only diagnostic; a failed log sink preserves the original refusal and hold.[unit] deploy/cloudflare-resident/runRegistration.test.ts::native timeout ending custody::*; deploy/cloudflare-resident/nativeOperation.test.ts::retained native timeout ending::*; deploy/cloudflare-resident/nativeScope.test.ts::native timeout closure validation::*
Canonical result fallback, unique alias/owner/key/hash checks, physical-retirement refusal, read-only inspection, exact tombstone retirement and lost-ACK readback preserve original evidence and no-repeat behavior.[unit] deploy/cloudflare-resident/checkOperation.test.ts::original recorded check inspection::*
Selector-only authenticated routes derive routing from canonical source and bind responses to the final source read before Gateway displays them.[unit] deploy/cloudflare-resident/checkOperation.test.ts::recorded check receiver::*

Durable resident placement ​

Before attach touches a resident, the registry commits the repository/thread's immutable placement in the same transaction that checks warm repository registration. Each row records schema, repository, thread, resident address and placement ID. Address metadata does not grant execution authority or certify a physical runtime. New capable clients select a serving member from the desired warm set. Existing placements never move. Legacy requests can use only a primary placement; a saved secondary requires its exact address.

CriterionProof
Duplicate placement requests and registry restart return the original destination; separate threads and repositories retain separate rows.[unit] src/execution/residentPlacement.test.ts::durable resident placement::returns the first saved destination after a restart and a changed proposed destination; ::keeps separate placements for separate repositories and threads; deploy/cloudflare-resident/repoRegistration.test.ts::primary resident placement::concurrent requests and a new registry instance recover one primary placement
A committed write with a lost acknowledgment is recovered by exact lookup; an uncommitted failure yields no destination and a later call can create the first row. Corrupt or foreign stored rows refuse without overwrite.[unit] src/execution/residentPlacement.test.ts::durable resident placement::reads the original placement after a committed write loses its acknowledgment; ::allows a new placement after a write did not commit and no destination was acknowledged; ::refuses corrupt saved placement without overwriting it…
Missing and cold repository registrations refuse placement. An unsupported stored replica refuses instead of falling back to the primary.[unit] deploy/cloudflare-resident/repoRegistration.test.ts::primary resident placement::refuses missing and cold repositories while a warm repository can still attach; ::refuses an unsupported saved replica instead of routing to the primary
The attach route waits for committed placement before returning a resident destination; a storage exception prevents that route from proceeding.[unit] deploy/cloudflare-resident/repoRegistration.test.ts::primary resident placement::routes attach only after primary placement commits and stops on a lost storage reply
Multiple warm replicas, complete legacy-binding import, replica-aware run recovery and cancellation, and warm-idle retention operate together.[gap] In a disposable scope, request two warm members, reconcile under the deployment fence, and prove old/new thread placement, resumed commands, original-target cancellation with an unaffected sibling, and idle retention. Local source tests do not prove Cloudflare activation or sustained warm capacity.

Exact resident address ​

An attach answer can include its immutable residentId and placementId. The executor records both with the workspace before commands resume. Every later request retains that address, and status/restore probes on a resumed run name the same thread and address. The Worker compares an explicit address with the saved placement before routing. The run owner, attachment fence, kernel identity and deployment admission remain required independently. Legacy requests without either field retain the primary address only when no secondary placement exists for that thread. Malformed or partial address fields cannot become legacy fallbacks. Before an attach acknowledges the address pair, status, restore and wake probes still carry the thread key and discover its saved placement. These read-only probes never allocate placement.

CriterionProof
Workspace serialization and terminal metadata retain the replica address. Cancellation matching distinguishes two replica addresses and two placement identities even when other fields match. Malformed saved address metadata cannot authorize legacy cancellation.[unit] src/execution/residentRoute.test.ts::resident runtime address::retains the replica through workspace serialization and terminal settlement; ::refuses cancellation of a different replica or placement even when the old target fields match; ::keeps malformed address metadata from becoming a legacy cancellation target
After attach, commands and later reattachments send the saved address. An attach reply that changes or omits an expected address refuses before commands resume.[unit] src/execution/residentRoute.test.ts::resident runtime address::sends the acknowledged address on commands and later reattachments; ::rejects an attachment answer that changes or drops the saved address
Resumed status and restore waits select the original replica, and the reattached binding retains the address.[unit] src/execution/residentRoute.test.ts::resident runtime address::selects the exact recorded resident for status and restore waits; src/execution/factory.test.ts::makeExecutor resident selection::on a resume (ctx.reattach): re-attach, never re-provision::a resumed resident probes and reattaches only its saved replica address
Restore and wake probes use the thread's saved destination before its first successful attach, including a drain-to-wake handoff. Explicit saved pairs stay exact. Unassigned capacity probes and legacy primary discovery allocate no placement. A failed member probe logs its repository, physical address and sanitized error; other serving candidates remain available.[unit] src/execution/factory.test.ts::makeExecutor resident selection::a restoring resident — the one held /await-restore request (item 27)::restores the thread's saved secondary while the primary is down…; src/execution/residentRoute.test.ts::resident runtime address::recovers a committed secondary before attach acknowledges its address…; deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet lifecycle::discovers saved thread placement and probes unassigned capacity without allocating or moving it
A failed reattachment with an explicit resident address cannot restart without that address. An ordinary resume with malformed saved metadata or a changed/missing expected reply ends with a setup refusal; existing pilot preservation still takes precedence.[unit] src/core/dispatcher.test.ts::run ledger write-through (docs/reference/specs/run-history.md item 35)::an ordinary resume with a %s resident address fails setup without restarting on the primary
The data route accepts only the saved thread/address pair; a different replica refuses without replacing that pair. Legacy data requests retain primary routing.[unit] deploy/cloudflare-resident/repoRegistration.test.ts::primary resident placement::a data route honors only the exact saved replica and leaves legacy traffic on the primary

Fixed warm pools ​

repo reconfigure <slug> --warm-residents N reserves capacity for the requested count. The reply separates the requested count from registered members. New addresses stay pending until post-upload deployment reconciliation imports historical primary placements under its exclusive swap fence. A lower count stops new placement on excess members but retains their addresses, work and capacity reservations. Only explicit offboard removes the pool. A later lower fleet cap blocks new addresses, but still permits changes that use retained members.

CriterionProof
Pending and retained addresses count against the fleet cap. Pending members receive no new placement. Complete fenced import activates the pool; old threads stay on the primary and saved secondary threads stay there after downscale.[unit] deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet registry::reserves pending members against the cap and activates only after a complete fenced import; src/execution/residentMembers.test.ts::resident replica membership::*
A later lower cap permits a lower warm count or warming retained members without adding addresses; new addresses remain capped.[unit] deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet registry::allows a lower warm count and retained members above a later reduced cap
Import requires one reconciliation permit and the same live swap fence. Evicted bindings, run registrations/fences, dynamic UID owners and legacy spent owners participate. A changed fence restarts enumeration; a conflicting page rolls back.[unit] deploy/cloudflare-resident/replicaMigration.test.ts::replica activation maintenance::*; deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet registry::restarts enumeration after a fence change and rolls back a conflicting page
One storage page can expand to more than 256 legacy owners. Bounded chunks commit primary placements, but only the final chunk advances the storage cursor or activates the pool. A lost chunk reply replays the page without replacing prior placements.[unit] deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet lifecycle::imports a large legacy owner page in bounded chunks and replays a lost chunk reply before advancing
Offboard retains the catalog and placements until all active members have removal receipts. Both the initial registry transition and per-member removal bind the original pool revision. Delayed retries cannot destroy a newly onboarded primary. A retired address cannot be initialized by a stale watchdog. Cleanup checks that member's storage prefix.[unit] deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet registry::retains offboard catalog and placements until every member has a removal receipt; ::resident fleet lifecycle::serializes offboard retries and rejects an old removal after the primary is onboarded again; ::resident fleet lifecycle::refuses a delayed offboard after its initial pool was removed and replaced; ::resident fleet lifecycle::keeps a retired address closed to a stale watchdog after cleanup
Desired members continue refresh when idle; excess members retain ordinary idle parking. Refresh instances have separate addresses and payloads.[unit] deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet lifecycle::keeps desired members on the warm cadence while excess idle members can park; deploy/cloudflare-resident/refresh.test.ts::the refresh module — the Workflow entrypoint leaves worker.ts, which re-exports it for the binding::creates separate refresh instances and payloads for two residents of one repository
Fresh primary and secondary provisioning report the exact physical member after warm; a failed or unacknowledged report retains its pending marker. Reconciliation forwards Worker version and timestamp, and one rejected member cannot replace a sibling's successful result.[unit] deploy/cloudflare-resident/drain.test.ts::fresh provisioning image reports::retains the warm member's report until the registry acknowledges it…; deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet lifecycle::reconciles siblings separately and holds the exact unverified member
Reconciliation and readiness distinguish physical members of one repository. An unverified member holds its own address. Native-operation recovery requires a member number for a multi-member pool and retains the original journal selector.[unit] deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet lifecycle::reconciles siblings separately and holds the exact unverified member; ::resident fleet lifecycle::requires a member for native recovery and retains its original journal selector; src/deploy/residentReadiness.test.ts::*
Active count still equals the listed resident rows for readiness. reservedCount includes pending and retained addresses and excludes cold repositories. Chat, initial dashboard seed and live feed show reserved, active and pending counts separately; older listings retain their prior output.[unit] deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet lifecycle::reports active rows separately from pending and retained slot reservations; src/core/commands/repo.test.ts::repo.list::shows reserved slots separately from active and pending replicas; src/channels/residentsView.test.ts::createResidentsViewHandler::carries pending replicas into the initial seed without changing the active count; web/src/pages/residents.test.ts::ResidentsIndexPage::shows reserved slots before replicas activate and updates active and pending counts from the feed
The dashboard lists separate residents, joins a run to one member, and opens the requested member detail. Unknown addresses cannot show the primary instead.[unit] src/channels/residentsModel.test.ts::*; src/channels/residentsView.test.ts::createResidentsViewHandler::opens the selected replica and refuses an unknown address without showing its sibling
Numeric debug workload inspection uses the selected catalog member and its stored physical identity. Raw thread-placement fields remain invalid. An omitted number keeps workload inspection on the primary, including a pool; other pool debug operations still require a number.[unit] deploy/cloudflare-resident/replicaFleet.test.ts::resident fleet lifecycle::routes workload inspection to the selected member and rejects raw placement fields; deploy/cloudflare-resident/runRegistration.test.ts::resident workload admission::labels separate members and their admission refusals with the stored physical resident ID; ::does not label an uninitialized or mismatched secondary as the primary
The command requests a pool without treating pending addresses as ready. Member rebuild uses a dedicated endpoint so an older Worker cannot ignore the selector.[unit] src/core/commands/repo.test.ts::repo reconfigure::requests a warm pool without reporting pending replicas as active; src/core/residentAdmin.test.ts::*

Parent work targets ​

A managed PR child uses its parent's admitted publication binding to form the typed operation target. This target belongs to one work item, not its conversation. Prompt links and history from other work do not select another PR. The resolver still checks access and reads the native PR head. Native head, workspace and publication-owner checks remain required before effects. A resumed child keeps its original reservation target through promotion while resolving from the parent tag. Standalone Review requests keep their existing explicit target resolution.

CriterionProof
Parent targets survive competing child citations and history; separate repositories can carry the same PR number in one conversation. The persisted target retains parent provenance.[unit] src/core/repoContext.test.ts::resolveRepoContext: accepted operation authority::resolves each parent work target despite another PR in the same conversation; src/core/dispatcher.test.ts::*::a managed review inherits its parent target before reading misleading child text

Explicit staging risk ​

CriterionProof
One selected staging record stays stored with an immutable abandonment audit and unobserved ending. The common global gate excludes only its exact matching record; a neighbor still blocks.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::excludes only the exact staging record while retaining its original data and unknown result; ::commits one exact staging abandonment audit without deleting the request
Maintenance survives ordinary drain removal, alarm expiry and last-image callbacks. Only an authenticated fresh manual release can reopen the selected fixture after the upload window expires; the original record remains and the window is not renewed.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::installs a durable hold that refuses new work after ordinary drain removal; ::keeps maintenance through ordinary alarm and last-image callbacks; ::uses the authenticated HTTP risk path and releases only through a fresh decision after expiry
Retained deployment reports image readiness under its exact hold, makes no undrain request and rejects another source. Cancellation retains maintenance. The staging controller waits for an explicit matching release within its deadline.[unit] src/deploy/residentDrainRun.test.ts::retained staging deployment::*; src/deploy/staging.test.ts::isolated staging deployment::waits only for the selected explicit release and stops at its deadline
Selected maintenance binds the exact full source commit from the structured build stamp. A real timestamped build can initialize, claim, reconcile, seal and release without changing source authority. A later build of the same commit preserves its held/released generation; wrong, dirty or unknown source and an unselected changed generation refuse. A fresh successor needs the exact released predecessor and preserves its original records. Timestamped artifact IDs and Worker upload identities keep their separate semantics.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::starts a stamped Registry and completes selected claim, reconciliation and release without changing source authority; ::admits the selected post-upload reconcile from a stamped source and refuses wrong source or generation; ::refuses wrong, unknown and dirty stamped source or a changed held generation at Registry startup; src/deploy/buildStamp.test.ts::buildId (what a stored artifact compares against)::*

The operator sequence uses these existing HTTP fields. The selected attempt object is { ...policy.attempt, holdId: policy.holdId, generation: policy.generation }; nested policy fields are defined by stagingMaintenanceSchema in src/deploy/residentMaintenance.ts. A request cannot enable or replace that trusted policy.

RouteExisting bearer scopeRequest or returned fields
GET /deploy-admissions?id=<original-id>adminReturns admission.id, admission.startedAt, admission.recordSha256; the digest covers the full stored record.
POST /draindrainSelected decision drain: minutes: 90, reason, by, attempt. The held policy clamps its end to the fixed deadline.
POST /staging-abandon-admissionadminid, expectedStartedAt, expectedRecordSha256, expectedFixtureSha256, expectedDecisionSha256, attempt. This handler sets the decision fence itself before its atomic risk decision.
POST /reconciledrainRequires imageTarget: { application, version, image } from the authenticated native application read after upload, plus attempt while selected maintenance is held. The application must be this Worker's Resident application; the version is a nonnegative integer and image is a nonempty reference. The response echoes the accepted imageTarget beside each member result. GET /residents exposes each current-artifact member's live.imageTarget.
POST /staging-seal-windowdrainattempt; the normal retained runner calls it after exact readiness. A new Worker version after the fence and zero blocking obligations are required.
POST /staging-release-maintenanceadminexpectedStartedAt, expectedRecordSha256, attempt, releaseDecisionSha256. The new decision digest must differ from the original risk decision.
GET /staging-maintenancereadReturns current maintenance, its policySha256 and successor installation receipt. Optional holdId selects retained history without changing current selection. Window state is a projection, not renewed authority. First contact can finish initialization of the already uploaded trusted policy; it does not select or repeat an operation. It does not return a full account/resource inventory.

Selected bootstrap producer ​

CriterionProof
A selected force bootstrap compares the qualified old Cloudflare Worker version UUID to the authenticated fence’s swapVersion UUID; newer producer diagnostics, when present, must agree. The source commit and timestamped build remain separate. The selected fixture needs complete idle member and fence data; only one independently qualified old global admission can remain. A changed or unverified producer refuses even under force; an HTTP denial or malformed response cannot become a bootstrap fallback. Calls without the bootstrap version guard retain normal preflight behavior.[unit] deploy/cloudflare-resident/preflight.test.mjs::selected staging maintenance preflight::refuses a changed bootstrap Worker version through the public preflight even under force; ::accepts the known-old fence Worker UUID without newer diagnostics; ::never lets a selected bootstrap force an HTTP refusal or malformed response; deploy/cloudflare-resident/preflight.test.mjs::*

Staging successor operations ​

CriterionProof
A rendered trusted policy can install a distinct next generation only from the exact released predecessor, completed window, release receipt, immutable audit and unchanged original request. Account, script, origin, resource and fixture scope stay fixed. One transaction creates the current pointer, hold and installation receipt; consumed predecessor records are unchanged.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::installs a separately selected successor without changing consumed predecessor evidence; ::refuses changed predecessor evidence and leaves the stored episode untouched
Failed storage commits roll back. Lost acknowledgement is reconciled by exact same-policy startup without another installation or deadline renewal. Missing pointer/receipt, changed retained evidence or a competing successor refuses rather than restoring legacy state.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::rolls back a failed installation receipt without changing predecessor or pointer; ::reconciles a committed installation after lost acknowledgement without renewing it; ::refuses missing selection or changed retained evidence instead of falling back to legacy; ::rejects a competing successor selected from an already superseded predecessor
Each successor needs a fresh explicit risk decision for the same original request. Its own audit/window is consumed by normal claim, fence, seal and explicit release. Historical hold, window, release and original audit remain unchanged; original request ending is still unobserved and debt blocks again after release. Metadata history reads do not change current selection.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::uses a fresh successor risk window through normal claim seal and explicit release
The selected bootstrap admits only an exact released predecessor under its original one-shot CI identity, source/Worker UUID, immutable inputs and readiness reserve. It accepts the authenticated withdrawn legacy fence under separately qualified physical preconditions; ordinary force remains refused. It verifies the new installation receipt and a null risk window.[unit] deploy/cloudflare-resident/preflight.test.mjs::selected staging maintenance preflight::admits only the separately selected exact released predecessor under successor bootstrap; src/deploy/stagingHeldBootstrap.test.ts::held CI bootstrap::bootstraps the exact released predecessor and verifies its new installation receipt; ::refuses changed released predecessor or missing new installation proof
Source merge does not prove uploader containment, native/member/owner/private readiness, a successful installation or later E2E acceptance. Unknown prior effects stay unknown and no original operation is replayed or renewed.[gap] operator-qualified successor installation and normal CI acceptance on the exact selected containing candidate.

| A successor may retain acknowledged terminal workspaces with positive coarse counters. Final fenced executing/unknown activity must be zero and retention counts complete; native, owner, UID, private and image guards remain independent. It does not require cleanup to zero a display and does not prove native ending. | [unit] src/deploy/stagingHeldBootstrap.test.ts::held CI bootstrap::keeps acknowledged retained workspace counters for the final fenced activity classification; deploy/cloudflare-resident/preflight.test.mjs::selected staging maintenance preflight::admits only the separately selected exact released predecessor under successor bootstrap; deploy/cloudflare-resident/runRegistration.test.ts::cancelled owner attachment::* |

| Refusals name the selected hold, generation and closed failed condition. Invalid identifiers remain explicitly unverified; raw policies, request bodies and secret values are not included. | [unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::refuses changed predecessor evidence and leaves the stored episode untouched; ::refuses missing selection or changed retained evidence instead of falling back to legacy |

The trusted policy's optional predecessor quote uses these field names. All values identify the prior released operation; they do not select new values or certify original ending.

text
predecessor = {
  holdId, generation,
  attempt: { id, controllerId, sourceCommit, profileSha256, configSha256, deadline },
  fixtureSha256, decisionSha256, policySha256, releasedAt, releaseDecisionSha256,
  request: { id, startedAt, recordSha256 }
}

Attached browser work scope ​

Runtime infrastructure is shared. A trusted personal scope returns unavailable runtime data and names the All people view; it does not report empty infrastructure or zero counts. All people and unscoped legacy callers retain current authorized runtime reads.

CriterionEvidence
Shared runtime data is withheld only for an attached personal scope, while unscoped reads retain the live listing[unit] src/channels/residentsView.test.ts::createResidentsViewHandler::withholds shared runtime rows only for an attached personal view and preserves legacy unscoped reads

| Unavailable shared runtime shows no stream or invented zero statistics | [unit] web/green/pages/personalReports.test.ts::Green personal reports::* |

Replacement Runtime facts ​

CriterionEvidence
Runtime details retain provisioned/refresh times, worktree dependencies and exact eviction time/reason alongside the shared disk admission facts[unit] web/src/pages/residents.test.ts::ResidentDetailPage::shows the interesting facts: state, ref, sha, lockfile hash, provisioned/refreshed, snapshot stamp, schedules, command table, ::lists thread worktrees (ref, sha linked to its commit, deps mechanism, attach times), newest first, marking evicted ones

Terminal native scope cleanup ​

CriterionProof
An admin can retire one exact ended run's original native scope with a terminal-purpose permit. The receiver freezes original journal fingerprints before control, keeps neighboring scopes active, and retains history, workspaces and unknown command outcomes.[unit] deploy/cloudflare-resident/nativeOperation.test.ts::terminal native scope retirement::closes only the selected terminal scope and retains its original unknown bytes; ::authorizes the real terminal HTTP receiver and freezes its exact Memory permit
Missing terminal authority, changed owner/scope/hash, malformed selectors or an oversized inventory cannot start native control. Older authority replies cannot fall back to cancellation or process control.[unit] deploy/cloudflare-resident/nativeOperation.test.ts::terminal native scope retirement::*; deploy/cloudflare-resident/nativeScope.test.ts::native run scope reservations::refuses a changed expected scope before native control or closing the original
Prepared journal bytes remain immutable across late begin, identity, settlement, close and recovery callbacks. An identical retry continues only the prepared scope's monotonic closure; a completed receipt needs no further native request. Lost replies remain unconfirmed until exact readback succeeds.[unit] deploy/cloudflare-resident/nativeOperation.test.ts::terminal native scope retirement::*

Ordinary deployment preparation ​

CriterionProof
An ordinary deployment check records one exact acquisition and seals existing upload readiness before returning permission. New work remains fenced. A positive observed admission count withdraws preparation before sealing, even if that admission finishes before the seal transaction.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::seals one original ordinary fence before returning upload readiness; ::withdraws before sealing when the last admission ends after its count is read
Cancellation before the readiness seal withdraws only the original acquisition, retains the ordinary drain and independent work, and fences a delayed preparation by its tombstone. A lost sealed reply stays unknown.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::cancels a stalled preparation and cannot seal or clear a later neighboring acquisition; ::retains a cancellation tombstone that arrives before its delayed preparation; ::keeps a sealed fence unknown after a lost ready acknowledgment; ::retains independent fleet debt when an idle native view cannot authorize sealing
Lost preparation and abandonment replies can be qualified only by the original retained disposition. Failed writes roll back; malformed authority and changed producers cannot remove or seal a fence.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::withdraws the exact preparation when its installation acknowledgment is lost; ::qualifies a lost abandonment reply only from the original retained disposition; ::keeps a lost cleanup acknowledgment unverified when its original receipt cannot be read; ::rolls back fence installation when its acquisition record cannot commit; ::rejects an unreadable retained acquisition instead of clearing its fence; ::binds mutations to the actual Registry producer and refuses changed producers
Legacy tuple controls cannot replace, validate or clear an acquisition fence, including a malformed selector. Hold and image rewrites preserve selector presence.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::preserves malformed selector presence through hold rewrites and rejects legacy controls; ::does not let a delayed tuple-only admission recovery borrow an acquisition fence; ::commits one exact staging abandonment audit without deleting the request; ::starts a stamped Registry and completes selected claim, reconciliation and release without changing source authority
Existing post-upload, staging release and expiry authority stays independent. Whole-drain removal never fabricates an acquisition receipt; guarded admin recovery can quote only a sealed acquisition.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::keeps post-upload hold and image-report removal independent of acquisition history; ::admits only post-upload reconcile and retains unrelated permits; ::starts a stamped Registry and completes selected claim, reconciliation and release without changing source authority; ::allows original expiry without inventing abandonment or reusing the acquisition id; ::quotes only sealed acquisitions for the existing independent admin recovery
Known-old, staging, explicit force and qualified bootstrap paths retain their existing protocol. A failed new protocol request cannot fall back to a legacy mutation. One preparation deadline and one separate cleanup deadline remain bounded.[unit] deploy/cloudflare-resident/preflight.test.mjs::ordinary deployment acquisition client::*; deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::cannot fall back to an old Registry mutation when acquisition RPCs are unavailable
Incoming request cancellation is consumed only by the new preparation handler; an ordinary reconcile request retains its existing adapter contract.[unit] deploy/cloudflare-resident/drain.test.ts::global deploy admission evidence::keeps an aborted ordinary reconcile request on its existing adapter contract
A deployed receiver advertises the new capability and delivers real client-disconnect events; an ordinary deployment uses its sealed receipt successfully. Retained acquisition volume is measured before live acceptance.[gap] Requires release-owner qualification on the deployed build. Local AbortController tests do not prove Cloudflare signal delivery or live deployment acceptance.

Explicit operator abandonment of one deployment permit ​

An admin can accept the risk of an unobserved request ending and remove its exact deployment obligation. This operation does not establish runtime containment, command completion or upload readiness. Automatic safe cancellation remains a separate dependency: it needs a trusted caller-to-scope association and revocation enforced at every effect boundary.

GET /deploy-admission-abandonment?id=<id> returns a read-only quote with the full record digest, original source, current producer and ordinary drain owner. POST /abandon-deploy-admission accepts that quote, a caller-known operationId and a decision containing by, an HTTPS reference, issuedAt, expiresAt and acceptUnobservedRequestRisk: true. Decision times are epoch milliseconds. The decision starts within the quoted drain and ends no later than that original drain. GET /deploy-admission-abandonment?id=<id>&operationId=<operation-id>&expectedRecordSha256=<original-quote-digest> reads only the exact retained receipt, including after the drain expires or the whole POST reply is lost. An optional requestSha256 also compares the full normalized request digest when the caller has it.

CriterionProof
Existing admin authentication protects both routes. A quote reads metadata only. Exact source, producer, record, ordinary drain owner and a fresh explicit risk decision are required; changed, acquired or fenced drains refuse. Selected staging maintenance retains its separate contract.[unit] deploy/cloudflare-resident/operatorCancel.test.ts::operator deploy admission abandonment::*
One Registry transaction archives the full original record, authenticated admin credential digest, exact decision and drain before removing only the selected obligation. The receipt states operator-abandoned, requestEnd: unobserved, nativeOutcome: unverified, runtimeContainment: not-established and uploadAuthorized: false. Native journals, workspaces, neighbors and the drain remain unchanged.[unit] deploy/cloudflare-resident/operatorCancel.test.ts::operator deploy admission abandonment::lets an admin record an explicit unknown-request risk decision; ::rolls back the exact permit when the audit cannot commit
After an acknowledged storage commit, the Registry emits one closed operator-abandonment-commit event with the original admission ID, operation ID, decision time and unobserved request ending. Transaction callback reevaluation cannot duplicate the event. Refusal, uncommitted writes and lost storage acknowledgements emit no success event; exact audit readback can still qualify a lost reply. A throwing log sink does not change the result. The event excludes operator references, credentials, original record fields and private errors; it proves neither response delivery nor native ending.[unit] deploy/cloudflare-resident/operatorCancel.test.ts::operator deploy admission abandonment::emits one closed committed event without copying the operator decision; ::keeps an acknowledged abandonment successful when the log sink throws; ::emits once after an acknowledged transaction despite callback reevaluation; ::emits no success event for refusal, failed commits or lost commit replies
The immutable tombstone prevents delayed normal release, resource annotation and identity reuse from changing the decision. Missing or malformed acknowledgement remains unknown until the exact original audit qualifies it; no mutation is retried. Receipt reads need the original operation ID and quoted record digest and never infer success from absence.[unit] deploy/cloudflare-resident/operatorCancel.test.ts::operator deploy admission abandonment::retains the immutable tombstone through delayed cleanup and identity reuse; ::qualifies a lost commit reply only through its exact audit without replay; ::keeps a lost acknowledgement unknown when exact readback is unavailable; ::keeps quote and exact receipt reads free of mutations
A deployed admin can record one scoped risk decision and independently read its original audit without changing native work or granting upload authority.[gap] Requires release-owner qualification on the deployed build and an explicitly authorized exact target. Local source tests do not establish live acceptance or original request ending.