A deploy fence separates execution from workspace retention

Context

The resident keeps a run registration while its workspace needs preservation, even after the run is terminal. The operator's runsInFlight count includes those registrations so cleanup and container replacement keep protecting the files. A deploy preflight that treats that count as execution can wait forever on a terminal workspace. A drain normally exempts the registered owner so a running run can reattach; a deploy cannot safely read activity and then swap isolates while that exception remains open.

Decision

Keep the retention count and the workspaces intact. During a normal deploy, require the drain bearer to close registered reattach in the registry's durable drain record before reading every resident. Ask the run ledger for each registration's exact owner. Count a matching live owner and resident operations as executing, a matching non-provisional terminal owner as retained, and every missing or mismatched answer as unknown. The deploy proceeds only with a live fence that has at least 20 minutes remaining for upload and readiness, zero executing and unknown owners, and safe lifecycle states. The runner asks for a 90-minute drain around its 60-minute wait so that budget remains at the last retry. A refused read clears only its own reattach fence; the runner keeps the drain until it completes or explicitly lifts it.

Consequences

• A Worker without the fence endpoint refuses the new preflight. Installing the endpoint on an older Worker needs a separately reviewed bootstrap; the normal release cannot prove this first swap safe.
• The owner read needs the state Worker. An unavailable answer delays a release, but never authorizes removing a workspace or interrupting an unverified run.
• The fence expires with the drain if a runner dies. A successful deploy holds it through image reconciliation and the ordinary undrain.

Alternatives

• Delete terminal workspaces based on their checked-out commit: that SHA does not prove private files or commits were saved remotely.
• Ignore retained registrations in the old count: that would weaken container and cleanup protection.
• Use the ordinary drain alone: its registered reattach exception leaves an admission race during the isolate swap.

Amended 2026-10-14

Re-evaluation. The fence was accepted because a quiet snapshot must exclude every admitted request while protecting terminal workspaces. A durable permit still represents that request obligation. Caller-side finally cannot settle it after a crash, and a lost admission reply previously left the caller without the Registry-created ID.

New callers create the permit UUID and a hash of normalized original provenance before the first RPC. An uncertain reply permits one exact stored read, never another admission or native submission. Only the original living stack can continue after positively matching its admitted record. Caller death and unknown legacy debt remain held for separately verified recovery.

The Registry atomically writes an exact release receipt and deletes the permit. An uncertain end reply qualifies only through that receipt; absence alone stays unknown. The latest 128 identity-bearing receipts bound storage. Eviction narrows acknowledgement recovery without granting work or retiring active debt. Request cleanup is not native completion.

The original fence predicates and recovery authority remain intact. Legacy callers reaching the upgraded Registry can release only legacy records; keyed releases require the exact hash. Old Registry code cannot enforce this boundary, so rollback with keyed debt requires separate migration/recovery qualification. Producer identity and withdrawn-fence/readback status keep consumer diagnostics honest during rollout.

Amended 2026-10-14: recovery audit quotes

Re-evaluation. The fence protects the request obligation and independent retained workspaces. Manual recovery was accepted only with an operator-verified original request-end receipt and idle owner/lifecycle/process views. Comparing only the permit key after those reads could delete a changed record; a lost commit reply also left no exact original-record receipt to qualify. Preserve the accepted ending requirement and the independent guards while binding recovery to the record the operator selected.

An admin obtains the original start and full flat-record hash through the existing inventory route's exact-ID query. Recovery compares both again in the deletion transaction, retains the complete original record and the receipt URL hash, and records request ending as operator-attested. The hash identifies the stored record; neither the hash nor URL syntax authenticates the ending fact. Native outcome remains unverified and independent obligations remain unchanged. Existing audit keys cannot be overwritten or reused by a new admission. An uncertain mutation reply allows one matching audit read in the original living handler, never another deletion; missing, altered or unreadable evidence stays unknown.

Quoted recovery uses a separate HTTP address and Registry RPC. Older code rejects the new address or method. The upgraded service refuses legacy recovery requests without deleting a row. A return to an older version therefore keeps the record protected. Neither interface can fall back to the old unchecked mutation.

Recovery interface. Replace POST /recover-admission with POST /recover-quoted-admission. The new request also requires expectedStartedAt and expectedRecordSha256 from GET /deploy-admissions?id=<id>. The compact inventory without the query remains unchanged. Operators must verify the external original request-end association against that quoted record before submitting recovery. An old producer needs a separately qualified bootstrap; merge does not establish uptake or cleanup of an existing permit.

Open abandonment dependency. Operator abandonment is a distinct disposition with an unobserved original request end. No trusted containment/retirement producer for an unidentified legacy global caller exists in this contract. The endpoint therefore refuses abandonment and does not convert upload/version changes, process-zero observations or caller booleans into a witness. A future audited abandonment or bounded staging risk contract needs its own qualified effect scope, exact original-row/fence comparison and independent native/private obligation records. This amendment supplies request-ended recovery hardening, not an executable abandonment path or proof that a legacy request ended.

Amended 2026-10-14: explicit staging risk and maintenance

Re-evaluation. A request record is accounting, not a privilege grant. An unknown original caller has no request-end witness. A staging operator can accept a separate, finite fixture risk without inventing an ending. Default profiles disable this action. A trusted staging policy binds the account, Worker, fixture evidence, selected controller, source, profile and configuration. The digests identify the operator's retained evidence; they do not certify deployed bindings or remaining credential scope. The operator must qualify future old authority before bootstrap. Historical remote outcomes can remain unknown.

The Registry retains the original record and an immutable abandonment audit. One common calculation excludes only the matching record from one selected upload window. Other global requests and native, owner, process and private retention guards remain. A changed or absent original record invalidates the exception. The window lasts at most the existing 90-minute drain limit. A separate durable maintenance record blocks new admission through ordinary drain expiry, image reports and restarts. A recorded released generation cannot reinstall itself. Old producers cannot enforce this new hold, so bootstrap and unsupported rollback need the operator's qualified fixture boundary.

The selected normal controller claims the window through an atomic fence transfer. It uses force zero, verifies the exact source/profile/configuration, keeps ordinary image checks, and reports image readiness while maintenance remains held. It does not call undrain. The existing staging wrapper waits within the selected deadline for one explicit Admin release before validation. Expiry, unreadable state or a changed identity fails that wait without a mutation or replay.

A release after upload-window expiry requires a fresh separately recorded operator decision, the unchanged original audit and record, and idle member owner/lifecycle/process views. Other global requests still refuse release. This release preserves the unknown original outcome and cannot renew upload eligibility. An uncertain mutation permits one exact receipt read, never another mutation. Source tests and review qualify the code; merge does not establish bootstrap, live capability scope or staging acceptance.

Amended 2026-10-15: source authority and artifact identity

Re-evaluation. The selected policy binds one exact source commit. That choice remains valid. A real Worker build also has a timestamp, and its timestamped artifact ID identifies a rebuild for override expiration and provenance. Comparing a selected commit to that composite artifact ID prevents the intended maintenance path from starting. It does not provide stronger source authority.

Maintenance initialization and selected reconciliation compare the policy's full source commit to the structured build stamp's commit. They do not parse or shorten an artifact ID. A wrong, dirty, unknown or shortened source still refuses. The timestamped build ID, Worker upload/version metadata and native kernel boot identity remain separate. The maintenance generation is the policy's numbered hold record. A same-source rebuild cannot replace that held record, renew its deadline or change its recorded released state back to held. Existing exact policy, request, hash, fence, count, image and unknown-effect checks remain.

Image verification also needs its own identity. A source commit can produce two Worker artifacts and two native application targets. Registry mode takes its intended image from the structured plan and waits for that exact image in the native read. An unchanged selected image can support a Worker-only redeploy without increasing the application version. Build mode has an opaque Dockerfile intent; its own-application change or no-change announcement is a supported hint that must agree with structured native reads. Missing, unrecognized or foreign announcements cannot approve an old build-mode target. A no-change hint requires identical native reads. If Wrangler prints a container change, the runner also waits within its existing readiness deadline for a later native application version and the printed image, when present. This read prevents registration lag from selecting the old image. The deploy runner then freezes the application name, version and image from its authenticated post-upload read, sends that target to /reconcile, and requires the response and every physical member readback to match. Before reporting readiness, it reads the native application again. A changed or unreadable target fails readiness. A Worker-only upload with identical authenticated application reads before and after upload, no printed container change and complete current reports skips reconciliation. It does not reuse the per-artifact cache as proof; it makes no new image-target claim.

Each member stores the timestamped Worker build and the selected image target with its progress. For example, a completed report for commit A built at time T1 cannot verify the artifact built from A at T2. A completed report for application version 1 cannot verify version 2, even on the same Worker artifact. Either change requires a fresh guarded image decision. A different target cannot overwrite an earlier pending report; the old report must finish before a new target starts. Same-target retries retain replacement hydration and deferred-cycle behavior. No image decision deletes run, native-operation or private workspace history.

A commit-only cache is rejected because it cannot distinguish those artifacts or targets. A timestamp-only cache is rejected because the native target can change without a Worker rebuild. An authenticated target identifies the desired image; it does not prove physical uptake. The member's guarded cycle and subsequent report supply that separate evidence. The upgraded route requires the matching deploy caller; an older or incomplete caller refuses without a fallback. Rollout must qualify that source pair before deployment.

The target fields use exact equality: application name, nonnegative integer version and full image reference. Progress stores { build, target, cycleIssued, cycleStoppedAt?, containerBeforeCycle? }. A legacy completed record without a target starts a fresh decision. An earlier pending record keeps its original fields; a different or missing stored target returns a member error and fails that deployment without a new cycle. The existing report callback can finish the old pending report. Same-target retries use the existing ten-minute readiness bound in DRAIN.cycleBoundMinutes; failure retains the pending marker. The pending marker has no automatic expiry. A different-target request fails immediately instead of waiting indefinitely. An old report that cannot finish needs separately qualified operator recovery, not deletion or target reassignment by this repair.

The final native read is a point-in-time readiness check. It cannot lock Cloudflare against a later external change. The sole deployment controller and the staging operator's independent release qualification remain required. This change does not claim an atomic native rollout or establish live image acceptance from source tests.

Amended 2026-10-15: a separate held CI bootstrap

Re-evaluation. The staging decision was accepted because an operator can qualify one finite bootstrap while retaining the original request, native and private obligations. The first normal CI controller then consumes one separate risk window. This separation still applies when the bootstrap uses CI. A normal staging job must keep force zero and its full-stack and Review E2E acceptance contract.

Use one manual-only workflow with the existing staging Environment, private configuration reader and deploy-staging lock. Its distinct staging / bootstrap held status and artifact cannot satisfy normal staging acceptance. The job requires the original repository writer, an open same-repository PR, its exact current source commit and a matching controller revision/ref. It rejects run_attempt other than one at both the workflow and entry point. Before dispatch, the operator records the old account, Worker/version, storage bindings, fixture reset and the limits on any old caller's remaining capabilities. Idle counts do not prove that an old request ended. The job checks the selected old source and Worker version; the evidence digests identify the operator's retained qualification, not a machine certificate. A source merge or queued job does not select or renew that operation. The controller revision must equal the selected source, and its workflow reference must use the exact selected PR branch.

The entry point requires immutable profile and nested configuration references from the existing configuration repository. It freezes the validated bytes outside the checkout. It compares the full source, canonical profile and configuration hashes, hold, generation, selected controller label and fixed deadline. Queue, validation and installation time consume that deadline. The existing 90-minute maximum and 20-minute upload/readiness reserve apply. Controller labels identify selected policy data; they are not credentials or automatic GitHub run attestations. The receipt also records the actual actor, run, attempt and workflow source. The job requires its positive original GitHub run metadata. Complete manual workflow history must contain only that run for the policy's attempt ID. A prior, missing or unreadable matching run refuses. A fresh dispatch cannot reuse that retired attempt. This check relies on preserved GitHub run history; deleting that history is outside the supported recovery contract.

Call the existing direct deployment runner once with Resident only, force and retained maintenance. A read-only precheck requires the selected old source and a complete fixture inventory without an existing maintenance record. The inventory count must equal its rows; each physical member occurs once, and its repository set must equal the policy resources. Pending member activation refuses. These observations do not settle the original request or native effects. The preflight also compares the qualified old Worker version in the fence response. Unreadable, malformed or changed producer evidence refuses even under force. Only staging deploy, read and drain credentials reach this operation. No Admin or production credential is present.

A successful receipt requires the exact new source and Worker native capability, native application/image and member evidence, the exact held policy and a null risk window. It reports only image-ready, maintenance held. It performs no risk decision, normal claim, seal, release, reset, undrain or E2E action. The later first normal retained CI run remains the sole risk-window consumer.

For example, the upload can land while its response is lost. The job retains its original intent and reports an unverified result. It preserves closure and debt, uploads its own bounded receipt and stops. A rerun cannot upload again. An operator can adopt an independently proved original result through a read-only reconciliation. A new physical operation requires a new qualified selection after the old outcome is reconciled. The entry point cannot create that selection.

An optional bootstrap mode in the normal workflow is rejected because its authorization, prior-run history and acceptance selectors would all need a new discriminator. Separate job names alone do not keep its same-head manual requests out of that history. A separate workflow leaves those selectors intact. The production workflow is rejected because it uses a different account and credential contract. A separate manual job adds no credential provider or permission model. A missing receipt or failed artifact upload leaves the bootstrap job failed or unverified. It never starts another operation. Its source tests do not establish live capability scope, old request ending or staging acceptance.

The bootstrap requires fresh member reconciliation for the selected native application, version and image even when the image is unchanged. Its final registry read must contain every original member, no added member and no pending activation. Ordinary Worker-only updates keep their existing unchanged-image path. The old fence response must include complete idle fixture data, valid fence fields and at most one independently qualified old global admission. Matching the Worker UUID alone cannot admit malformed or busy member data.

The known-old fence response contains the Cloudflare Worker version as draining.swapVersion. It does not contain the newer deployDiagnostic object or the GET registry count. The bootstrap compares that authenticated fence UUID to the qualified previous Worker UUID. If newer producer diagnostics exist, they must agree. Source commit, timestamped build label, Worker UUID and native application/version/image stay separate.

The runner rechecks the selected 20-minute readiness reserve after native preparation and before the deploy command. The final preflight checks it again after the authenticated fence read. Force cannot bypass either check. The plain Node entry uses the deployment clock; the reserve value comes from the shared DRAIN budget.