First coding baseline compatibility
An existing explicit policy can require a bounded startup-smoke command before a coding model starts. This deprecated compatibility path is separate from the default task-directed coding checks, which need no configured command. The smoke result is not automatically a useful baseline for the requested change. Readiness still checks prerequisites; firstAction explicitly selects execution of the configured testCommand. The runner executes it in the attached coding checkout after patch preparation, before opening the harness. It never selects a command from the request or model output and never uses a disposable operation checkout.
- Code:
src/core/firstTest.ts,src/core/firstTestReceipt.ts,src/execution/execResult.ts,src/core/dispatch/runLoop.ts,src/core/dispatcher.ts,src/core/budgets.ts,src/core/runEvents.ts,src/execution/seedPlan.ts,src/execution/factory.ts,src/execution/cloudflareSandbox.ts,src/config/validate.ts,config/config.example.yaml,deploy/cloudflare-sandbox/worker.ts - Tests:
src/core/firstTest.test.ts,src/core/firstTestWorkspace.test.ts,src/core/dispatch/runLoop.test.ts,src/config.test.ts,src/execution/cloudflareSandbox.test.ts,src/execution/factory.test.ts
| Criterion | Proof |
|---|---|
Admission requires explicit firstAction: {kind: baseline_test, policyVersion, timeoutMs} under the trusted ready requirement. The time limit is 1–120 seconds, clipped to the run's remaining setup budget. The operator selects the declared test command; the gate does not infer a command or substitute a whole-suite default. Timeout is enforced; command semantics and test scope remain the operator's responsibility. Config absence and readiness-only declarations stay unchanged; readiness can omit testCommand, but an explicit firstAction requires it. Saved policy survives resume, including after live configuration removes the startup command. | [unit] src/config.test.ts::pilot ready environment config::*; src/core/dispatch/runLoop.test.ts::runLoop first coding test::does not opt an absent first-action policy into a test; src/core/firstTest.test.ts::first coding test::refuses malformed prior receipts and insufficient run budget |
| The runner durably commits operation intent before executing the baseline and completion before the first provider turn or model tool event. Exit 0 with empty output completes; ordinary nonzero exit is a red baseline. This proves command completion, not that assertions ran. | [unit] src/core/firstTest.test.ts::first coding test::durably records intent and empty successful output before allowing model work, ::delivers a normal nonzero test exit as a red baseline with bounded streams; src/core/dispatch/runLoop.test.ts::runLoop first coding test::persists exit… |
| The receipt binds run, requester, thread/unit, repository, ref, full head, checkout path, backend, container, dependency identity, prepared patch fingerprint, command hash and policy version. Matching resident completion survives restart without rerunning the baseline. Initial seeded completion requires a fresh backend restore and binds its actual depsBackupId, carried into the recorded workspace binding. Cached initial seeds and seeded receipt reuse stay held because no authoritative current dependency identity read is available; saved metadata and workspace markers cannot satisfy that check. Foreign, changed or malformed bindings refuse reuse. | [unit] src/core/firstTest.test.ts::first coding test::reuses an exact durable completion after restart without replaying the test, ::refuses a changed…, ::refuses changed…; src/core/firstTest.test.ts::first coding test seeded resume identity::*; src/execution/factory.test.ts::makeExecutor pilot ready environment::admits a seeded sandbox only with a dependency archive, matching source, and a passed tool check |
Missing tools/dependencies, changed actual head/ref/path and unsupported typed execution refuse before model work. The resident uses its attached path; a supported seed uses /workspace/checkout. A cold or unnamed checkout cannot satisfy the gate. | [unit] src/core/firstTest.test.ts::first coding test::refuses preflight exit…, ::refuses an unsupported executor without parsing its rendered output, ::runs only in the supplied seeded checkout; src/core/firstTestWorkspace.test.ts::first coding test bound checkout::* |
| A lost response, malformed result, timeout, killed command or reserved guard exit 200 stays unknown. Resume holds that same operation and original workspace without replay, model work, replacement writer or release. A known resident pre-execution refusal can re-preflight after repair on the same operation, retaining its refusal history. Persistence failure cannot admit model work. The caller sees a named hold and an explicit operator reconciliation action. | [unit] src/core/firstTest.test.ts::first coding test::keeps exit…, ::keeps a lost transport unknown without replaying the command, ::requires durable…; src/core/firstTest.test.ts::first coding test review boundaries::*; src/core/dispatch/runLoop.test.ts::runLoop first coding test::holds unknown completion… |
| Sandbox typed execution preserves raw exit/streams/truncation, rejects missing result fields, and sends an uncertain command once. Legacy string APIs remain presentation paths only. | [unit] src/execution/cloudflareSandbox.test.ts::Cloudflare typed command outcome::* |
The bounded, redacted receipt is saved on the fenced ledger row and projected as a pinned first_test run note before model work. The coding context receives command-completion facts and untrusted output, without making output an authorization source. | [unit] src/core/dispatch/runLoop.test.ts::runLoop first coding test::persists exit… |
| Automatic reconciliation of a possibly executed operation requires a shared typed backend operation lookup. This implementation deliberately holds unknown intent; it cannot recover an unreceived result automatically. | [gap] Backend operation lookup is not available. Never infer completion from text or a model-writable workspace file. |
| The deployed resident and seeded paths preserve a completed receipt through controlled restore/recycle, and expose an exact operation/workspace/head receipt before the first model event. | [gap] Source tests do not prove live restore/recycle behavior. Obtain a controlled receipt under separate live-validation authorization after deployment. |
Command-scope enforcement beyond the explicit operator declaration is unproven; it requires a typed repository-operation policy, not parsing shell text.
Seeded receipt reuse needs a backend-owned workspace identity readback that includes the current dependency archive. Cached seed responses depend on that marker and cannot authorize even a first baseline attempt. Until authoritative readback exists, an operator must stop the held seeded run before authorizing a fresh attempt; the runner never trusts the workspace marker as a completion or current-identity receipt.
The legacy firstTest receipt and its holds are not migrated into task-check receipts. A completed legacy operation retains its original reuse rules; an unknown operation retains its original owner, workspace and no-replay hold. Removing a live configuration field affects new admissions only.