Lane honesty: four ways GET /lane and the trail reported the wrong thing #115
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "agent/lane-honesty"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Stacked on
agent/lane-stages(PR #99). Targetingmainbecause CI only fires on PRs into main; merge after #99, or retarget once #99 lands. The diff below includes #99 commits.Four defects that share one property: every decision the lane made was correct, and every surface an author or operator reads was wrong. They share files, so this is one change.
1. A member enqueued during a build was invisible for the whole build
enqueuereturns the instant the event is in the mpsc channel, and nothing reads that channel until the worker returns frompump— for a real lane, ~45 minutes. Observed:POST /laneanswered{"detail":"queued \pr-6956`","ok":true}andqueue_depth` stayed 0 across 36s of polling. An author reads that as "the lane never got it" and re-submits.The host now keeps its own set of accepted-but-not-yet-stepped ids and reports the union, de-duplicated against the lane's own queue/in_flight/ejections. The merge is at read time, not publish time — the worker may not republish for the length of a build, so a
withdrawin that window has to take effect on the next read.queuednames them.2.
/lanereportedphase=idlewhile the lander ranNot fixable the way the build case was. By the time
LandAndPublishexecutes, the lane is genuinelyIdle— green verdict in,in_flightemptied — while the lander moves the trunk for up to 7200s. The phase is honest and still misleading.pump_observednow brackets every blocking action with aLaneActivity, and the snapshot carriesactivity+landingbesidephase. It reports the driver's activity; it does not falsify the phase.3.
materializecould fail with a bareos error 2naming nothingObserved:
lane-build generation=13 outcome=infra reason=candidate tree could not be materialized: No such file or directory (os error 2). The candidate root, the scratch parent, the repo, and agitthat cannot be spawned all produce exactly those bytes, with different fixes. Both fallible filesystem steps now name the operation and the path (MaterializeError::infra_at, preservingio::ErrorKind), andgit()annotates spawn failures with the cwd it tried.4. The lane spun a generation every ~30s against an unreachable preview daemon
51 wasted generations in one night. The backoff was not missing — it could not fire.
LaneState::nowmoves only onLaneEvent::Tick, and the host's ticks sit unread in the channel for the whole of a blocking action because the worker is insideexecute. So every deadline the outcome computes is anchored at the moment the attempt started. Pointing the preview slot takes ~24-35s against a 30-tick backoff, soinfra_retry_after = started + 30was already in the past when written. On the land path a 7200s budget against a 30-tick backoff can never delay anything.The driver now re-syncs the lane's clock from the host's own tick stream immediately after each blocking action, before the outcome is applied. Deliberately
advance_clock, not aTick: a Tick also runsmaybe_start_build, and after a failed land the phase is alreadyIdlewith the members re-enqueued — so a Tick there would start the next build beforeLandFailedinstalls the backoff. The retry is paced, never removed.Tests
lanehost::a_member_enqueued_during_a_build_is_visible_immediately— asserts on the FIRST read, no polling loop, so a merely-shorter window cannot passlanehost::withdrawing_a_member_still_in_the_channel_removes_it_from_the_snapshotlane_policy::the_lane_reports_who_is_queued_not_just_how_manylanehost::the_snapshot_says_landing_while_the_lander_runs— also assertsphasestaysidle, and that it returns tosettledlanehost::the_snapshot_says_building_while_the_legs_run(regression guard)lanetree::a_scratch_dir_that_cannot_be_created_names_the_path_and_the_step(real ENOTDIR)lanetree::a_git_that_cannot_be_spawned_names_the_directory(real ENOENT)lanedrv::materialize_context_tests— 3 tests pinned against the exact generation-13 stringlane_policy::an_infra_backoff_is_measured_from_the_failure_not_the_attempt— attempt takes LONGER than the backoff; asserts the boundary ticklane_policy::advancing_the_clock_does_not_itself_start_a_build— proven by stepping the machine both ways, never by comparing two constantsValidated against a Rust-faithful Python port of
LaneState::step,pump_observedandLaneSnapshotbefore pushing (cargo is hook-blocked; CI is the only build). The port measured OLD=9 vs NEW=5 generations per 300 wall-seconds against a failing preview, and caught two things review did not: a mis-measured land assertion, and the publish-time-vs-read-time merge bug in defect 1.All 9
lane-attribution-mutation-test.shanchors verified still present and uniquely-matching after thelane.rsedits.🤖 Generated with Claude Code
The build lane runs a real release build — 39 minutes measured, 66% of it in the server compile alone. Today a type error costs that entire cycle, because the check engine has no way to say "prove the cheap thing first". Three additions, all opt-in: stage: N checks group by stage; stages run in ascending order and a stage starts only if every earlier stage was green. Default 0 puts every existing check in one stage — an unstaged manifest behaves exactly as before. fail_fast (profile) stop on the first REQUIRED red and kill what is still in flight. Off by default: for a governance profile you want the full picture, and reporting one red while hiding nine makes a developer fix them one round-trip at a time. It earns its keep only where the remaining work is expensive and certain to be discarded. target_key: K which CARGO_TARGET_DIR a leg compiles into. This is the one that makes parallelism real: under a warm target EVERY command check was a single mutual-exclusion class, so declaring two compile legs in parallel bought nothing — they serialized on cargo's own .cargo-lock. Distinct keys give distinct dirs. The deploy this models has always used separate target dirs for its native and wasm builds; this closes the gap. Honesty properties the tests pin, because they are where this could quietly go wrong: * A skipped check is REPORTED, not dropped. A caller seeing fewer results than checks cannot distinguish a truncated run from a shorter pipeline, and a lane deciding whom to eject would be reasoning from a silently incomplete picture. * A cancelled check reports SKIPPED, never a red or a timeout of its own. We killed it on purpose; blaming it would send someone debugging a failure that never happened. * Skips are never cached — a skip describes the RUN, not the CHECK, so caching one would keep a check skipped after the failure was fixed. * A non-required red never trips fail-fast. Cancelling a build over a failing advisory would silently promote it to a gate. * target_key is sanitised, not trusted: it becomes a path component, and a manifest must not be able to escape the parent with `../`. Cancellation reuses the existing kill_process_tree path rather than inventing a second one — the grandchild-reaping problem is identical to a timeout's. Note the rollout order this obliges: reject_unknown() makes an unrecognised key a HARD error, so the daemon must ship these keys and the fleet must roll BEFORE any repo declares them. A repo that declares early reds the manifest for every agent pushing there, not just its author. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>Two gaps that made the merged lane unusable in practice. 1. NO CandidateTree implementation existed outside tests. `grep 'impl CandidateTree' crates/` returned exactly one hit, in lane_reusability.rs, and it returns a hardcoded path without touching the filesystem. So the lane could be driven by a test vector and by nothing else. `lanetree::GitCandidateTree` is the real one: `git worktree add --detach <scratch> <base>`, then `git merge --no-ff` each member in submission order. It MERGES rather than overlaying files, because members carry commit shas and "base + every member" is the entire claim the lane makes — a tree assembled any other way would let the lane report a verdict about a candidate that was never assembled. A conflict returns Err, i.e. infrastructure, never a red. A member that cannot merge has not failed a build; it has failed to produce a candidate. Blaming it would be arbitrary anyway — the other side of a conflict is equally "responsible" — and wrong attribution is how a team learns to route around its own gate. The failing member IS named in the error, since "B conflicts when applied after A" is actionable and "the candidate conflicted" is not. 2. The driver hardcoded `Green { artifact: None }`. So PointerLander always took its "nothing to publish" branch and `.cargoless/latest-green` NEVER advanced. Every lander test passed — the lander was correct — and the lane still published nothing. That looks exactly like a working lane until someone reads the pointer. The seam was untested because RecordingLander ignored its artifact argument: every test asserted WHO landed, none asserted WHAT shipped. It now records the payload, and two tests pin the seam in both directions — an artifact reaches the lander, and a check-only lane hands it None rather than an empty string. `Some("")` would advance the pointer to nothing: a silent rollback wearing a success's clothes. ProfileLegRunner grew `artifact_path`, read ONLY on green. A red build may have left a stale artifact in a warm target dir, and publishing that is precisely what "never publish red" exists to prevent. LegOutcome also carries per-leg reports now. ProfileLegRunner was discarding ProjectCheckReport.results — the per-check id/tree/duration — which is exactly the visibility a staged lane needs to say "stage 1 rejected this in 4 minutes, stages 2-3 never ran" instead of "the build failed". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>`POST /lane/enqueue` has been answering "build lane not enabled on this daemon" since the lane merged, because nothing implemented the service methods — the routes, wire types, policy, driver and docs all shipped and the host did not. This adds it. `ServeVerdictState::with_lane(repo, state_dir, base_ref, profile, artifact_path)` builds a GitCandidateTree + ProfileLegRunner + PointerLander and spawns a LaneHost. Opt-in and off by default: a lane merges and publishes, so a daemon must never acquire one as a side effect of a default. The caller names the profile because the legs are the PROJECT'S — that is what keeps the lane reusable rather than a tf-multiverse feature. The three service methods keep the contract the trait defaults established: * A laneless daemon still ERRORS on enqueue rather than accepting. Answering "queued" when no lane exists leaves the caller waiting forever for a build that will never run. * `lane_snapshot` still returns None on a laneless daemon, so `GET /lane` 404s instead of reporting an empty lane. "No lane here" and "a lane with nothing in it" are different answers. * `id` and `head` are required and NOT defaulted — a member with no identity cannot be attributed, ejected or reported on, and an anonymous candidate must never enter a queue that can move the trunk. `changed_files` stays optional: a caller that cannot compute a diff still gets queued and accepts unattributable reds. The snapshot reports each ejection's `kind` (attributed vs unattributed) alongside its files, because the two are cleared by different things and an author needs to know which one they have. One real bound fix: `mpsc::Sender` is `Send` but not `Sync`, and `VerdictService: Send + Sync` shares the service across connection threads. The sender is now behind a Mutex held only for the send — a pointer hand-off, never a build — so it cannot become the contention point the snapshot design exists to avoid. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>The stage/fail_fast/target_key tests I added are worth exactly as much as their ability to fail. This repo has shipped machinery that exits 0 while doing nothing more than once, and a suite that passes either way converts "unverified" into "verified" — worse than no suite. So the existing harness grows five mutations against project_checks.rs, each mapping onto a property the feature claims: 5. stages do not gate — a red stage no longer halts the next, so the expensive release build runs for a candidate the cheap check already rejected. Without this the whole feature is decoration. 6. fail_fast never trips — in-flight work runs to completion on a candidate that cannot land. 7. fail_fast over-trips — cancels on a NON-required red, silently promoting every advisory into a gate. 8. target_key ignored — every leg back on one CARGO_TARGET_DIR, so legs declared parallel serialize on cargo's .cargo-lock and the parallelism is imaginary. 9. skipped reports green — a check that never ran reports GREEN, so a gate reads "unknown" as "passed". Fails OPEN, which is the direction that actually hurts. These mutate a different file and run a different suite than the four lane.rs ones, so they get their own backup and runner rather than being forced through the existing `mutate`. The EXIT trap now restores both files on every path — a harness that leaves a mutated source behind would be a far nastier bug than the one it hunts. All five anchors verified against the current source before committing; a mutation that fails to APPLY is still reported as a survivor, so a moved pattern shows up as a loud failure rather than a silent skip. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>The operator asked directly: "have you tested it with queueing a few different ones as well?" The honest answer was no — `lane_policy.rs` covers multi-member cases at the PURE POLICY level, but only one driver-level test involved more than one member, and none covered two independent reds or re-admission. Four tests, each an outcome someone would actually hit: * three members, TWO independent reds owned by different members — both owners ejected, innocent third ships. Catches the naive implementation that ejects the first owner it finds, and the timid one that gives up and holds everyone. * a member arriving mid-build QUEUES rather than preempting, and gets its own candidate. Driver-level on purpose: the policy test proves no second build is *decided*, this proves none is *started*, which is the part that would burn a runner. On a real lane a cancelled CI run is permanently not-green, so preempting strands work that was about to pass. * an ejected member returns when its new head touches the file that failed, and actually ships. Without this an ejection is a dead end rather than a hold. * and stays out when its push touches nothing implicated — asserted on the CANDIDATE COUNT, not just the ejection, because the real cost of getting this wrong is a build. At tens of minutes each, a README edit buying a slot turns the lane into a treadmill. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>The `build` job's log carried a warning I would otherwise have shipped: warning: method `with_lane` is never used --> crates/cargoless/src/serveapi.rs:1490:12 = note: `#[warn(dead_code)]` on by default Which is exactly right, and worse than it looks: clippy runs with `-D warnings` here, so this was a red waiting to happen — and more to the point, a lane nothing can turn on is the "machinery that exits 0 while doing nothing" pattern this repo keeps re-learning. I wired the daemon to HOST a lane and then gave no way to ask for one. Three env vars at the serve-loop's construction site: CARGOLESS_LANE_PROFILE the cargoless.checks.yaml profile to run CARGOLESS_LANE_BASE base ref for candidates (default: main) CARGOLESS_LANE_ARTIFACT artifact to publish on green; unset = a check-only lane that proves the merged tree compiles and leaves the pointer alone rather than advancing it to nothing Opt-in and off unless asked. A lane merges and publishes, so a daemon must never acquire one as a side effect of starting up — the operator names the profile, because the legs are the PROJECT'S build and that is what keeps the lane reusable rather than a tf-multiverse feature. It announces itself on boot. A lane that can move the trunk must be visible in the log rather than inferred from behaviour, same reasoning as the resolved-caps lines just below it. Not written as a let-chain: this workspace is Edition 2024 / MSRV 1.85 and `if let ... && cond` does not compile here. Incidental finding while chasing this: `build` and `fmt` had 0 successes in 10 attempts over 6h while test got 7 and clippy 4 — but their logs both end `🏁 Job succeeded` with `Finished dev profile in 44.21s`. They PASSED; the runner never wrote the result back. The DB status is the unreliable part, not the build. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>An audit of what has actually run against real I/O found three defects, one of which meant the feature did not work at all. 1. NOTHING CALLED LaneHost::tick. `LaneState`'s clock advances ONLY on `LaneEvent::Tick` (lane.rs:457). Nothing anywhere called `tick` — not production, not one test. With `capture_window_ticks: 60` as the default, `now` stays 0 forever, the window never elapses, and `maybe_start_build` returns early every time. A freshly enabled lane would accept submissions, report "queued", and build NOTHING until a 10th member arrived and tripped the max_members path. That is the worst failure shape available: it looks like a broken transport or bad auth, not a missing heartbeat. Every test that starts a build sets `capture_window_ticks: 0` or drives Tick by hand — the DEFAULT config was used by production and by no test. Ejection TTLs lapse on the same signal, so `eject_ttl_ticks: 3600` — the backstop guaranteeing nothing is stuck forever — never fired either. Now driven from the serve loop beside the existing activity tick. 2. AN UNKNOWN LANE PROFILE WAS A QUEUE-WIPING FOOTGUN. `profile_for` falls back to `include: ["*"]` with a 12-SECOND budget for a name it does not recognise, and `"*"` matches every check regardless of tier. So a typo in CARGOLESS_LANE_PROFILE selects the WHOLE manifest — including any 25-80 minute release build — times nearly all of it out, and hands the lane ~130 error diagnostics pinned at `cargoless.checks.yaml:1:1`. Those are attributable to nobody, so the lane ejects the entire queue as `Unattributed` on its first build: the gate appearing to decide everyone is guilty, on evidence that is a scheduler artifact. With defect 1 unfixed the TTL would never lapse it either. The daemon now REFUSES to start on an unknown profile and names the declared ones. `profile_for` is left alone — it is shared with `checks run --profile x`, where the fallback is merely slow. 3. THE LANDER DID NOT FOLLOW THE ARTIFACT SETTING. `with_lane` hardcoded `PointerLander` even for a check-only lane. Harmless today (it takes the no-artifact branch) but one refactor from advancing a pointer to nothing. Adds `ReportOnlyLander` and selects it when no artifact is declared, so the safe configuration is also the default one and the two cannot disagree. Plus the test file this whole audit was shaped around: tests/lane_real_io.rs. `ProfileLegRunner` — the ONLY production LegRunner, the type that turns a candidate tree into a verdict — had zero coverage of any kind, and no test at any level composed two real components. The tested sets were {real git tree, no legs}, {fake tree, fake legs, real host} and {real subprocesses, no lane}. The shipped set is all four real. Six tests, no fakes: real git repo, real worktree, real merges, real bash subprocesses, real diagnostics parsed back. They pin that a red carries REAL file paths (without which attribution silently degrades to "blame nobody, hold everyone"), that a missing artifact on green is infra rather than a false accusation, that the full pipeline advances the pointer, and that a report-only lane really does not. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>CI caught a real compile error in the lane boot-validation I added: error[E0277]: `ManifestError` doesn't implement `std::fmt::Display` --> crates/cargoless/src/servedrv.rs:477:63 `{e}` does not compile for it. `{:?}` would compile and print a struct dump, which is the wrong fix — the type carries `path`, `line` and `message` as public fields, so it can render as `path:line: message`, the shape every other tool in this repo uses and the one an operator can act on without decoding Rust syntax. Note what this says about the earlier verdict: `test: SUCCESS` on this tree was the LIB tests passing. The binary did not compile. A green `test` job is not a green build, and the two are separate CI jobs for exactly this reason. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>`a_traversing_target_key_cannot_escape_the_parent_dir` asserted `!seen.contains("..")`. But `../../etc` sanitises to the leaf `.._.._etc` — which still contains ".." as a SUBSTRING while being an ordinary directory name. The dots are no longer path components, which is exactly what the sanitiser is for. So the test reddened on working code. That is the worse kind of wrong test: the natural way to make it pass is to weaken the sanitiser until the output stops containing dots, which would trade a correct implementation for a green check. Now asserts the property that actually matters — containment — twice: the resolved path starts with the run root, and no path COMPONENT is `ParentDir`. Both hold regardless of how the leaf is spelled. Found by the mutation harness refusing to run: "FAIL: the engine suite is red BEFORE any mutation — fix that first". That guard earned its keep; without it the harness would have reported mutation results from a suite that was already broken. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>The mutation harness reported: SURVIVED — fail_fast trips on a non-required red the suite passes on code that cancels a build over a FAILING ADVISORY, promoting non-required checks into gates It was right. `a_non_required_red_never_trips_fail_fast` had the advisory exit INSTANTLY while the required check slept 1s — so a fail_fast that wrongly counted non-required reds would fire, but the required check had usually already finished. The assertion passed either way and the test proved nothing about the property it is named for. Sleep is now 6s, so a wrongful cancel lands mid-flight and leaves `real.out` unwritten. The sibling `without_fail_fast_a_red_does_not_cancel_anything` had the identical weakness and gets the same margin — it guards fail_fast staying OPT-IN, which fails the same way for the same reason. Worth stating plainly: this is the mutation harness earning its keep. Two timing-shaped holes in tests I wrote yesterday, both invisible to a passing run, both found by deliberately breaking the code and watching the suite not notice. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>A failing test turned out to be reporting a real product defect, not a bad assertion. `LaneHost`'s worker published its snapshot only AFTER `pump` returned. But `pump` runs the whole build inside itself — tens of minutes for a real lane — and the transition that flips the phase to `Building` happens on the line just before that blocking call. So for the entire duration of every build, `GET /lane` answered: {"phase": "idle", "queue_depth": 0, "in_flight": []} That is precisely the window the endpoint exists to explain. An author whose change stopped moving would look, see "idle", and reasonably conclude the lane never received their submission — or was broken. The one question the product surface is for, answered wrongly, for the whole time it matters. `LaneDriver::pump_observed` takes a callback fired after every state transition and before the actions it produced are executed. `pump` stays as a wrapper so existing callers are untouched. The host publishes from that callback, and once more after, so a terminal state is never lost. The test also had a genuine race — it asserted immediately after the legs signalled, which is a few instructions before the worker publishes. It now polls briefly, and asserts the two properties separately: that the read does not BLOCK (the reason snapshots exist at all) and that it REFLECTS the running build (the reason this commit exists). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>The fixture-precondition assertion added with the cargo-json advisory did its job on the first CI run: it reported the advisory as Green when the test needs it genuinely Red, instead of passing vacuously. Cause is in the fixture, not the engine. `unquote_yaml` (yamlscan.rs:517) replaces `\n` BEFORE `\\`, so the `\\n` written in the manifest string became a real newline rather than a literal backslash-n. That split `printf '%s\n' '{...}'` across two lines, printf got a broken format string, and no parseable cargo JSON ever reached stdout — so there was no error diagnostic, so the tree was Green. `echo` instead: it supplies the trailing newline itself, so the escape was never needed. Verified by porting unquote_yaml's exact replace ORDER and running the resulting argv: no real newline, exit 1, one line that parses as reason=compiler-message level=error with the span intact. This is the second time this test's fixture, not its assertions, was the problem — which is precisely why the precondition assertion is there rather than trusting that a red fixture stays red. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>Running the legs in the daemon is a privilege escalation, and on tf-multiverse it is a demonstrated one. Verified 2026-07-31 from inside the `serve` container of cargoless-serve-witness: $ id -u # 1000 $ [ -r /workspace/tf-multiverse/.git/config ] # READABLE $ git push --dry-run origin HEAD:refs/heads/probe * [new branch] # PUSH RIGHTS `env | grep FORGEJO` in that container returns nothing, which makes it look unprivileged — the token is injected into repo-bootstrap and repo-sync. But those sidecars write it into remote.origin.url on the SHARED volume, and serve reads it as the same uid. Check the file, not the environment. Since `cargo` executes build.rs and proc-macros from the tree it compiles, and that tree is the candidate merge of unreviewed code, "we only compile it, we don't run it" is false and the blast radius is push access to the trunk. tf-multiverse already solved this: merge-train-candidate.yml compiles an unmerged train commit holding no deploy key, no kubeconfig, no Plane credential and no merge token, with the contract pinned by merge-train-candidate-unprivileged-test.sh. DispatchLegRunner is the cargoless half of that shape — publish the candidate on a ref, hand it to an external builder, parse the cargo JSON it reports back. Cargoless stays forge-agnostic: it pushes a ref and runs a command. What that command does — dispatch a workflow, submit a Job, ssh a builder — is the operator's business, exactly as LaneLander is for landing. A red still carries real file paths, so attribution survives the move out of process. Reuses project_checks::kill_process_tree (now pub(crate)) rather than hand-rolling a kill: it sweeps setpgid escapees as well as the process group, which is what stops a timed-out build leaking grandchildren to init. That needs the child to be a session leader, so the spawn does setsid too — process_group alone would leave the sweep inert. Tests assert the security property rather than describing it: a candidate carrying a build.rs-shaped payload must leave no mark, and a mutation that runs the dispatcher inside the tree fails that assertion. Logic validated end to end by a Python port of the same git/subprocess sequence (8/8, mutant caught). Not yet wired to a daemon env knob — that lands with the tf-mv adapter, so this commit adds capability without changing any running behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>testjob 222404 was CANCELLED (status=3), not redA member that cannot be merged onto the base used to report `LaneBuildOutcome::Infra`, because every `materialize()` failure did — the trait returned `io::Result`, so a merge conflict and an ENOSPC were the same value. `Infra` ejects nobody by design ("nothing was compiled, so nothing was judged"), which is right for a runner dying and wrong here: the conflicting member never left the queue, and every subsequent candidate re-included it. Observed against tf-multiverse on 2026-08-02. Generations 2, 3, 4 and 5 each died identically: lane-build generation=N outcome=infra reason=candidate tree could not be materialized: member `pr-10462` (77377ef0) could not be merged onto the candidate: git [...] exited 1 `git merge-tree` confirmed a genuine conflict — five markers against dev — and two other real PRs sat behind it for the whole time. The lane livelocked. This is the mirror image of the misattribution `EjectReason` was built to prevent. Instead of blaming an innocent member we exonerated a guilty one, and paid with a queue that could never drain. ## The change `CandidateTree::materialize` now returns `Result<PathBuf, MaterializeError>`: * `Conflict { id, files, reason }` — git named the member before we inferred anything. Reported as the new `LaneBuildOutcome::Conflict`, which ejects that member alone and requeues everyone else at the front WITHOUT the infra backoff, because the next candidate is genuinely different (it no longer contains the conflicting member) rather than a retry of the same one. * `Infra(io::Error)` — fetch failed, worktree could not be created, disk full. Unchanged behaviour: nobody's fault, everyone stays queued. The ejection is `Attributed` with the conflicting paths, so `readmission_decision`'s existing files-based gate applies unchanged: a new head touching one of those paths readmits, anything else stays out. When git cannot report the paths it falls back to `Unattributed`, which still ejects and readmits on any new head — the honest answer when we cannot say which files to watch, and still not a livelock, because the member is out until its head moves. `GitCandidateTree` reads the unmerged paths BEFORE `merge --abort`; the abort clears the index and with it the only record of which files collided. ## Why a conflict is the member's fault The other side of a conflict is equally "responsible" for the *content* — that was the original reasoning, and it is why this was classified as infrastructure. But it is not responsible for the *decision*: the base is what everyone else already agreed on, so the member that cannot apply to it is the one that has to move. That is exactly the judgement a merge queue exists to make. ## Test `an_unmergeable_member_is_ejected_and_the_queue_keeps_moving` builds a real conflict in a real git repo and asserts three things. The third is the regression guard the old code would have failed: after the ejection, a fresh tick must NOT put the conflicting member back in flight. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>The `lane policy mutation proof` job caught a non-exhaustive match: the per-build trail writer in `lanedrv.rs` matches `&LaneBuildOutcome` and did not handle the new `Conflict` variant. Written as a genuine arm rather than `_ => {}`. Today a conflict is detected while materialising, which returns early and writes its own `outcome=conflict` line, so this arm is unreachable — but if a future path ever produces a conflict *after* materialisation, a wildcard would silently swallow the verdict and the trail would lose it. A verdict outliving the tree it was computed from is the reason this trail exists. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>The driver turned a lander `Err` straight into `LaneEvent::Enqueue` for every member. `pump` feeds follow-up events back into the lane, and `maybe_start_build` then starts the next candidate immediately — so a lander that keeps failing made the lane rebuild the same tree as fast as it could loop, each turn a real multi-minute build holding the preview slot. The only bound was `MAX_STEPS = 64` per pump, which is a runaway backstop, not a policy. The realistic trigger is the base moving and the forge's compare-and-swap rejecting the push, which on a busy trunk persists for many minutes. Arming `CARGOLESS_LANE_LAND` is what puts a real forge behind that path, so this is worth having before, not after. The build path has had pacing since the first deployment, and its comment says exactly why: "The backoff is what stops this being a hot loop. Without it the retry runs the instant `maybe_start_build` is reached, hits the same broken condition, and spins as fast as the failure returns." A failed land is infrastructure by the driver's own reasoning — the build was GREEN, so nobody's code is at fault — so it should inherit the same pacing rather than get a second mechanism. New `LaneEvent::LandFailed { reason, members }` sets `infra_retry_after` and counts toward `infra_failures`. It is emitted BEFORE the re-enqueues: every `Enqueue` ends in `maybe_start_build`, so with a zero capture window the very first one would otherwise start the build before any backoff existed. It carries the member ids for the same reason — they are not in the queue yet when it runs. Members still requeue exactly as before. Losing green work to a push race stays the worst available outcome; this only changes how fast the retry comes. Test: a lander that always fails, with `capture_window_ticks: 0` — the worst case, where nothing else would hold the member back. Asserts the member survives, that it is WAITING rather than building (this was `Building` in a loop before), and that the failure is reported with its retry budget instead of being silent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>`expire_ejections` removed the ejection and pushed `LaneAction::Readmit`, but never called `admit`. So a member whose TTL lapsed left `ejected`, never reached `queue`, and was simply gone — while the log said "re-admitted". Observed in production today. Three members ejected `infrastructure` by a preview outage hit their TTL and vanished: `queue_depth: 0`, nothing building, and no trace in `GET /lane` that anything had been lost. The lane looked idle and healthy with three real PRs silently dropped. The TTL is the BACKSTOP for a member the attribution stranded — the last thing standing between a wrong ejection and a PR that never moves again. A backstop that discards what it was protecting is worse than none, because the reported outcome is identical either way: `Readmit` is emitted on both paths. `on_force_readmit` already did this correctly, reconstructing the member from `Ejection`'s retained `head` and `changed_files`. `expire_ejections` now does the same. Those fields exist for exactly this. Why no test caught it: `ttl_expiry_readmits_as_a_backstop` asserted only that `ejection("A")` was gone and that a `Readmit` action was emitted — both true of the broken code. It now also asserts the member is back in the lane, with its head and changed set intact. Without `head` the candidate cannot be built; without `changed_files` every later red it rides is unattributable and holds the whole queue instead of ejecting one member. Added a mutation for the class, since this is the second lane bug that shipped because a test checked the announcement rather than the effect. Anchored on a string verified to occur exactly once — including after rustfmt, which is what would silently have broken it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>Third attempt, and the first that changes behaviour. Both earlier ones reported "SURVIVED — infra retry has no backoff" and both blamed the test suite, which was innocent each time: 1. 7714b9f..c86d823 — a bare replace on the two-line assignment DRIFTED. `LandFailed` sets `infra_retry_after` with byte-identical text and appears earlier in lane.rs, so `replace(..., 1)` mutated that one and left this rung untouched. 2.221c054— anchoring on `infra_failures.saturating_add(1)` injected `infra_retry_after = None` at the TOP of the infra arm. The real assignment happens later in the same arm and simply overwrote it. The mutation applied cleanly and changed nothing. Now anchored on the comment directly above the assignment ("failure returns."), which is unique to this rung, and it REPLACES the assignment rather than adding one before it. Verified four things before pushing, rather than the two I checked last time: the anchor occurs exactly once; the mutation applies; **the field is not re-assigned later in the same arm** (the check that would have caught attempt 2); and the mutated offset falls after the `LandFailed` arm (the check that would have caught attempt 1). A mutation that edits the wrong line, or the right line at a point where the edit is overwritten, is worse than no mutation: it accuses the suite of a blindness it does not have, and sends you writing a redundant test for a rung that is already covered. The proof's own contract is "prove the tests can fail" — a mutation has to be held to the same standard. The new "TTL expiry drops the member" mutation added in1d2f206is caught, so that rung is genuinely covered. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>A candidate build takes minutes. In that window a member can LAND — merged by hand, or carried by an earlier candidate. Its head is then already an ancestor of the base, and `git merge --no-ff` does not fail: it writes an EMPTY commit and returns 0. Verified against real git: merge rc=0 ; commits added: 0 So the candidate builds, goes GREEN, and the lander is handed a roster naming a PR that is already closed. While landing was report-only that was untidy. With auto-merge armed it is a real merge API call against a merged PR — precisely what makes an auto-merger untrustworthy. `GitCandidateTree::materialize` now checks `git merge-base --is-ancestor <head> HEAD` before merging each member and returns a new `MaterializeError::Stale`. HEAD rather than the base ref, so it also catches a member already carried by an EARLIER member of the same roster. Fails CLOSED: if git cannot answer, the check returns false and the merge proceeds. A false negative costs one empty commit; a false positive drops live work. The driver ejects it via the existing `Conflict` outcome with no files, which the state machine already classifies `Unattributed` — one ejection path, not two. The trail line says `outcome=stale` so it is never confused with a genuine merge conflict. Ejected by NAME rather than silently skipped: a green candidate that never contained the member must not look like one that did. Test asserts all three properties — the landed member is ejected, not as Infrastructure (infra ejects nobody, so it would ride every future candidate), and the unmerged co-rider survives. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>The single POST that points the preview slot decides whether a 20-45 minute candidate build happens at all, and it had no retry. Worse, the failure test is `post.status` — CURL's exit code — so a connection refused during a rolling update is indistinguishable from a permanent failure. Measured 2026-08-02. The preview rolled twice in 90 minutes (a Flux apply of a merged PR, then a second kill) and every candidate racing it died instantly: generation=7 outcome=infra GET /app failed: curl: (7) Failed to connect ...after 27 MINUTES of real compilation generation=8 outcome=infra could not point preview slot "lane" generation=9 outcome=infra could not point preview slot "lane" generation=11 outcome=infra could not point preview slot "lane" Each one a fresh ejection and backoff for a member whose code was never at fault, and pr-10394 has now spent ~7 hours in this loop without landing. Retry up to POINT_ATTEMPTS (5) with POINT_RETRY_DELAY (6s) between — about 30s of tolerance, which covers the observed gap between a preview pod being killed and its replacement answering. Deliberately small. This covers a daemon that is RESTARTING, not one that is gone: a genuinely absent daemon must still surface as Infra quickly rather than hang the lane behind a long retry. The error message now reports the attempt count so a real outage is still legible. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>The preview slot reports setup failures through the same `last_red_reason` field as compile failures, and the leg turned every one of them into TreeState::Red. So on 2026-08-02, after a PVC fault removed the lane slot's worktree directory, the next candidate produced: last_red_reason=git checkout 8b6af9d3... failed: fatal: cannot change to '/workspace/cargoless-state/app/lane/worktree': No such file or directory and the lane ejected pr-10394 — a member whose code compiles fine — in TWELVE SECONDS. No tf-multiverse build can go red that fast; the elapsed time alone said it was not a verdict. A checkout/setup failure is OUR problem. It must report Infra so everyone stays queued and the lane retries, instead of a false accusation that also leaves the real fault unnoticed. Same rule as the already-tested `a_missing_artifact_on_green_is_infrastructure_not_a_red`. reason_is_infrastructure FAILS TOWARD RED by design. A false "infrastructure" verdict would keep a genuinely broken candidate queued until it landed, which is worse than a false accusation — an accusation at least stops the merge. So it matches only phrases that cannot plausibly come from a compiler or a failing test: git checkout / cannot change to / no such file or directory / worktree creation / ENOSPC / fetch / repository not found. Two tests. One asserts the exact reason string from the incident classifies as infrastructure, plus four other setup failures. The other pins the fail-toward-red direction across seven real red reasons — including "respawn ... failed health probe: no 200 on /health", which IS a code verdict (the tree does not boot) and must stay Red. All 12 cases validated against a faithful port before commit; local compilation is hook-blocked by design. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>The lane produced five green candidates on 2026-08-02 and merged nothing. Every one was killed at exactly 600s. `CommandLander::timeout` was `600s`, justified by "landing is a push plus N PR reconciles — seconds, not minutes." That is true of a lander that lands itself. The one actually configured is `scripts/ci/lane-land.sh`, which delegates to `scripts/merge-train-controller --land` — and the controller does not just push. It re-derives the candidate, publishes a merge-train ref, DISPATCHES A CANDIDATE BUILD and waits for the verdict, under its own `TRAIN_BUILD_MAX_WAIT_SECS` of 5400. A parent budget below the delegate's own ceiling cannot ever observe an outcome. It SIGKILLs a healthy land mid-wait and reports infrastructure failure, the driver re-enqueues, and the lane rebuilds the same members forever. The comment was not wrong about landing; it was describing a different lander than the one we run. 7200s, above the delegate's 5400 with room for the forge round-trips that bracket it, overridable via CARGOLESS_LANE_LAND_TIMEOUT_SECS. This does not remove the timeout — it makes it mean what it says. The invariant to keep is parent > delegate. The second half is why this took a day to find. The reason was DISCARDED: `execute()` returns `Vec::new()` for `LaneAction::Report` and nothing wrote a trail line for a failed land, so lane-runs.log read lane-build generation=1 outcome=green lane-build-start generation=2 members=pr-10394@4fc9c6c5 with nothing in between — a green candidate silently rebuilding, which is indistinguishable from a lane that never tried to land. I diagnosed it from Forgejo status timestamps, not from our own trail, which is exactly what the trail (task 32) exists to prevent. Both outcomes now write a line: a land is the only step that moves the trunk and has the same claim on the trail as the verdict that authorised it. Timestamps, UTC: green + merge-train/queued at 22:15:17, slot heartbeat moves 22:25:39. 600s, to the second. Same shape at 21:27, 13:08, 12:55, 12:07. The parse half is split out as a pure fn so its five tests need no environment: `set_var` is unsafe in Edition 2024, and this crate already carries a known env-lock flake from exactly that pattern (CGLS-26 warm-target). Tests assert against the controller's real 5400 rather than a copy of our own default, so lowering it back toward the delegate fails. All 9 vectors validated against a Rust-faithful port before commit (local compilation is hook-blocked by design). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>A red build's reason was `tail_lines(&format!("{stdout}\n{stderr}"), 20)`. That makes real reds undiagnosable, because rustc prints warnings AFTER the error that stopped the build, then a one-line summary. Observed 2026-08-03 on a broken `triform-physics` on dev: 63 warnings, so the last 20 lines were GUARANTEED to be warnings plus error: could not compile `triform-physics` (lib) due to 1 previous error; 63 warnings emitted The 1039-char reason stored on the slot began mid-warning at line 1587 and contained no diagnostic at all. I could see WHICH FILES were implicated (app_ops.rs:630, lifecycle.rs:533, from warning bodies) but never WHAT WAS WRONG — on the one build where it mattered, because that red was falsely ejecting innocent PRs from the build lane and I needed to prove the base was broken, not the member. There was nowhere else to look. No build log is written to disk, the app-serve container log does not carry it, and NO CI WORKFLOW COMPILES THIS CRATE — the preview is the only thing that builds the debug server. The truncated string was the only evidence in existence and it had discarded the error. So anchor on the first line that OPENS an error and keep the window after it. rustc emits the diagnostic body (`-->`, the source excerpt, `= help:`) immediately following that line, which is exactly what a human needs. `opens_an_error` deliberately REJECTS the trailing summary: `error: could not compile ...` and `error: aborting due to N previous errors` are not diagnostics, and anchoring on them would keep the precise useless line the old code kept. It accepts `error:` and `error[E1234]:`, trims leading whitespace (cargo indents sub-crate output), and rejects warning/note/help. Falls back to `tail_lines` when no error line is found — a step can fail without rustc (missing binary, shell error) and the tail is right for those. Five tests, the first reproducing the exact 2026-08-03 shape (error, then 63 warnings, then the summary) and asserting the excerpt LEADS with `error[E0308]` and carries `app_ops.rs:630`. Plus: the summary is not an anchor, a non-rustc failure keeps its tail, an indented error still anchors, and empty input does not panic (the slice arithmetic is the risk). Logic validated against a Rust-faithful port before commit — old capture contained E0308: false; new capture: true. Local compilation is hook-blocked by design. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>The lane ejected FOUR members in 90 minutes on 2026-08-03 for a defect none of them caused, and one of them — pr-10572 — changes a single YAML file and no Rust at all. It was ejected for `error: could not compile triform-physics`. A YAML change cannot break a Rust compile. The accusation was impossible on its face and the lane made it anyway, twice. Root cause upstream: a commit added a `debug!()` call without adding `debug` to the file's `use tracing::{...}`. The base stopped compiling. The candidate is base + members, so every candidate inherited that failure, and the lane attributed each one to whichever member happened to be aboard. Nothing in the lane could tell the two apart, because from the candidate's own red they are INDISTINGUISHABLE. The evidence needed was already on hand: the base slot builds the base with NO candidate merged in, and it is in the SAME `GET /app` snapshot the runner has already fetched. Base red + candidate red with the same failure => infrastructure, eject nobody. One map lookup, no extra request. Compared on the trailing 60 chars, not the whole string. The two builds run at different shas in different worktrees, so their reasons diverge in leading path and step detail while ending in the same compiler verdict — `error: could not compile X (lib) due to N previous errors; M warnings emitted`. That tail is what identifies the failure; the head is noise. Both real fixtures are in the tests and they DO differ up front, which is the point. Fails toward attributing, deliberately. Every "cannot tell" shape — no such slot, missing field, unparseable snapshot — yields empty, and empty is never a match: no evidence must not read as agreement. A missed base-red costs one false ejection, which the lane already survives; a wrongly-suppressed red would let a broken member land. Off unless `CARGOLESS_LANE_BASE_SLOT` names a slot, so a project with no base-only slot is unaffected. On tf-multiverse that is `dev`. Six tests, built on the real production strings from the incident: the false ejection is caught; the comparison survives differing leading detail; a genuine member fault is still attributed; empty never matches; the named slot is read (not the first one); and five unreadable-snapshot shapes each still let the red through. Logic validated against a Rust-faithful port before commit — all nine assertions pass, including the five-shape unreadable sweep. Local compilation is hook-blocked by design. Does NOT fix the timing tell recorded alongside this: a red returning in 6 minutes when a real build takes 90 is base-shaped, and the lane still does not know that. This guard is the evidence-based half; the cheap heuristic is separate work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>2e6ca58, 5/6 green`LaneEvent::Withdraw` has existed and been tested since the state machine was written. Nothing could reach it: there was no verb on either transport. So a member entered the lane and left only by building green or being ejected by a red it caused — and a member that can do NEITHER rebuilds forever. pr-10394 is that member. It is `failure` on "dev portal SSR check", a REQUIRED context in dev's branch protection, so the forge refuses the merge however green the candidate builds. The lander said so in as many words — [train] single-member train red: PR #10394 is the cause — and the lane immediately started generation 15 with the same member, its third build. Each costs ~45 minutes on the single preview slot, which is the entire throughput of the queue. The only way to stop it was to restart the daemon, which discards the whole in-memory queue including innocent members. The verb is the small half. The real bug is that `Withdraw` cleared `queue` and `ejected` but NOT `in_flight`, which reads as complete and is not: `on_build_finished` takes `in_flight` and requeues whatever it finds there on any non-green outcome. A member withdrawn mid-build therefore came BACK when the build it was withdrawn from ended — minutes later, unobserved, in exactly the case the verb exists for. Validated against a Rust-faithful port: without the `in_flight` line the withdrawn member ends up EJECTED by that build; with it, gone. It deliberately does NOT cancel the running build. The candidate tree is materialised and the compile is already paid for; killing it would waste that work and deny a verdict to any other member aboard. The build finishes and is attributed to whoever is left — and if that is nobody, the outcome applies to nobody and the lane goes idle. `LaneHost::withdraw` does NOT pre-check the snapshot the way `readmit` does. The snapshot is one pump behind, and a member enqueued during a build sits in the channel where the snapshot cannot see it at all, so a "not found" refusal would make the verb useless in precisely the situation that motivates it. The lane answers authoritatively; an unknown id is a harmless no-op. Added to BOTH transports because transport/unix.rs's match is exhaustive on purpose — its comment says a new verb should fail to compile in both until someone decides what it means there, rather than silently working over HTTP only. It should. Two tests, both on the path that was broken: a member withdrawn mid-build is absent after that build ends (and does not acquire an ejection — it is gone, not blamed), and withdrawing one of several leaves the others owning the red they caused. The pre-existing test only covered withdrawing an already-ejected member, which is why this survived. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>Four defects that share one property: every decision the lane made was correct, and every surface an author or operator reads was wrong. A gate whose status surface lies is worse than one with no surface, because the lie gets acted on. 1. A MEMBER ENQUEUED DURING A BUILD WAS INVISIBLE FOR THE WHOLE BUILD. `enqueue` returns the instant the event is in the mpsc channel, and nothing reads that channel until the worker returns from `pump` — for a real lane, ~45 minutes. Observed: `POST /lane` answered `{"detail":"queued `pr-6956`","ok":true}` and `queue_depth` stayed 0 across 36s of polling. An author reads that as "the lane never got it" and re-submits, which is the correct inference from what we told them. The host now keeps its own set of accepted-but-not-yet-stepped ids and reports the union, de-duplicated against the lane's own queue/in_flight/ejections. The merge happens at READ time, not at publish time: the worker may not republish for the length of a build, so a `withdraw` in that window has to take effect on the next read rather than the next republish. `queued` names them, because `queue_depth: 3` still cannot be reconciled against your own PR. 2. `/lane` REPORTED `phase=idle` WHILE THE LANDER RAN. Same family, worse consequence, and NOT fixable the way the build case was. By the time `LandAndPublish` executes, the lane is genuinely `Idle` — green verdict in, `in_flight` emptied — while the lander moves the trunk for up to 7200s (the real one delegates to a merge-train controller that waits on its own candidate build). The phase is honest and still misleading, so no amount of transition-watching would have helped. `pump_observed` now brackets every BLOCKING action with a `LaneActivity`, and the snapshot carries `activity` + `landing` beside `phase`. It reports the driver's activity; it does not falsify the phase. 3. `materialize` COULD FAIL WITH A BARE `os error 2` NAMING NOTHING. Observed: `lane-build generation=13 outcome=infra reason=candidate tree could not be materialized: No such file or directory (os error 2)`. The candidate root, the scratch parent, the repo and a `git` that cannot be spawned all produce exactly those bytes, with different fixes. Both fallible filesystem steps now name the operation and the path via `MaterializeError::infra_at`, preserving `io::ErrorKind`; `git()` annotates spawn failures with the cwd it tried (a missing cwd, not only a missing binary, yields `os error 2` — the PVC-fault shape). 4. THE LANE SPUN A GENERATION EVERY ~30s AGAINST AN UNREACHABLE PREVIEW DAEMON. 51 wasted generations in one night. The backoff was not missing — it could not fire. `LaneState::now` moves only on `LaneEvent::Tick`, and the host's ticks sit unread in the channel for the whole of a blocking action because the worker is inside `execute`. So every deadline the outcome computes is anchored at the moment the attempt STARTED. Pointing the preview slot takes ~24-35s (5 attempts, 6s apart) against a 30-tick backoff, so `infra_retry_after = started + 30` was already in the past when it was written, and the drained tick backlog cleared it on arrival. On the land path it is starker: a 7200s budget against a 30-tick backoff can never delay anything. `LaneDriver` now re-syncs the lane's clock from the host's own tick stream immediately after each blocking action, before the outcome is applied. Deliberately `advance_clock`, NOT a `Tick`: a Tick also runs `maybe_start_build`, and after a failed land the phase is already `Idle` with the members re-enqueued — so a Tick there would start the next build before `LandFailed` installs the backoff, reintroducing the exact hot loop that event was added to prevent. The retry is paced, never removed. TESTS — which pins which defect DEFECT 1 lanehost::tests::a_member_enqueued_during_a_build_is_visible_immediately Asserts on the FIRST read after the POST, with no polling loop, so a fix that merely shortened the window cannot pass. lanehost::tests::withdrawing_a_member_still_in_the_channel_removes_it_from_the_snapshot The read-time-merge property. Found by writing the test. lane_policy::the_lane_reports_who_is_queued_not_just_how_many `queued`/`queue_depth` describe the same set; in-flight members leave the queue, which is what the host de-duplicates against. DEFECT 2 lanehost::tests::the_snapshot_says_landing_while_the_lander_runs Blocks inside the lander and asserts `activity=landing` + the roster, AND that `phase` stays `idle` — the fix reports the activity, it does not falsify the phase. Also asserts it returns to `settled`, since a successful land emits no follow-up event. lanehost::tests::the_snapshot_says_building_while_the_legs_run The case that already worked, pinned against regression. DEFECT 3 lanetree::tests::a_scratch_dir_that_cannot_be_created_names_the_path_and_the_step Real ENOTDIR (scratch parent inside a regular file), not a constructed error. lanetree::tests::a_git_that_cannot_be_spawned_names_the_directory Real ENOENT from a missing cwd. lanedrv::materialize_context_tests — three tests, pinned against the exact generation-13 string, including that `io::ErrorKind` survives the annotation. DEFECT 4 lane_policy::an_infra_backoff_is_measured_from_the_failure_not_the_attempt Attempt takes LONGER than the backoff — the real preview-point case. Asserts the boundary tick, so a backoff that is present but effectively zero fails. Asserts the retry still happens. lane_policy::advancing_the_clock_does_not_itself_start_a_build The Tick-vs-advance_clock distinction, proven by stepping the machine both ways from the same state — never by comparing two constants, which `clippy::assertions_on_constants` folds away. Validated against a Rust-faithful Python port of `LaneState::step`, `pump_observed` and `LaneSnapshot` before pushing (cargo is hook-blocked; CI is the only build). The port measured OLD=9 vs NEW=5 generations per 300 wall-seconds against a failing preview, and caught two things review did not: a mis-measured land assertion of mine, and the publish-time-vs-read-time merge bug in defect 1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>`GET /lane` now reports `activity: landing` while the lander runs, but a snapshot dies with the pod and the TRAIL is what is left afterwards. And the trail had exactly the same gap: `run_build` writes `lane-build-start` before blocking, `execute`'s land arm wrote nothing until the land returned. So a daemon rolled or killed during a land left `outcome=green` followed by silence — a record indistinguishable from a lane that never tried to land at all. That is the same shape that hid a 600s lander timeout for a full day on 2026-08-02: five greens, five kills, and the only evidence was Forgejo status timestamps. A land is the only step that moves the trunk and it can run for up to 7200s. It gets a start line naming the roster by `id@head`, for the same reason a build does — and because `in_flight` is already empty by that point, so nothing else can answer "who was being landed". TEST lane_real_io::the_verdict_and_per_leg_timings_outlive_the_candidate_worktree Extended. Asserts the start line exists, names the roster by id@head, and PRECEDES the outcome line — a start written after the land would prove nothing about the window it exists to cover. Read after the whole repo is deleted, like the rest of that test. Also folds in two cosmetic fixes to the previous commit: two intra-doc links pointed at a private const (`LAND_TIMEOUT_DEFAULT_SECS`), and one `&x == y` comparison is written `x == *y`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>b40f9bc