fix(lane): close abandoned generations on boot (rebased onto main) #142

Merged

triform-admin merged 2 commits from agent/lane-abandoned-verdict-v2 into main 2026-08-04 15:07:54 +00:00

Conversation 0 Commits 2 Files changed 2 +240

triform-admin commented 2026-08-04 05:28:36 +00:00

Owner

Copy link

Replaces #137, which went unmergeable when #139 landed on main and touched the same files.

Cherry-picked onto current main (1753920). The conflict was in lanedrv.rs: main added point_retry_budget_tests, this branch adds generation_after + abandoned_generation_tests. Both are additive and disjoint — resolved by keeping BOTH, verified byte-identical to each side.

run_build writes lane-build-start generation=N before compiling and the terminal lane-build generation=N outcome=... only after, so a daemon killed mid-build leaves a start with no end — 16 such orphans on the shadow lane right now. This closes them on the next boot with outcome=abandoned, making an orphaned start impossible by construction.

🤖 Generated with Claude Code

Replaces #137, which went unmergeable when #139 landed on main and touched the same files. Cherry-picked onto current main (1753920). The conflict was in `lanedrv.rs`: main added `point_retry_budget_tests`, this branch adds `generation_after` + `abandoned_generation_tests`. Both are additive and disjoint — resolved by keeping BOTH, verified byte-identical to each side. `run_build` writes `lane-build-start generation=N` before compiling and the terminal `lane-build generation=N outcome=...` only after, so a daemon killed mid-build leaves a start with no end — 16 such orphans on the shadow lane right now. This closes them on the next boot with `outcome=abandoned`, making an orphaned start impossible by construction. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

triform-admin added 1 commit 2026-08-04 05:28:36 +00:00

fix(lane): close abandoned generations on boot — an orphaned start is now impossible ... e4637848ff

`run_build` writes `lane-build-start generation=N` BEFORE it compiles and the
matching `lane-build generation=N outcome=…` only after. A daemon killed
mid-build therefore leaves a start with no end, and the replacement process
begins at generation 0 and never mentions N again.

That makes an abandoned build INDISTINGUISHABLE from one still running. The
trail stops being able to answer "is the lane working?" after the fact — which
is the entire reason it exists.

Measured on the shadow lane 2026-08-03/04: 15 orphaned starts across 200,
roughly one per pod roll. cargoless-lane-shadow is Flux-managed and lane work
lands on dev continuously, so it rolled five times in one day; since the ingest
CronJob began feeding the lane continuously it is almost never idle when a roll
lands, so nearly every roll now destroys work in flight.

On construction — before the lane accepts anything — read the trail back and
append `outcome=abandoned reason=daemon restarted before this build reported`
for every generation left open. The invariant that restores: EVERY
lane-build-start has a terminal line.

Deliberately reconstructed, not persisted. The trail already records what was
in flight, so there is no second file to keep consistent — the same instinct
that lets SHA-bound enrollment survive a restart. And it is infallible by
construction: a missing, unreadable or unwritable trail leaves the status quo,
because observability must never be the thing that stops the lane starting.

Tests (7): the interrupted case closes; a completed build is NOT re-closed
(no contradictory second verdict); a `lane-leg` line is not mistaken for a
verdict, which would leave a genuinely abandoned build open forever; closing is
idempotent so a crash loop cannot append one `abandoned` per boot; a missing
trail neither panics nor gets created; and the prefix parser refuses to let
`lane-build generation=` match a `lane-build-start` line.

No local cargo (CI-only by design): logic verified by a line-for-line Python
port over six fixtures including the real production trail shape (generations
9 and 1 both open) and the lane-land lines that must be ignored — all pass.
`rustfmt --check` clean, braces balanced, fn confirmed at module scope (not
nested in a cfg(test) mod), all required imports already present.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

triform-admin added 1 commit 2026-08-04 09:25:38 +00:00

chore: re-roll — the test job wedged (52 min with no runner update vs a 7-min average) ... 6ce39170b1

Not a code fault and not fleet slowness: four sibling jobs on this same sha
went green (build, clippy, fmt, lane policy mutation proof), and this job was
the OLDEST running in the whole docker fleet — 59 min against a 24h average of
7 and a max of 20. `updated` had not advanced since `started`, so the runner
claimed it and stopped reporting.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

triform-admin merged commit 9686e22e64 into main 2026-08-04 15:07:54 +00:00

triform-admin deleted branch agent/lane-abandoned-verdict-v2 2026-08-04 15:07:55 +00:00

triform-admin referenced this pull request from a commit 2026-08-04 15:07:56 +00:00

fix(lane): close abandoned generations on boot (#142)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

triform/cargoless!142

No description provided.