triform/cargoless
Template
1
0
Fork 0
Code Issues 1 Pull requests 12 Projects Releases Packages Wiki Activity Actions 7

fix(lane): close abandoned generations on boot — an orphaned start is now impossible #137

Open
iggy wants to merge 3 commits from agent/lane-abandoned-verdict into main
pull from: agent/lane-abandoned-verdict
merge into: triform:main
Conversation 0 Commits 3 Files changed 2 +240
iggy commented 2026-08-04 00:59:33 +00:00
Contributor
Copy link

Summary

Every lane-build-start now ends up with a terminal line. On construction —
before the lane accepts any work — the trail is read back and any generation
left open by a previous process is closed with:

[cargoless:obs] lane-build generation=N outcome=abandoned \
  reason=daemon restarted before this build reported

Why

run_build writes the start before compiling and the outcome only after,
so a daemon killed mid-build leaves a start with no end — and the replacement
begins at generation 0 and never mentions N again. An abandoned build becomes
indistinguishable from one still running, which defeats the entire purpose
of a durable trail.

Measured on the shadow lane 2026-08-03/04: 15 orphaned starts across 200,
roughly one per pod roll. cargoless-lane-shadow is Flux-managed and lane work
lands on dev continuously, so it rolled five times in one day. Since the
ingest CronJob began feeding the lane continuously it is almost never idle when
a roll lands, so nearly every roll now destroys work in flight.

Design note

Deliberately reconstructed, not persisted. The trail already records what
was in flight, so there is no second file to keep consistent — the same
instinct that lets SHA-bound enrollment survive a restart.

Infallible by construction: a missing, unreadable or unwritable trail leaves
the status quo. Observability must never be the thing that stops the lane
starting.

Test plan

Seven tests, chosen so each one fails for a distinct reason:

  • an interrupted build is closed, and starts == terminals afterwards;
  • a completed build is not re-closed — no contradictory second verdict;
  • a lane-leg line is not mistaken for a verdict (that would leave a
    genuinely abandoned build open forever);
  • closing is idempotent, so a crash loop cannot append one abandoned per
    boot and bury the real history;
  • a missing trail neither panics nor gets created;
  • the prefix parser refuses to let lane-build generation= match a
    lane-build-start line.

No local cargo (CI-only by design): logic verified by a line-for-line Python
port over six fixtures, including the real production trail shape
(generations 9 and 1 both open) and the lane-land lines that must be ignored
— all pass. rustfmt --check clean, braces balanced, function confirmed at
module scope rather than nested in a cfg(test) mod, all imports already
present.

Non-goals

  • Resuming the interrupted build. This makes the loss visible and bounded;
    the member roster is already re-offered by the ingest CronJob within 10 min,
    so nothing is lost — only time.
  • Persisting LaneState (generation counter, queue). Separate change.

Related

Completes the pair with lane-land-start, which already landed: a roll
mid-land was likewise indistinguishable from one that never tried.

## Summary Every `lane-build-start` now ends up with a terminal line. On construction — before the lane accepts any work — the trail is read back and any generation left open by a previous process is closed with: ``` [cargoless:obs] lane-build generation=N outcome=abandoned \ reason=daemon restarted before this build reported ``` ## Why `run_build` writes the start **before** compiling and the outcome only after, so a daemon killed mid-build leaves a start with no end — and the replacement begins at generation 0 and never mentions N again. An abandoned build becomes **indistinguishable from one still running**, which defeats the entire purpose of a durable trail. Measured on the shadow lane 2026-08-03/04: **15 orphaned starts across 200**, roughly one per pod roll. `cargoless-lane-shadow` is Flux-managed and lane work lands on `dev` continuously, so it rolled **five times in one day**. Since the ingest CronJob began feeding the lane continuously it is almost never idle when a roll lands, so nearly every roll now destroys work in flight. ## Design note Deliberately **reconstructed, not persisted**. The trail already records what was in flight, so there is no second file to keep consistent — the same instinct that lets SHA-bound enrollment survive a restart. Infallible by construction: a missing, unreadable or unwritable trail leaves the status quo. Observability must never be the thing that stops the lane starting. ## Test plan Seven tests, chosen so each one fails for a distinct reason: - an interrupted build is closed, and starts == terminals afterwards; - a **completed** build is not re-closed — no contradictory second verdict; - a `lane-leg` line is **not** mistaken for a verdict (that would leave a genuinely abandoned build open forever); - closing is **idempotent**, so a crash loop cannot append one `abandoned` per boot and bury the real history; - a missing trail neither panics nor gets created; - the prefix parser refuses to let `lane-build generation=` match a `lane-build-start` line. No local cargo (CI-only by design): logic verified by a line-for-line Python port over six fixtures, including the **real production trail shape** (generations 9 and 1 both open) and the `lane-land` lines that must be ignored — all pass. `rustfmt --check` clean, braces balanced, function confirmed at module scope rather than nested in a `cfg(test)` mod, all imports already present. ## Non-goals - Resuming the interrupted build. This makes the loss **visible and bounded**; the member roster is already re-offered by the ingest CronJob within 10 min, so nothing is lost — only time. - Persisting `LaneState` (generation counter, queue). Separate change. ## Related Completes the pair with `lane-land-start`, which already landed: a roll mid-*land* was likewise indistinguishable from one that never tried.
fix(lane): close abandoned generations on boot — an orphaned start is now impossible
Some checks failed
ci / build (pull_request) Successful in 1m54s
Details
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / clippy (pull_request) Successful in 1m28s
Details
ci / fmt (pull_request) Successful in 3m19s
Details
ci / test (pull_request) Successful in 19m21s
Details
ci / lane policy mutation proof (pull_request) Failing after 19m17s
Details
560c442e83 
`run_build` writes `lane-build-start generation=N` BEFORE it compiles and the
matching `lane-build generation=N outcome=…` only after. A daemon killed
mid-build therefore leaves a start with no end, and the replacement process
begins at generation 0 and never mentions N again.

That makes an abandoned build INDISTINGUISHABLE from one still running. The
trail stops being able to answer "is the lane working?" after the fact — which
is the entire reason it exists.

Measured on the shadow lane 2026-08-03/04: 15 orphaned starts across 200,
roughly one per pod roll. cargoless-lane-shadow is Flux-managed and lane work
lands on dev continuously, so it rolled five times in one day; since the ingest
CronJob began feeding the lane continuously it is almost never idle when a roll
lands, so nearly every roll now destroys work in flight.

On construction — before the lane accepts anything — read the trail back and
append `outcome=abandoned reason=daemon restarted before this build reported`
for every generation left open. The invariant that restores: EVERY
lane-build-start has a terminal line.

Deliberately reconstructed, not persisted. The trail already records what was
in flight, so there is no second file to keep consistent — the same instinct
that lets SHA-bound enrollment survive a restart. And it is infallible by
construction: a missing, unreadable or unwritable trail leaves the status quo,
because observability must never be the thing that stops the lane starting.

Tests (7): the interrupted case closes; a completed build is NOT re-closed
(no contradictory second verdict); a `lane-leg` line is not mistaken for a
verdict, which would leave a genuinely abandoned build open forever; closing is
idempotent so a crash loop cannot append one `abandoned` per boot; a missing
trail neither panics nor gets created; and the prefix parser refuses to let
`lane-build generation=` match a `lane-build-start` line.

No local cargo (CI-only by design): logic verified by a line-for-line Python
port over six fixtures including the real production trail shape (generations
9 and 1 both open) and the lane-land lines that must be ignored — all pass.
`rustfmt --check` clean, braces balanced, fn confirmed at module scope (not
nested in a cfg(test) mod), all required imports already present.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chore: re-roll — the mutation-proof job hit a full runner disk, not a code fault
Some checks failed
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / clippy (pull_request) Successful in 5m29s
Details
ci / build (pull_request) Successful in 9m36s
Details
ci / test (pull_request) Failing after 9m19s
Details
ci / fmt (pull_request) Successful in 9m22s
Details
ci / lane policy mutation proof (pull_request) Successful in 10m27s
Details
d9ce335263 
The job caught two mutants correctly and then died on
`No space left on device` (forgejo-runner-5 at 92%). build/test/clippy/fmt all
passed on this same commit. Pruned 1.4G of unused images on that runner; this
empty commit re-dispatches.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chore: re-roll — warm-target env-lock flake, not this change
All checks were successful
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / build (pull_request) Successful in 2m24s
Details
ci / test (pull_request) Successful in 3m51s
Details
ci / clippy (pull_request) Successful in 3m50s
Details
ci / fmt (pull_request) Successful in 8m21s
Details
ci / lane policy mutation proof (pull_request) Successful in 11m59s
Details
7459273a64 
`test` failed on three project_checks warm-target cases
(distinct_target_keys_run_concurrently_under_a_warm_target,
a_shared_target_key_still_serializes, checks_without_a_stage_all_share_stage_zero).

Not mine: this branch touches only lanedrv.rs and serveapi.rs. The family is
the known load-flake — three failing together points at contention on the
shared warm target, and `test` is green on 5 of the last 8 cargoless commits
from other authors and red on 2, so it is intermittent fleet-wide rather than
specific to this tree.

The mutation-proof job PASSED this run, confirming the runner-5 disk fix held.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
All checks were successful
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / build (pull_request) Successful in 2m24s
Details
ci / test (pull_request) Successful in 3m51s
Details
ci / clippy (pull_request) Successful in 3m50s
Details
ci / fmt (pull_request) Successful in 8m21s
Details
ci / lane policy mutation proof (pull_request) Successful in 11m59s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin agent/lane-abandoned-verdict:agent/lane-abandoned-verdict
git switch agent/lane-abandoned-verdict

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff agent/lane-abandoned-verdict
git switch agent/lane-abandoned-verdict
git rebase main
git switch main
git merge --ff-only agent/lane-abandoned-verdict
git switch agent/lane-abandoned-verdict
git rebase main
git switch main
git merge --no-ff agent/lane-abandoned-verdict
git switch main
git merge --squash agent/lane-abandoned-verdict
git switch main
git merge --ff-only agent/lane-abandoned-verdict
git switch main
git merge agent/lane-abandoned-verdict
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
triform/cargoless!137
No description provided.