triform/cargoless
Template
1
0
Fork 0
Code Issues 1 Pull requests 18 Projects Releases Packages Wiki Activity Actions

feat(lane): record ejections in the durable trail, and name the dispatcher reason on EX_TEMPFAIL #174

Open
triform-admin wants to merge 5 commits from agent/lane-eject-trail into main
pull from: agent/lane-eject-trail
merge into: triform:main
Conversation 0 Commits 5 Files changed 2 +219 -6
triform-admin commented 2026-08-24 17:19:24 +00:00
Owner
Copy link

An ejection was the only lane decision missing from the durable trail.

Every other outcome writes a [cargoless:obs] lane-* line — build-start, green, red, conflict, stale, infra, land-start, land. But LaneAction::Eject and LaneAction::Readmit hit a Vec::new() arm in LaneDriver::execute and vanished.

That matters because GET /lane reports only live ejections: the moment one lapses on its TTL or the member is re-admitted, the sentence is gone. So the one decision an author is asked to act on left no durable record, and after a pod restart "who was ejected, and why" was unrecoverable.

Not hypothetical — production reds at generations 70, 191, 202 and 276 on 2026-08-22 each held members. By the time the wire was sampled the ejections had lapsed, and the trail showed only:

[cargoless:obs] lane-build generation=191 outcome=red diagnostics=1

…with nothing about who it held or why. Now:

[cargoless:obs] lane-eject id=<id> cause=<cause> kind=<kind> why=<sentence>
[cargoless:obs] lane-readmit id=<id> why=<why>

Design notes

  • Written in the driver, not LaneState, so the state machine stays free of IO — the same split every other trail line already follows.
  • The sentence comes from EjectReason::describe_for, and kind from the same three-way match LaneSnapshot::of publishes, so the durable record cannot drift from what the wire and the forge status say.
  • Report keeps its existing no-op: it is a progress notification, not a decision, and logging every building (generation N) would bury the decisions.

Test: an_ejection_is_recorded_in_the_durable_trail_with_its_sentence drives a real ejection through a real LaneDriver with a real trail file and asserts the line carries the member id, the cause, and the author-facing sentence — not just an enum tag. It uses the already_landed path so no compile is needed, keeping it fast and hermetic.

Local cargo is hook-blocked by policy, so the compile is proven by CI.

Second commit: name the dispatcher's reason on EX_TEMPFAIL

Found while watching the deployed lane-shadow during this PR's soak, and it is
the same defect one field over, in the same function.

run_to_completion captures the dispatcher's stdout and stderr into
combined. The EX_ROSTER_STALE branch parses that text. The EX_TEMPFAIL
branch three lines above discards it and reports a bare exit 75 — but a
real dispatcher reaches that exit a dozen ways (unfetchable ref, moved
candidate, no merge base, unreachable preview slot) and names which one on
stderr every time.

Measured on the deployed lane (3510 generations, 2d20h, 0 restarts):

1164  outcome=infra      <-- LARGEST bucket, ahead of landed (523)
 909  outcome=conflict
 594  outcome=green
 523  outcome=landed

986 of those 1164 say only "dispatcher reported a transient failure (exit
75)". The lane's most common failure was its least explained. Generations
548 / 550 / 551 spun on the same roster with no recorded cause.

The fix takes the last non-blank line (shells print context first, verdict
last), neutralises control characters, and bounds the length — so it is safe to
paste into a trail line, a forge status, or a PR comment. Silence reads as
silence ("and it printed no reason"), never as an empty quote.

Six unit tests cover the last-word case, trailing blanks, silence, a runaway
10k-char line, multi-byte truncation (slicing by byte would panic the lane
during an infra failure — turning a diagnosis into an outage), and control
characters.

Not in this PR: the lane still publishes no lane-wide reason, so an infra
stall is indistinguishable from a healthy idle lane on GET /lane — nobody is
ejected, so ejections[].why never fires. Filed as CGLS-48 rather than
widening this PR.

🤖 Generated with Claude Code

**An ejection was the only lane decision missing from the durable trail.** Every other outcome writes a `[cargoless:obs] lane-*` line — build-start, green, red, conflict, stale, infra, land-start, land. But `LaneAction::Eject` and `LaneAction::Readmit` hit a `Vec::new()` arm in `LaneDriver::execute` and vanished. That matters because `GET /lane` reports only **live** ejections: the moment one lapses on its TTL or the member is re-admitted, the sentence is gone. So the one decision an author is asked to *act on* left no durable record, and after a pod restart "who was ejected, and why" was unrecoverable. Not hypothetical — production reds at generations **70, 191, 202 and 276** on 2026-08-22 each held members. By the time the wire was sampled the ejections had lapsed, and the trail showed only: ``` [cargoless:obs] lane-build generation=191 outcome=red diagnostics=1 ``` …with nothing about who it held or why. Now: ``` [cargoless:obs] lane-eject id=<id> cause=<cause> kind=<kind> why=<sentence> [cargoless:obs] lane-readmit id=<id> why=<why> ``` **Design notes** - Written in the **driver**, not `LaneState`, so the state machine stays free of IO — the same split every other trail line already follows. - The sentence comes from `EjectReason::describe_for`, and `kind` from the same three-way match `LaneSnapshot::of` publishes, so the durable record **cannot drift** from what the wire and the forge status say. - `Report` keeps its existing no-op: it is a progress notification, not a decision, and logging every `building (generation N)` would bury the decisions. **Test:** `an_ejection_is_recorded_in_the_durable_trail_with_its_sentence` drives a real ejection through a real `LaneDriver` with a real trail file and asserts the line carries the member id, the cause, **and the author-facing sentence** — not just an enum tag. It uses the `already_landed` path so no compile is needed, keeping it fast and hermetic. Local cargo is hook-blocked by policy, so the compile is proven by CI. --- ## Second commit: name the dispatcher's reason on `EX_TEMPFAIL` Found while watching the deployed `lane-shadow` during this PR's soak, and it is the **same defect one field over**, in the same function. `run_to_completion` captures the dispatcher's stdout **and** stderr into `combined`. The `EX_ROSTER_STALE` branch parses that text. The `EX_TEMPFAIL` branch three lines above **discards it** and reports a bare `exit 75` — but a real dispatcher reaches that exit a dozen ways (unfetchable ref, moved candidate, no merge base, unreachable preview slot) and names which one on stderr every time. Measured on the deployed lane (3510 generations, 2d20h, 0 restarts): ``` 1164 outcome=infra <-- LARGEST bucket, ahead of landed (523) 909 outcome=conflict 594 outcome=green 523 outcome=landed ``` **986 of those 1164 say only "dispatcher reported a transient failure (exit 75)".** The lane's most common failure was its least explained. Generations 548 / 550 / 551 spun on the same roster with no recorded cause. The fix takes the last non-blank line (shells print context first, verdict last), neutralises control characters, and bounds the length — so it is safe to paste into a trail line, a forge status, or a PR comment. Silence reads as silence ("and it printed no reason"), never as an empty quote. Six unit tests cover the last-word case, trailing blanks, silence, a runaway 10k-char line, **multi-byte truncation** (slicing by byte would panic the lane *during* an infra failure — turning a diagnosis into an outage), and control characters. **Not in this PR:** the lane still publishes no *lane-wide* reason, so an infra stall is indistinguishable from a healthy idle lane on `GET /lane` — nobody is ejected, so `ejections[].why` never fires. Filed as **CGLS-48** rather than widening this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(lane): record ejections and re-admissions in the durable trail
Some checks failed
ci / build (pull_request) Failing after 56s
Details
ci / test (pull_request) Failing after 54s
Details
ci / fmt (pull_request) Successful in 41s
Details
ci / clippy (pull_request) Failing after 51s
Details
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / lane policy mutation proof (pull_request) Successful in 3m14s
Details
946f33d99a 
An ejection was the ONLY lane decision missing from `<state_dir>/lane-runs.log`.
Every other outcome writes a `[cargoless:obs] lane-*` line — build-start,
green, red, conflict, stale, infra, land-start, land — but `LaneAction::Eject`
and `LaneAction::Readmit` hit a `Vec::new()` arm in `LaneDriver::execute` and
vanished.

That mattered because `GET /lane` reports only LIVE ejections: the moment one
lapses on its TTL or the member is re-admitted, the sentence is gone. So the
one decision an author is asked to ACT on left no durable record, and after a
pod restart "who was ejected, and why" was unrecoverable.

Not hypothetical: production reds at generations 70, 191, 202 and 276 on
2026-08-22 each held members, and by the time the wire was sampled the
ejections had lapsed. The trail showed `outcome=red diagnostics=1` and nothing
about who it held.

    [cargoless:obs] lane-eject id=<id> cause=<cause> kind=<kind> why=<sentence>
    [cargoless:obs] lane-readmit id=<id> why=<why>

Written in the DRIVER, not in `LaneState`, so the state machine stays free of
IO — the same split every other trail line already follows. The sentence comes
from `EjectReason::describe_for`, and `kind` from the same three-way match
`LaneSnapshot::of` publishes, so the durable record cannot drift from what the
wire and the forge status say. `Report` keeps its existing no-op: it is a
progress notification, not a decision, and logging every "building
(generation N)" would bury the decisions.

Test: `an_ejection_is_recorded_in_the_durable_trail_with_its_sentence` drives a
real ejection through a real `LaneDriver` with a real trail file and asserts
the line carries the member id, the cause, AND the author-facing sentence —
not just an enum tag. It uses the `already_landed` path so no compile is
needed, keeping it fast and hermetic.

Local cargo is hook-blocked by policy, so the compile is proven by CI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(test): read the lane generation before the &mut borrow
Some checks failed
ci / build (pull_request) Successful in 1m0s
Details
ci / test (pull_request) Failing after 1m5s
Details
ci / fmt (pull_request) Successful in 48s
Details
ci / clippy (pull_request) Successful in 46s
Details
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / lane policy mutation proof (pull_request) Successful in 3m19s
Details
64ebe3ea7e 
CI failed all three compiling jobs (build, test, clippy) on one error in the
test I added:

    error[E0502]: cannot borrow `lane` as immutable because it is also
                  borrowed as mutable
      --> crates/cargoless-core/tests/lane_real_io.rs:1357:25

`drv.pump(&mut lane, LaneEvent::BuildFinished { generation: lane.generation(),
… })` takes the mutable borrow in the first argument and then reads `lane`
immutably while constructing the second. Hoisted the read to a `let` above the
call.

Test-only; the driver change in the parent commit is untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(test): assert the ejection line's SHAPE, not one specific cause
Some checks failed
ci / build (pull_request) Successful in 1m5s
Details
ci / test (pull_request) Failing after 2m18s
Details
ci / fmt (pull_request) Successful in 48s
Details
ci / clippy (pull_request) Successful in 58s
Details
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / lane policy mutation proof (pull_request) Successful in 3m52s
Details
0a94a5f35b 
CI proved the feature works — the trail line is written exactly as intended:

    [cargoless:obs] lane-eject id=a cause=build_failure kind=unattributed
                    why=build failed, but the errors are in files NO queued
                    change touches, so the cause could not be attributed…

but the test asserted `cause=already_landed`. The fixture's `manifest` leg
reports Red as soon as the member is enqueued, so the real ejection is a
build_failure and the Stale event never decides the outcome. The assertion was
wrong about the fixture, not about the behaviour.

Now asserts what actually matters: the line names the member, carries a cause
AND a kind, and its `why=` is a real sentence rather than an enum tag. Pinning
one of the four causes made the test a statement about fixture timing.

Verified the assertion keeps its teeth against both regressions it exists to
catch:

    pre-fix (no line at all)  -> FAIL (caught)
    tag-only (why= empty)     -> FAIL (caught)
    correct real output       -> PASS

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chore: re-roll CI (2 known load-flakes, neither in this diff)
Some checks are pending
ci / build (pull_request) Waiting to run
Details
ci / test (pull_request) Waiting to run
Details
ci / lane policy mutation proof (pull_request) Waiting to run
Details
ci / fmt (pull_request) Waiting to run
Details
ci / clippy (pull_request) Waiting to run
Details
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Waiting to run
Details
d2c2bcb946 
The `test` job failed 648-passed/2-failed on:
  project_checks::tests::distinct_target_keys_run_concurrently_under_a_warm_target
  transport::unix::tests::subscribe_streams_transitions_over_the_socket

Both are timing-sensitive under load (a concurrency-overlap assertion and a
socket read timeout) and neither file is in this diff, which touches only
lanedrv.rs and lane_real_io.rs. The new trail test PASSED in that run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat(lane): name the dispatcher's reason on EX_TEMPFAIL
All checks were successful
ci / build (pull_request) Successful in 1m23s
Details
ci / test (pull_request) Successful in 1m21s
Details
ci / fmt (pull_request) Successful in 42s
Details
ci / clippy (pull_request) Successful in 48s
Details
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / lane policy mutation proof (pull_request) Successful in 3m8s
Details
7058ac1d99 
The EX_TEMPFAIL branch discarded the dispatcher's stderr and reported only
"exit 75". A real dispatcher reaches that exit a dozen ways -- unfetchable
ref, moved candidate, no merge base, unreachable preview slot -- and names
which one on stderr every time. `run_to_completion` already captures it: the
same `text` the EX_ROSTER_STALE branch three lines below parses.

Measured on the deployed lane-shadow: `infra` is the LARGEST outcome bucket
(1164 of 3510 generations, ahead of `landed` at 523), and 986 of those say
only "dispatcher reported a transient failure (exit 75)". The lane's most
common failure was its least explained, and three consecutive generations
(548/550/551) spun on the same roster with no recorded cause.

Same defect shape as the ejection sentence in this PR: a reason computed,
then dropped at the boundary. Last non-blank line (shells print context
first, verdict last), control chars neutralised and length bounded so it is
safe to paste into a trail line, forge status, or PR comment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
triform-admin changed title from feat(lane): record ejections and re-admissions in the durable trail to feat(lane): record ejections in the durable trail, and name the dispatcher reason on EX_TEMPFAIL 2026-08-24 19:02:53 +00:00
All checks were successful
ci / build (pull_request) Successful in 1m23s
Details
ci / test (pull_request) Successful in 1m21s
Details
ci / fmt (pull_request) Successful in 42s
Details
ci / clippy (pull_request) Successful in 48s
Details
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / lane policy mutation proof (pull_request) Successful in 3m8s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin agent/lane-eject-trail:agent/lane-eject-trail
git switch agent/lane-eject-trail

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff agent/lane-eject-trail
git switch agent/lane-eject-trail
git rebase main
git switch main
git merge --ff-only agent/lane-eject-trail
git switch agent/lane-eject-trail
git rebase main
git switch main
git merge --no-ff agent/lane-eject-trail
git switch main
git merge --squash agent/lane-eject-trail
git switch main
git merge --ff-only agent/lane-eject-trail
git switch main
git merge agent/lane-eject-trail
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
triform/cargoless!174
No description provided.