triform/cargoless
Template
1
0
Fork 0
Code Issues 1 Pull requests 18 Projects Releases Packages Wiki Activity Actions

Lane: let the project define its own batch size ([lane] in tf.toml) #181

Open
triform-admin wants to merge 8 commits from agent/lane-project-batch-size-v2 into main
pull from: agent/lane-project-batch-size-v2
merge into: triform:main
Conversation 1 Commits 8 Files changed 11 +934 -21
triform-admin commented 2026-08-31 18:28:27 +00:00
Owner
Copy link

Supersedes #179, rebased onto current main. (#179 branched from agent/lane-restart-continuity; the lane subsystem then landed on main by a different route, with a redesign that conflicts structurally — see "What the rebase dropped" below.)

The build lane batches at most 10 members per cycle and nothing can change it. LaneConfig::default() hardcodes max_members: 10, and with_lane builds its LaneState via LaneState::new — which is with_config(root, LaneConfig::default()). So the knob is documented as tunable in two docs and settable from nowhere.

This was mis-diagnosed once: a shell script's TRAIN_MAX_MEMBERS was raised 10→20, found inert 8 minutes later, and reverted. That post-mortem concluded "the real limiter is inside the cargoless binary" without locating it. It is lane.rs. The other visible knob, CARGOLESS_BATCH_MAX_MEMBERS (default 40), governs the check coalescer — a different subsystem sharing a field name, which is what made the earlier search stall.

What lands

[lane]
max_members = 25
capture_window_ticks = 300

Five settings, layered default < tf.toml < CARGOLESS_LANE_* env, mirroring the existing Telemetry* config family. Defaults are unchanged, so a project that sets nothing behaves exactly as before.

Two things worth reviewer attention

Validation is deliberately non-uniform. Four settings refuse 0, each for a different reason — max_members = 0 is not a deadlock but an unbounded loop of real release builds with an empty roster (the queue-empty early return makes len >= 0 unconditionally true, so take = 0 drains nobody while the phase still flips to Building). But capture_window_ticks = 0 is the documented build-immediately mode every unit test and single-developer repo uses, and must stay legal. A tidy all-fields-positive validator would break exactly that.

The CLI reader had to be fixed first (commit 1, independently valuable). apply_tf_toml hard-errors on any section outside [project]/[cache], so [lane] in a repo's tf.toml would make cargoless check/watch/build fail. Already latent for [fleet]/[telemetry], escaping only because Cmd::Serve early-returns before that reader runs.

What the rebase dropped

#179 carried a commit fixing load_active_build, which called Self::new(root) internally and so silently reverted a recovered lane to built-in defaults. Main has since replaced journal-based recovery with trail reconstruction (close_abandoned_generations) and that function no longer exists, so the fix is moot — the bug it addressed cannot occur in the new model. Everything else carried over.

Observability

A build-lane-policy boot line carries every value with a _src= tag (so "the tf.toml was read" is distinguishable from "the default coincided"), plus a policy block on GET /lane. There's also an advisory when max_members is raised against a defaulted capture_window_ticks: the lane only builds early once the queue reaches the cap, so raising it alone does nothing on sparse arrivals — and concluding "the knob is broken" is precisely how the last investigation went.

Verification

Not compiled locally — cargo is hook-blocked per CLAUDE.md, so CI is the first build. Every touched file passes rustfmt --check. The predecessor #179 went fully green (build/test/clippy/fmt/lane-mutation) on the same code against its own base, including one round where CI correctly caught a bad test of mine.

crates/cargoless-core/tests/lane_settings.rs is the load-bearing test: it writes max_members = 3 to a real tf.toml and asserts three members ride the build and two wait. If any seam regresses to LaneConfig::default(), all five ride one build and it goes red.

🤖 Generated with Claude Code

Supersedes #179, rebased onto current main. (#179 branched from `agent/lane-restart-continuity`; the lane subsystem then landed on main by a different route, with a redesign that conflicts structurally — see "What the rebase dropped" below.) The build lane batches at most 10 members per cycle and nothing can change it. `LaneConfig::default()` hardcodes `max_members: 10`, and `with_lane` builds its `LaneState` via `LaneState::new` — which is `with_config(root, LaneConfig::default())`. So the knob is documented as tunable in two docs and settable from nowhere. This was mis-diagnosed once: a shell script's `TRAIN_MAX_MEMBERS` was raised 10→20, found inert 8 minutes later, and reverted. That post-mortem concluded "the real limiter is inside the cargoless binary" without locating it. It is `lane.rs`. The other visible knob, `CARGOLESS_BATCH_MAX_MEMBERS` (default 40), governs the **check coalescer** — a different subsystem sharing a field name, which is what made the earlier search stall. ### What lands ```toml [lane] max_members = 25 capture_window_ticks = 300 ``` Five settings, layered `default < tf.toml < CARGOLESS_LANE_* env`, mirroring the existing `Telemetry*` config family. **Defaults are unchanged**, so a project that sets nothing behaves exactly as before. ### Two things worth reviewer attention **Validation is deliberately non-uniform.** Four settings refuse `0`, each for a different reason — `max_members = 0` is not a deadlock but an unbounded loop of real release builds with an empty roster (the queue-empty early return makes `len >= 0` unconditionally true, so `take = 0` drains nobody while the phase still flips to `Building`). But `capture_window_ticks = 0` is the documented build-immediately mode every unit test and single-developer repo uses, and must stay legal. A tidy all-fields-positive validator would break exactly that. **The CLI reader had to be fixed first** (commit 1, independently valuable). `apply_tf_toml` hard-errors on any section outside `[project]`/`[cache]`, so `[lane]` in a repo's tf.toml would make `cargoless check`/`watch`/`build` fail. Already latent for `[fleet]`/`[telemetry]`, escaping only because `Cmd::Serve` early-returns before that reader runs. ### What the rebase dropped #179 carried a commit fixing `load_active_build`, which called `Self::new(root)` internally and so silently reverted a recovered lane to built-in defaults. Main has since replaced journal-based recovery with trail reconstruction (`close_abandoned_generations`) and that function no longer exists, so the fix is moot — the bug it addressed cannot occur in the new model. Everything else carried over. ### Observability A `build-lane-policy` boot line carries every value with a `_src=` tag (so "the tf.toml was read" is distinguishable from "the default coincided"), plus a `policy` block on `GET /lane`. There's also an advisory when `max_members` is raised against a defaulted `capture_window_ticks`: the lane only builds early once the queue *reaches* the cap, so raising it alone does nothing on sparse arrivals — and concluding "the knob is broken" is precisely how the last investigation went. ### Verification Not compiled locally — `cargo` is hook-blocked per CLAUDE.md, so CI is the first build. Every touched file passes `rustfmt --check`. The predecessor #179 went fully green (build/test/clippy/fmt/lane-mutation) on the same code against its own base, including one round where CI correctly caught a bad test of mine. `crates/cargoless-core/tests/lane_settings.rs` is the load-bearing test: it writes `max_members = 3` to a real tf.toml and asserts three members ride the build and two wait. If any seam regresses to `LaneConfig::default()`, all five ride one build and it goes red. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
`apply_tf_toml` hard-errors on any section outside `[project]`/`[cache]`.
The daemon's own resolver (cargoless-core::config) owns `[fleet]`,
`[telemetry]` and — as of the next commit — `[lane]`, over the SAME shared
tf.toml. So a repo that configures `serve` made `cargoless check` / `watch`
/ `build` fail with "unknown section `[fleet]`".

That has been latent, not harmless: it escaped only because `Cmd::Serve`
early-returns in main.rs BEFORE this reader runs. Any reorder of that
dispatch reintroduces it. `[lane]` would have hit it immediately, because a
lane knob lives in the repo people run `cargoless check` in.

Skipping the section header alone is not enough — the key loop would then
reject `max_members` through its `_` arm — so this tracks a `foreign` flag
and skips the keys too.

The two-reader contract is preserved, not dissolved: this reader still owns
and typo-checks `[project]`/`[cache]`, and a section nobody owns (`[serve]`)
is still a hard error. Both are asserted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 150e4bda27)
The build lane's batch size was unreachable. `with_lane` builds its
`LaneState` with `LaneState::new`, which is `with_config(root,
LaneConfig::default())`, so `max_members` was pinned at 10 in production —
documented as a tunable knob in two docs and settable from nowhere. That
cost one investigation already: a shell-script constant was raised 10->20,
found inert, and reverted without locating the real bound.

Adds a `LaneSettings` family mirroring the `Telemetry*` trio: its own
provenance struct, its own resolver, its own tolerant tf.toml scanner
reusing the shared strip_comment/unquote. Ladder is `default < tf.toml <
env`; there is deliberately no CLI layer, because no `--lane-*` flag exists
and an always-default injection struct is a knob whose effect is invisible.

It WRAPS `lane::LaneConfig` rather than mirroring its five fields: that type
is what `with_config` takes and it carries the production forensics behind
each default, so duplicating the integers would mean two places to change
and a conversion that can drift.

Validation is deliberately NOT a uniform "must be > 0". Four zeros each
wedge the lane — `max_members = 0` dispatches empty-roster builds forever
(the queue-empty early return makes `len >= 0` unconditionally true, so
`take = 0` drains nobody while the phase still flips to Building);
`infra_max_attempts = 0` ejects the whole roster on the first transient
failure; `infra_backoff_ticks = 0` is the documented hot loop;
`eject_ttl_ticks = 0` lapses an ejection on the next tick. But
`capture_window_ticks = 0` is the documented build-immediately mode that
every unit test and single-developer repo uses, and must stay legal.

`parse_lane_num` is generic over usize/u64/u32 (no lossy casts) and reports
the key that actually failed — `parse_sampler_arg` hardcodes its key, which
would name the wrong field for five numeric settings. It returns a typed
error rather than falling back like the binary's `configured_batch_*`
helpers: letting `max_members = "fourty"` boot silently as 10 is the exact
invisible-knob failure this removes.

No caller yet, so nothing behaves differently. Also corrects the module doc,
which claimed the CLI reader hard-errors on ANY unknown section — no longer
true after the previous commit — and fills in the `[telemetry]` keys the
BadTfToml inventory string had been missing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 9edd6f8812)
`LaneState` had no way to report its own config, so the effective policy
could only be inferred from behaviour. The boot line and `GET /lane` both
need it, and `max_members` spent the lane's whole life documented as tunable
while being reachable from nowhere — exactly the invisible-knob shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This is where the knob goes live. `servedrv` resolves `[lane]` from the
repo's own tf.toml (plus CARGOLESS_LANE_* env) and passes it to `with_lane`,
which builds the lane with `with_config` instead of `LaneState::new` — the
latter being `with_config(root, LaneConfig::default())`, which is why
`max_members` was pinned at 10 in production while two docs described it as
tunable.

Resolution fails CLOSED, unlike telemetry, which logs and continues with
defaults because it must never wedge the daemon. This one can move the
trunk, and the failure being removed here is precisely a lane running a
policy nobody chose.

The resolved policy is announced on its own `build-lane-policy` line rather
than appended to `build-lane enabled` — that line's `where=`/`land=` fields
are the load-bearing security read and its shape should stay stable. Every
value carries a `_src=` tag, because "the tf.toml was read" and "the default
merely coincided" are otherwise indistinguishable.

It also emits a pairing advisory when `max_members` is raised while
`capture_window_ticks` is still defaulted. The lane builds early only once
the queue REACHES max_members and otherwise waits out the window, so a
bigger cap alone changes nothing on sparse arrivals. Someone who sets 40 and
sees batches of three has hit the pairing, not a bug — and letting them
conclude the knob is broken is exactly how the previous batch-size
investigation went.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The seam this covers was broken for the entire life of the lane, silently:
`with_lane` built its `LaneState` with `LaneState::new`, so `max_members`
stayed at the built-in 10 no matter what anyone configured — while being
documented as tunable in two places the whole time.

`a_tf_toml_batch_size_actually_bounds_a_build` writes `max_members = 3` to a
real tf.toml, resolves it, and enqueues five members: three ride the build,
two wait. If any seam between the file and the machine regresses to
`LaneConfig::default()`, all five ride one build and this goes red.

Also pins the one legal zero — `capture_window_ticks = 0` must survive
validation AND reach the machine — and that `max_members = 0` is refused
before it can reach a lane at all.

Kept out of `lane_policy.rs`, whose module doc stakes it on being a pure
state machine pinned "without launching a compiler"; these touch the
filesystem.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The boot line announces the resolved policy, but it dies with the pod.
`GET /lane` is what an operator polls at 3am, and it is the only way to
confirm from outside that the lane is running the policy the project
declared rather than the built-in default.

`LanePolicyView` is nested rather than flattened into LaneSnapshot's live
state fields: this is configuration, not state. Provenance deliberately does
not come along — `LaneConfig` carries none, and the boot log already answers
which layer won.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs(lane): document the [lane] section
Some checks failed
ci / build (pull_request) Successful in 1m7s
Details
ci / test (pull_request) Successful in 1m18s
Details
ci / fmt (pull_request) Successful in 38s
Details
ci / clippy (pull_request) Failing after 51s
Details
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / lane policy mutation proof (pull_request) Successful in 3m16s
Details
ba64a1c29c 
Both lane docs already listed `max_members` / `capture_window_ticks` /
`eject_ttl_ticks` as tunable knobs — one of them with "raise it when builds
are cheap and reds are rare" — while nothing could actually set them. They
now say where each setting is set, cover the two infra knobs that were
missing entirely, and carry a worked `[lane]` block.

Three things get said out loud because each has already cost someone time:
that `max_members` and `capture_window_ticks` are a pair and raising the cap
alone does nothing on sparse arrivals; that zero is refused for every knob
except `capture_window_ticks`, and why each rejection differs; and that this
`max_members` is not the check coalescer's identically-named field, which is
precisely what derailed the previous batch-size investigation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(serve): satisfy clippy on the widened with_lane
All checks were successful
ci / build (pull_request) Successful in 1m5s
Details
ci / test (pull_request) Successful in 1m6s
Details
ci / fmt (pull_request) Successful in 39s
Details
ci / clippy (pull_request) Successful in 43s
Details
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / lane policy mutation proof (pull_request) Successful in 3m27s
Details
5a8a00e026 
`clippy` was the only red job — build, test, fmt and the lane mutation proof
all passed — which is the documented signature of a lint the other jobs
cannot see.

Two of them. The lane policy parameter pushed `with_lane` to eight
arguments, one past the threshold; every one is a distinct boot-time
decision an operator makes independently, so bundling them into a struct
would trade an honest signature for an indirection with the same arity.
`#[allow]` with a reason, as this crate already does in five places.

The other: `lane_settings.lane` was cloned into the call while being its
last use. The provenance borrows end before it, so the value can simply
move.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
triform-admin force-pushed agent/lane-project-batch-size-v2 from 5a8a00e026
All checks were successful
ci / build (pull_request) Successful in 1m5s
Details
ci / test (pull_request) Successful in 1m6s
Details
ci / fmt (pull_request) Successful in 39s
Details
ci / clippy (pull_request) Successful in 43s
Details
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / lane policy mutation proof (pull_request) Successful in 3m27s
Details
to 253deb178d
All checks were successful
ci / build (pull_request) Successful in 55s
Details
ci / test (pull_request) Successful in 1m10s
Details
ci / fmt (pull_request) Successful in 43s
Details
ci / clippy (pull_request) Successful in 50s
Details
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / lane policy mutation proof (pull_request) Successful in 3m18s
Details
2026-08-31 20:13:17 +00:00 Compare
triform-admin commented 2026-08-31 20:23:18 +00:00
Author
Owner
Copy link

Green and ready to merge — but the merge API is stuck; please merge from the web UI.

CI on 253deb1, all six jobs:

job
build success
test success
clippy success
fmt success
lane policy mutation proof success
rust-analyzer latency harness skipped (as on every run)

Why it is not merged. The API reports mergeable: false with merge_base pinned at a9f970d — the tip this branch was originally cut from, now several commits stale — and POST /merge returns 405 "Please try again later". Verified independently in a clean clone that git merge-tree origin/main <branch> exits 0 with zero conflicts, so the merge really is clean; Forgejo's cached mergeability just never recomputed.

Attempted, in order, none of which moved merge_base: polling, close/reopen, rebase onto current main + force-push (new head), and retrying after CI went green.

Not pushing to main directly — that would bypass this PR and the repo's never-push-to-main rule over what is a UI cache bug. If the web Merge button also refuses, the instance likely needs a restart or forgejo doctor; the branch itself is fine.

On the rebase: pure replay. Byte-comparing the diffs before and after shows only blob hashes and line offsets differ — identical content, same 934 insertions / 21 deletions across the same 11 files. Main's three newer lane commits do not touch the seams this change depends on, and the one LaneConfig literal they add uses ..Default::default(), so the added fields do not break it.

**Green and ready to merge — but the merge API is stuck; please merge from the web UI.** CI on `253deb1`, all six jobs: | job | | |---|---| | `build` | success | | `test` | success | | `clippy` | success | | `fmt` | success | | `lane policy mutation proof` | success | | rust-analyzer latency harness | skipped (as on every run) | **Why it is not merged.** The API reports `mergeable: false` with `merge_base` pinned at `a9f970d` — the tip this branch was originally cut from, now several commits stale — and `POST /merge` returns `405 "Please try again later"`. Verified independently in a clean clone that `git merge-tree origin/main <branch>` exits 0 with **zero conflicts**, so the merge really is clean; Forgejo's cached mergeability just never recomputed. Attempted, in order, none of which moved `merge_base`: polling, close/reopen, rebase onto current main + force-push (new head), and retrying after CI went green. Not pushing to `main` directly — that would bypass this PR and the repo's never-push-to-main rule over what is a UI cache bug. If the web Merge button also refuses, the instance likely needs a restart or `forgejo doctor`; the branch itself is fine. **On the rebase:** pure replay. Byte-comparing the diffs before and after shows only blob hashes and line offsets differ — identical content, same 934 insertions / 21 deletions across the same 11 files. Main's three newer lane commits do not touch the seams this change depends on, and the one `LaneConfig` literal they add uses `..Default::default()`, so the added fields do not break it.
All checks were successful
ci / build (pull_request) Successful in 55s
Details
ci / test (pull_request) Successful in 1m10s
Details
ci / fmt (pull_request) Successful in 43s
Details
ci / clippy (pull_request) Successful in 50s
Details
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / lane policy mutation proof (pull_request) Successful in 3m18s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin agent/lane-project-batch-size-v2:agent/lane-project-batch-size-v2
git switch agent/lane-project-batch-size-v2

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff agent/lane-project-batch-size-v2
git switch agent/lane-project-batch-size-v2
git rebase main
git switch main
git merge --ff-only agent/lane-project-batch-size-v2
git switch agent/lane-project-batch-size-v2
git rebase main
git switch main
git merge --no-ff agent/lane-project-batch-size-v2
git switch main
git merge --squash agent/lane-project-batch-size-v2
git switch main
git merge --ff-only agent/lane-project-batch-size-v2
git switch main
git merge agent/lane-project-batch-size-v2
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
triform/cargoless!181
No description provided.