Lane: let the project define its own batch size ([lane] in tf.toml)

triform-admin commented

2026-08-28 04:51:30 +00:00

Owner

The build lane batched at most 10 members per cycle and nothing could change it. LaneConfig::default() hardcodes max_members: 10, and with_lane always built its LaneState via LaneState::new — so the knob was documented as tunable in two docs and settable from nowhere.

This was mis-diagnosed once before: a shell script's TRAIN_MAX_MEMBERS was raised 10→20, found inert 8 minutes later, and reverted. That post-mortem concluded "the real limiter is inside the cargoless binary" without locating it. It is lane.rs. The other visible knob, CARGOLESS_BATCH_MAX_MEMBERS (default 40), governs the check coalescer — a different subsystem that happens to share a field name, which is what made the earlier search stall.

What lands

A project declares its lane policy in its own tf.toml, versioned alongside the code whose build cost it describes:

[lane]
max_members = 25
capture_window_ticks = 300

Five settings, layered default < tf.toml < CARGOLESS_LANE_* env, mirroring the existing Telemetry* config family. Defaults are unchanged, so a project that sets nothing behaves exactly as before.

Three things worth reviewer attention

The restart path was silently reverting. load_active_build called Self::new(root) internally, so a lane that recovered an active generation dropped back to the built-in policy — invisible until someone counted members. It now takes the policy as a required parameter, making the revert unrepresentable. The existing recovery test already used a non-default config, so it now asserts the config survived.

Validation is deliberately non-uniform. Four settings refuse 0, each for a different reason — max_members = 0 is not a deadlock but an unbounded loop of real release builds with an empty roster (the queue-empty early return makes len >= 0 unconditionally true, so take = 0 drains nobody while the phase still flips to Building). But capture_window_ticks = 0 is the documented build-immediately mode every unit test and single-developer repo uses, and must stay legal. A tidy all-fields-positive validator would break exactly that.

The CLI reader had to be fixed first (commit 1, independently valuable). apply_tf_toml hard-errors on any section outside [project]/[cache], so [lane] in a repo's tf.toml would make cargoless check/watch/build fail. That was already latent for [fleet]/[telemetry] and escaped only because Cmd::Serve early-returns before that reader runs.

Observability

A build-lane-policy boot line carries every value with a _src= tag (so "the tf.toml was read" is distinguishable from "the default coincided"), plus a policy block on GET /lane — the only way to confirm a recovered lane kept its policy. There's also an advisory when max_members is raised against a defaulted capture_window_ticks, because the lane only builds early once the queue reaches the cap: raising it alone does nothing on sparse arrivals, and concluding "the knob is broken" is precisely how the last investigation went.

Verification

Not compiled locally — cargo is hook-blocked per CLAUDE.md, so this PR is the first build. Every touched file was rustfmt-ed. crates/cargoless-core/tests/lane_settings.rs is the load-bearing test: it writes max_members = 3 to a real tf.toml and asserts three members ride the build and two wait. If any seam regresses to LaneConfig::default(), all five ride one build and it goes red.

🤖 Generated with Claude Code

The build lane batched at most 10 members per cycle and nothing could change it. `LaneConfig::default()` hardcodes `max_members: 10`, and `with_lane` always built its `LaneState` via `LaneState::new` — so the knob was documented as tunable in two docs and settable from nowhere. This was mis-diagnosed once before: a shell script's `TRAIN_MAX_MEMBERS` was raised 10→20, found inert 8 minutes later, and reverted. That post-mortem concluded "the real limiter is inside the cargoless binary" without locating it. It is `lane.rs`. The other visible knob, `CARGOLESS_BATCH_MAX_MEMBERS` (default 40), governs the **check coalescer** — a different subsystem that happens to share a field name, which is what made the earlier search stall. ### What lands A project declares its lane policy in its own `tf.toml`, versioned alongside the code whose build cost it describes: ```toml [lane] max_members = 25 capture_window_ticks = 300 ``` Five settings, layered `default < tf.toml < CARGOLESS_LANE_* env`, mirroring the existing `Telemetry*` config family. **Defaults are unchanged**, so a project that sets nothing behaves exactly as before. ### Three things worth reviewer attention **The restart path was silently reverting.** `load_active_build` called `Self::new(root)` internally, so a lane that recovered an active generation dropped back to the built-in policy — invisible until someone counted members. It now takes the policy as a *required* parameter, making the revert unrepresentable. The existing recovery test already used a non-default config, so it now asserts the config survived. **Validation is deliberately non-uniform.** Four settings refuse `0`, each for a different reason — `max_members = 0` is not a deadlock but an unbounded loop of real release builds with an empty roster (the queue-empty early return makes `len >= 0` unconditionally true, so `take = 0` drains nobody while the phase still flips to `Building`). But `capture_window_ticks = 0` is the documented build-immediately mode every unit test and single-developer repo uses, and must stay legal. A tidy all-fields-positive validator would break exactly that. **The CLI reader had to be fixed first** (commit 1, independently valuable). `apply_tf_toml` hard-errors on any section outside `[project]`/`[cache]`, so `[lane]` in a repo's tf.toml would make `cargoless check`/`watch`/`build` fail. That was already latent for `[fleet]`/`[telemetry]` and escaped only because `Cmd::Serve` early-returns before that reader runs. ### Observability A `build-lane-policy` boot line carries every value with a `_src=` tag (so "the tf.toml was read" is distinguishable from "the default coincided"), plus a `policy` block on `GET /lane` — the only way to confirm a *recovered* lane kept its policy. There's also an advisory when `max_members` is raised against a defaulted `capture_window_ticks`, because the lane only builds early once the queue *reaches* the cap: raising it alone does nothing on sparse arrivals, and concluding "the knob is broken" is precisely how the last investigation went. ### Verification Not compiled locally — `cargo` is hook-blocked per CLAUDE.md, so this PR **is** the first build. Every touched file was `rustfmt`-ed. `crates/cargoless-core/tests/lane_settings.rs` is the load-bearing test: it writes `max_members = 3` to a real tf.toml and asserts three members ride the build and two wait. If any seam regresses to `LaneConfig::default()`, all five ride one build and it goes red. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

triform-admin added 10 commits

2026-08-28 04:51:31 +00:00

fix(lane): recover active generation after restart

ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped

Details

ci / lane policy mutation proof (pull_request) Failing after 1m0s

Details

ci / test (pull_request) Failing after 5m10s

Details

ci / clippy (pull_request) Failing after 5m15s

Details

ci / build (pull_request) Failing after 8m57s

Details

ci / fmt (pull_request) Successful in 14m1s

Details

647fab6a4b

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

fix(lane): require durable generation before dispatch

ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped

Details

ci / lane policy mutation proof (pull_request) Failing after 1m57s

Details

ci / build (pull_request) Failing after 3m36s

Details

ci / fmt (pull_request) Successful in 3m31s

Details

ci / test (pull_request) Failing after 3m36s

Details

ci / clippy (pull_request) Failing after 10m38s

Details

5cb6e23cae

fix(lane): keep recovery journal MSRV-compatible

ci / clippy (pull_request) Failing after 19s

Details

ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped

Details

ci / lane policy mutation proof (pull_request) Successful in 9m45s

Details

ci / fmt (pull_request) Successful in 10m7s

Details

ci / test (pull_request) Successful in 10m17s

Details

ci / build (pull_request) Successful in 15m37s

Details

5bcdff956f

fix(config): stop the CLI reader rejecting daemon-owned tf.toml sections 150e4bda27

`apply_tf_toml` hard-errors on any section outside `[project]`/`[cache]`.
The daemon's own resolver (cargoless-core::config) owns `[fleet]`,
`[telemetry]` and — as of the next commit — `[lane]`, over the SAME shared
tf.toml. So a repo that configures `serve` made `cargoless check` / `watch`
/ `build` fail with "unknown section `[fleet]`".

That has been latent, not harmless: it escaped only because `Cmd::Serve`
early-returns in main.rs BEFORE this reader runs. Any reorder of that
dispatch reintroduces it. `[lane]` would have hit it immediately, because a
lane knob lives in the repo people run `cargoless check` in.

Skipping the section header alone is not enough — the key loop would then
reject `max_members` through its `_` arm — so this tracks a `foreign` flag
and skips the keys too.

The two-reader contract is preserved, not dissolved: this reader still owns
and typo-checks `[project]`/`[cache]`, and a section nobody owns (`[serve]`)
is still a hard error. Both are asserted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(config): resolve the lane's policy from [lane] in tf.toml 9edd6f8812

The build lane's batch size was unreachable. `with_lane` builds its
`LaneState` with `LaneState::new`, which is `with_config(root,
LaneConfig::default())`, so `max_members` was pinned at 10 in production —
documented as a tunable knob in two docs and settable from nowhere. That
cost one investigation already: a shell-script constant was raised 10->20,
found inert, and reverted without locating the real bound.

Adds a `LaneSettings` family mirroring the `Telemetry*` trio: its own
provenance struct, its own resolver, its own tolerant tf.toml scanner
reusing the shared strip_comment/unquote. Ladder is `default < tf.toml <
env`; there is deliberately no CLI layer, because no `--lane-*` flag exists
and an always-default injection struct is a knob whose effect is invisible.

It WRAPS `lane::LaneConfig` rather than mirroring its five fields: that type
is what `with_config` takes and it carries the production forensics behind
each default, so duplicating the integers would mean two places to change
and a conversion that can drift.

Validation is deliberately NOT a uniform "must be > 0". Four zeros each
wedge the lane — `max_members = 0` dispatches empty-roster builds forever
(the queue-empty early return makes `len >= 0` unconditionally true, so
`take = 0` drains nobody while the phase still flips to Building);
`infra_max_attempts = 0` ejects the whole roster on the first transient
failure; `infra_backoff_ticks = 0` is the documented hot loop;
`eject_ttl_ticks = 0` lapses an ejection on the next tick. But
`capture_window_ticks = 0` is the documented build-immediately mode that
every unit test and single-developer repo uses, and must stay legal.

`parse_lane_num` is generic over usize/u64/u32 (no lossy casts) and reports
the key that actually failed — `parse_sampler_arg` hardcodes its key, which
would name the wrong field for five numeric settings. It returns a typed
error rather than falling back like the binary's `configured_batch_*`
helpers: letting `max_members = "fourty"` boot silently as 10 is the exact
invisible-knob failure this removes.

No caller yet, so nothing behaves differently. Also corrects the module doc,
which claimed the CLI reader hard-errors on ANY unknown section — no longer
true after the previous commit — and fills in the `[telemetry]` keys the
BadTfToml inventory string had been missing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(lane): make the lane's policy reachable from its owner 008d55b175

`load_active_build` called `Self::new(root)` internally, which is
`LaneConfig::default()`. So even once a policy is resolvable, a lane that
recovered an active generation after a restart would silently revert to the
built-in defaults — invisible until someone counted members in a build.

It now takes the resolved policy as a REQUIRED parameter rather than a
defaulted one. With three call sites (one production, two tests) the churn
is trivial, and a required parameter makes the revert unrepresentable
instead of merely discouraged — the same move `with_lane` already makes for
the lander/plan pairing.

The existing recovery test already built its interrupted lane with a
non-default `capture_window_ticks`, so it now asserts the recovered lane
kept it. That turns a mechanical signature fix into a regression test for
the exact seam that discarded config.

Also adds a `cfg()` accessor — `LaneState` had none, so the effective policy
could not be reported anywhere — and documents why the policy is NOT
journalled: config belongs to the live process, and a restart after an
operator raises `max_members` must run under the new value rather than
resurrect the one the dead pod booted with. The recovery schema stays at 1;
nothing about the durable shape changed.

`serveapi` passes `LaneConfig::default()` here, so this commit is
compile-only and behaviour-preserving. The next one hands it the project's
actual policy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(serve): hand the project's lane policy to the lane 9a3d008df4

This is where the knob goes live. `servedrv` resolves `[lane]` from the
repo's own tf.toml (plus CARGOLESS_LANE_* env) and passes it to `with_lane`,
which hands it to BOTH construction arms — the fresh lane and the recovered
one — so a restart is not a silent way back to the built-in defaults.

Resolution fails CLOSED, unlike telemetry, which logs and continues with
defaults because it must never wedge the daemon. This one can move the
trunk, and the failure being removed here is precisely a lane running a
policy nobody chose. The message names the two ways an out-of-range policy
bites and where to fix it.

The resolved policy is announced on its own `build-lane-policy` line rather
than appended to `build-lane enabled` — that line's `where=`/`land=` fields
are the load-bearing security read and its shape should stay stable. Every
value carries a `_src=` tag, because "the tf.toml was read" and "the default
merely coincided" are otherwise indistinguishable, and a knob that silently
failed to apply is what this replaced.

It also emits a pairing advisory when `max_members` is raised while
`capture_window_ticks` is still defaulted. The lane builds early only once
the queue REACHES max_members and otherwise waits out the window, so a
bigger cap alone changes nothing on sparse arrivals. Someone who sets 40 and
sees batches of three has hit the pairing, not a bug — and letting them
conclude the knob is broken is exactly how the previous batch-size
investigation went.

`with_lane` now takes 7 parameters including `self` — clippy's
`too_many_arguments` threshold. It does not fire at 7; the next one will.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

test(lane): prove a tf.toml batch size reaches the build a04c90ea8e

The seam this covers was broken for the entire life of the lane, silently:
`with_lane` built its `LaneState` with `LaneState::new`, so `max_members`
stayed at the built-in 10 no matter what anyone configured — while being
documented as tunable in two places the whole time.

`a_tf_toml_batch_size_actually_bounds_a_build` writes `max_members = 3` to a
real tf.toml, resolves it, and enqueues five members: three ride the build,
two wait. If any seam between the file and the machine regresses to
`LaneConfig::default()`, all five ride one build and this goes red.

Also pins the one legal zero — `capture_window_ticks = 0` must survive
validation AND reach the machine, because a validator that rejected it
breaks the documented single-developer mode and one that silently replaced
it makes a lone developer wait a minute per build — and pins that
`max_members = 0` is refused before it can reach a lane at all.

Kept out of `lane_policy.rs`, whose module doc stakes it on being a pure
state machine pinned "without launching a compiler"; these touch the
filesystem.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(lane): report the effective policy on GET /lane 5c99bafd28

The boot line announces the resolved policy, but it dies with the pod.
`GET /lane` is what an operator polls at 3am — and it is the only way to
confirm that a lane which RECOVERED an active build kept the configured
policy rather than reverting to the built-in one. That revert was silent
until two commits ago; leaving it unobservable would make the fix
unverifiable in production.

`LanePolicyView` is nested rather than flattened into LaneSnapshot's live
state fields: this is configuration, not state. Provenance deliberately does
not come along — `LaneConfig` carries none, and the boot log already answers
which layer won.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs(lane): document the [lane] section

ci / build (pull_request) Successful in 59s

Details

ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped

Details

ci / clippy (pull_request) Successful in 43s

Details

ci / fmt (pull_request) Successful in 3m44s

Details

ci / test (pull_request) Failing after 5m49s

Details

ci / lane policy mutation proof (pull_request) Successful in 6m10s

Details

4d8530bcec

Both lane docs already listed `max_members` / `capture_window_ticks` /
`eject_ttl_ticks` as tunable knobs — one of them with "raise it when builds
are cheap and reds are rare" — while nothing could actually set them. They
now say where each setting is set, cover the two infra knobs that were
missing entirely, and carry a worked `[lane]` block.

Three things get said out loud because each has already cost someone time:
that `max_members` and `capture_window_ticks` are a pair and raising the cap
alone does nothing on sparse arrivals; that zero is refused for every knob
except `capture_window_ticks`, and why each rejection differs; and that this
`max_members` is not the check coalescer's identically-named field, which is
precisely what derailed the previous batch-size investigation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

triform-admin added 1 commit

2026-08-28 05:26:50 +00:00

fix(test): the batch-size proof set a zero capture window and proved nothing

ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped

Details

ci / clippy (pull_request) Successful in 1m35s

Details

ci / fmt (pull_request) Successful in 4m3s

Details

ci / test (pull_request) Successful in 5m6s

Details

ci / lane policy mutation proof (pull_request) Successful in 6m15s

Details

ci / build (pull_request) Successful in 7m32s

Details

14cea5efbb

`a_tf_toml_batch_size_actually_bounds_a_build` set `capture_window_ticks = 0`
alongside `max_members = 3`. A zero window builds the first arrival on its
own, so the lane started a build with A alone and the cap was never reached:
`in_flight` was 1, not 3.

The window has to be OPEN for a cap test. A and B wait in it, C fills the
queue to the cap, and the build short-circuits the window because there is
nothing left to gather — which is the same shape `max_members_bounds_a_build`
in lane_policy.rs already uses, and why that one passes.

The zero window was copied from the sibling test below, where it is correct:
that one asserts build-immediately, not a bound.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

triform-admin referenced this pull request

2026-08-31 18:28:27 +00:00

Lane: let the project define its own batch size ([lane] in tf.toml) #181

triform-admin commented

2026-08-31 18:28:39 +00:00

Author

Owner

Superseded by #181, which is the same change rebased onto current main.

This branch was cut from agent/lane-restart-continuity. The lane subsystem has since landed on main by a different route, with a redesign that conflicts structurally — load_active_build is gone (recovery now reconstructs from the trail via close_abandoned_generations), spawn_persisted became spawn_with_intergeneration_yield, and with_lane took its own 7th parameter. Rebasing was cleaner than resolving those in place.

Closing in favour of #181.

Superseded by #181, which is the same change rebased onto current main. This branch was cut from `agent/lane-restart-continuity`. The lane subsystem has since landed on main by a different route, with a redesign that conflicts structurally — `load_active_build` is gone (recovery now reconstructs from the trail via `close_abandoned_generations`), `spawn_persisted` became `spawn_with_intergeneration_yield`, and `with_lane` took its own 7th parameter. Rebasing was cleaner than resolving those in place. Closing in favour of #181.

triform-admin closed this pull request

2026-08-31 18:28:41 +00:00

ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped

Details

ci / clippy (pull_request) Successful in 1m35s

Details

ci / fmt (pull_request) Successful in 4m3s

Details

ci / test (pull_request) Successful in 5m6s

Details

ci / lane policy mutation proof (pull_request) Successful in 6m15s

Details

ci / build (pull_request) Successful in 7m32s

Details

Pull request closed

Please reopen this pull request to perform a merge.

Rows
Columns

Lane: let the project define its own batch size ([lane] in tf.toml) #179

What lands

Three things worth reviewer attention

Observability

Verification

Pull request closed