triform/cargoless
Template
1
0
Fork 0
Code Issues 1 Pull requests 12 Projects Releases Packages Wiki Activity Actions 7

fix(appsvc): one dead preview slot held /readyz at 503 forever #113

Open
iggy wants to merge 3 commits from agent/readyz-liveness into agent/lane-stages
pull from: agent/readyz-liveness
merge into: triform:agent/lane-stages
Conversation 0 Commits 3 Files changed 4 +811 -53
iggy commented 2026-08-03 07:01:24 +00:00
Contributor
Copy link

The bug

/readyz required every ever-green instance to be currently serving. On the tf-multiverse preview merge last built 2026-08-02 and feature-x 2026-06-24 — both long dead, neither on anyone's critical path — so the probe was pinned at 503 indefinitely. That is a wedged-red readiness signal: it delists the pod, and with it the healthy dev and lane (one readinessProbe, all endpoints), and it says nothing at exactly the moment you need it to mean something.

What /readyz means now

"The daemon can accept and serve work."

ready  ⟺  some instance is currently serving
      AND  no LIVE instance is failing to serve a green it owns

Live is a liveness test — a fresh observable state change within STALE_AFTER_SECS (1h) — not "this slot exists and is not green". That distinction is the whole fix, and it is the lesson from the tf-multiverse watchdog that was widened to treat any busy slot as busy and was then made permanently unable to fire by six 41-day-dead slots.

An instance on someone's critical path re-arms its own stamp simply by being used (a push moves its ref → its phase advances → that is a state change), so a broken dev people are pushing to keeps the probe red for as long as it is broken. A slot nobody has touched in weeks emits no transitions and ages out — by rule, with no per-name allowlist to keep in sync with the instance set.

Both useless probes are structurally excluded

Failure mode What prevents it
always-green (useless) "some instance is currently serving" is a fact about the present with no stamp to age. Ageing can only ever remove a veto, never manufacture a green. Nothing serving ⇒ 503 however quiet the daemon has gone.
wedged-red (equally useless) a degraded instance vetoes only while it is live.

Why the stamp is trustworthy

The driver republishes every instance on every event. An unconditional re-stamp would let a busy dev keep a dead merge eternally fresh — the same widening failure in miniature. So publish diffs each incoming row against the one it replaces and carries the old stamp forward when nothing changed. A first-seen instance is stamped now(), not 0, so a hot-added preview is not born stale.

/readyz is also no longer latched at publish time: it is computed from the snapshot and the clock at read time, because a degraded instance goes stale by time passing, not by an event arriving — and a daemon whose control loop is wedged publishes nothing at all, precisely when a latch would freeze at its last value.

Stale state never becomes the thing that reports a problem

An aged-out instance keeps its full /app row plus last_change_unix / idle_secs / stale, and is named in readiness.stale_degraded. /readyz answers "is this daemon working"; /app answers "…and here is exactly what is not".

Tests pin both directions

Forward (the fix): one live slot + one 41-day-dead slot reports ready; the real four-instance production shape reports ready with merge+feature-x aged out and both named in stale_degraded.

Inverse (not just "always green"):

  • a daemon with nothing serving is not ready however ancient (incl. the empty case);
  • an actively-transitioning broken instance never ages out, across 20 near-horizon cycles;
  • a break on a live lane still turns the probe red with two aged-out siblings present;
  • a backwards clock fails toward not-ready (saturating age ⇒ everything reads live);
  • the disarm trap itself: a dead slot's stamp survives 400 republishes driven by a busy sibling;
  • every field of the row is covered — each one individually refreshes the stamp, a byte-identical republish does not.

Grounded against the running daemon (read-only)

Probed pod cargoless-preview-9b84c69cd-mhnbq in triform-staging. It is app-serve ready=false right now, with lane genuinely broken (serving_sha: null, phase=building, last_red_reason: cannot change to .../lane/worktree: No such file or directory, no transition in 19 min). Under this rule that pod correctly stays 503 — the fix does not paper over a real fault. The staleness horizon's lower bound is cited from the manifest's own cold-build budget (startupProbe: 180 × 10s = 30 min), cleared with 2× headroom.

The one sharp edge is documented rather than hidden: an instance degraded and silent for over an hour (a cold rebuild past the 60-minute mark) ages out mid-recovery. Accepted deliberately — a build silent for an hour is indistinguishable from a hung one, and exempting busy pipelines would let a hung build veto forever, reintroducing the exact wedge this removes.

Validation

No local cargo (CI-only by design). Logic was validated against a Rust-faithful Python port before pushing: all 17 test bodies replayed statement-for-statement, which caught a u64 underflow in a test fixture. rustfmt --edition 2024 clean.

Files: crates/cargoless-core/src/appsvc.rs, plus doc/manifest comment updates in docs/PREVIEW-STATUS.md, docs/design/D-APP-SERVE.md, deploy/cargoless-appserve.k8s.yaml.

Based on agent/lane-stages (PR #99), not main.

## The bug `/readyz` required **every** ever-green instance to be currently serving. On the tf-multiverse preview `merge` last built 2026-08-02 and `feature-x` 2026-06-24 — both long dead, neither on anyone's critical path — so the probe was pinned at 503 indefinitely. That is a *wedged-red* readiness signal: it delists the pod, and with it the **healthy** `dev` and `lane` (one readinessProbe, all endpoints), and it says nothing at exactly the moment you need it to mean something. ## What `/readyz` means now **"The daemon can accept and serve work."** ``` ready ⟺ some instance is currently serving AND no LIVE instance is failing to serve a green it owns ``` *Live* is a **liveness test** — a fresh observable state change within `STALE_AFTER_SECS` (1h) — **not** "this slot exists and is not green". That distinction is the whole fix, and it is the lesson from the tf-multiverse watchdog that was widened to treat any busy slot as busy and was then made permanently unable to fire by six 41-day-dead slots. An instance on someone's critical path **re-arms its own stamp simply by being used** (a push moves its ref → its phase advances → that is a state change), so a broken `dev` people are pushing to keeps the probe red for as long as it is broken. A slot nobody has touched in weeks emits no transitions and ages out — **by rule**, with no per-name allowlist to keep in sync with the instance set. ### Both useless probes are structurally excluded | Failure mode | What prevents it | |---|---| | **always-green** (useless) | "some instance is currently serving" is a fact about the *present* with no stamp to age. Ageing can only ever **remove** a veto, never manufacture a green. Nothing serving ⇒ 503 however quiet the daemon has gone. | | **wedged-red** (equally useless) | a degraded instance vetoes only while it is *live*. | ### Why the stamp is trustworthy The driver republishes **every** instance on **every** event. An unconditional re-stamp would let a busy `dev` keep a dead `merge` eternally fresh — the same widening failure in miniature. So `publish` **diffs** each incoming row against the one it replaces and carries the old stamp forward when nothing changed. A first-seen instance is stamped `now()`, not `0`, so a hot-added preview is not born stale. `/readyz` is also **no longer latched** at publish time: it is computed from the snapshot **and the clock** at read time, because a degraded instance goes stale by time passing, not by an event arriving — and a daemon whose control loop is wedged publishes nothing at all, precisely when a latch would freeze at its last value. ### Stale state never becomes the thing that reports a problem An aged-out instance keeps its full `/app` row plus `last_change_unix` / `idle_secs` / `stale`, and is named in `readiness.stale_degraded`. `/readyz` answers "is this daemon working"; `/app` answers "…and here is exactly what is not". ## Tests pin both directions **Forward (the fix):** one live slot + one 41-day-dead slot reports ready; the real four-instance production shape reports ready with `merge`+`feature-x` aged out and both named in `stale_degraded`. **Inverse (not just "always green"):** - a daemon with nothing serving is **not** ready however ancient (incl. the empty case); - an actively-transitioning broken instance **never** ages out, across 20 near-horizon cycles; - a break on a live `lane` still turns the probe red with two aged-out siblings present; - a backwards clock fails toward **not**-ready (saturating age ⇒ everything reads live); - the disarm trap itself: a dead slot's stamp survives 400 republishes driven by a busy sibling; - every field of the row is covered — each one individually refreshes the stamp, a byte-identical republish does not. ## Grounded against the running daemon (read-only) Probed pod `cargoless-preview-9b84c69cd-mhnbq` in `triform-staging`. It is `app-serve ready=false` right now, with `lane` genuinely broken (`serving_sha: null`, `phase=building`, `last_red_reason: cannot change to .../lane/worktree: No such file or directory`, no transition in 19 min). Under this rule that pod **correctly stays 503** — the fix does not paper over a real fault. The staleness horizon's lower bound is cited from the manifest's own cold-build budget (`startupProbe: 180 × 10s` = 30 min), cleared with 2× headroom. The one sharp edge is documented rather than hidden: an instance degraded *and* silent for over an hour (a cold rebuild past the 60-minute mark) ages out mid-recovery. Accepted deliberately — a build silent for an hour is indistinguishable from a hung one, and exempting busy pipelines would let a hung build veto forever, reintroducing the exact wedge this removes. ## Validation No local cargo (CI-only by design). Logic was validated against a Rust-faithful Python port before pushing: all 17 test bodies replayed statement-for-statement, which caught a `u64` underflow in a test fixture. `rustfmt --edition 2024` clean. Files: `crates/cargoless-core/src/appsvc.rs`, plus doc/manifest comment updates in `docs/PREVIEW-STATUS.md`, `docs/design/D-APP-SERVE.md`, `deploy/cargoless-appserve.k8s.yaml`. Based on `agent/lane-stages` (PR #99), not `main`. 
`/readyz` required EVERY ever-green instance to be currently serving. On
the tf-multiverse preview `merge` last built 2026-08-02 and `feature-x`
2026-06-24 — both long dead, neither on anyone's critical path — so the
probe was pinned at 503 indefinitely. That is a wedged-red readiness
signal: it delisted the pod, and with it the *healthy* `dev` and `lane`
(one readinessProbe, all endpoints), and it said nothing at exactly the
moment you needed it to mean something.

/readyz now means "the daemon can accept and serve work":

    ready  <=>  some instance is currently serving
           AND  no LIVE instance is failing to serve a green it owns

"Live" is a LIVENESS test — a fresh observable state change within
`STALE_AFTER_SECS` (1h) — not "this slot exists and is not green". That
distinction is the whole fix, and it is the lesson from the tf-multiverse
watchdog that was widened to treat any busy slot as busy and was then made
permanently unable to fire by six 41-day-dead slots. An instance on
someone's critical path re-arms its own stamp simply by being used (a push
moves its ref, which advances its phase), so a broken `dev` people are
pushing to keeps the probe red for as long as it is broken. A slot nobody
has touched in weeks emits no transitions and ages out — by rule, with no
per-name allowlist to keep in sync with the instance set.

Both useless probes are structurally excluded:

- NOT always-green: "some instance is currently serving" is a fact about
  the present with no stamp to age, so ageing can only ever REMOVE a veto,
  never manufacture a green. Nothing serving => 503 however quiet the
  daemon has gone.
- NOT wedged-red: a degraded instance vetoes only while live.

The stamp is trustworthy because `publish` DIFFS each incoming row against
the one it replaces and carries the old stamp forward when nothing changed.
The driver republishes every instance on every event, so an unconditional
re-stamp would let a busy `dev` keep a dead `merge` eternally fresh — the
same widening failure in miniature. A first-seen instance is stamped now(),
not 0, so a hot-added preview is not born stale.

`/readyz` is no longer latched at publish time: it is computed from the
snapshot AND the clock at read time, because a degraded instance goes stale
by time passing, not by an event arriving — and a daemon whose control loop
is wedged publishes nothing at all, precisely when a latch would freeze.

Stale state never becomes the thing that reports a problem: an aged-out
instance keeps its full `/app` row plus `last_change_unix` / `idle_secs` /
`stale`, and is named in `readiness.stale_degraded`. /readyz answers "is
this daemon working"; /app answers "...and here is exactly what is not".

Tests pin both directions. Forward: one live slot + one 41-day-dead slot
reports ready, and the real four-instance production shape reports ready
with `merge`+`feature-x` aged out. Inverse: a daemon with nothing serving
is NOT ready however ancient, an actively-transitioning broken instance
never ages out across 20 near-horizon cycles, a break on a live `lane`
still turns the probe red with two aged-out siblings present, and a
backwards clock fails toward not-ready (saturating age => everything reads
live). Plus the disarm trap itself: a dead slot's stamp survives 400
republishes driven by a busy sibling.

Validated against a Rust-faithful Python port before pushing (all 17 test
bodies replayed statement-for-statement).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two corrections after checking the rule against the running preview
(pod cargoless-preview-9b84c69cd-mhnbq, read-only):

1. The lower bound now cites the number that actually constrains it —
   the deployment's own `startupProbe` cold-build budget of 180 x 10s =
   30 minutes. One hour clears that with 2x headroom, so an instance
   grinding through a full cold rebuild after losing its child keeps
   vetoing the whole way. "A cold Leptos build is minutes" was the right
   instinct against the wrong reference point.

2. The sharp edge is no longer hypothetical. The live daemon is serving
   `/readyz` 503 right now with `lane` stuck in phase=building,
   serving_sha=null, last_red "cannot change to .../lane/worktree: No
   such file or directory" — 19 minutes without a transition. That is
   exactly the degraded-and-not-transitioning shape that ages out at the
   hour mark, so the doc says so plainly instead of calling it unlikely.
   Still accepted: a build silent for an hour is indistinguishable from a
   hung one, and exempting busy pipelines would let a hung build veto
   forever — the very wedge this design removes.

Comments only; no behaviour change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
iggy changed target branch from agent/lane-stages to main 2026-08-03 07:01:54 +00:00
iggy referenced this pull request from a commit 2026-08-03 07:02:19 +00:00
chore: trigger CI
chore: trigger CI
All checks were successful
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / lane policy mutation proof (pull_request) Successful in 9m40s
Details
ci / clippy (pull_request) Successful in 9m47s
Details
ci / fmt (pull_request) Successful in 10m16s
Details
ci / test (pull_request) Successful in 13m26s
Details
ci / build (pull_request) Successful in 13m44s
Details
9f6e4d5204 
The workflow fires on `pull_request: branches: [main]` only, so a push to
a branch (or a PR based on another agent branch) runs nothing. PR #113 was
retargeted to `main`; this empty commit produces the synchronize event that
actually starts the matrix. No content change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
iggy changed target branch from main to agent/lane-stages 2026-08-03 07:17:43 +00:00
All checks were successful
ci / rust-analyzer latency harness (S1 / AC#2) (pull_request) Has been skipped
Details
ci / lane policy mutation proof (pull_request) Successful in 9m40s
Details
ci / clippy (pull_request) Successful in 9m47s
Details
ci / fmt (pull_request) Successful in 10m16s
Details
ci / test (pull_request) Successful in 13m26s
Details
ci / build (pull_request) Successful in 13m44s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin agent/readyz-liveness:agent/readyz-liveness
git switch agent/readyz-liveness

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch agent/lane-stages
git merge --no-ff agent/readyz-liveness
git switch agent/readyz-liveness
git rebase agent/lane-stages
git switch agent/lane-stages
git merge --ff-only agent/readyz-liveness
git switch agent/readyz-liveness
git rebase agent/lane-stages
git switch agent/lane-stages
git merge --no-ff agent/readyz-liveness
git switch agent/lane-stages
git merge --squash agent/readyz-liveness
git switch agent/lane-stages
git merge --ff-only agent/readyz-liveness
git switch agent/lane-stages
git merge agent/readyz-liveness
git push origin agent/lane-stages
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
triform/cargoless!113
No description provided.