Witness daemon leaks registered scratch worktrees across pod restarts (fast-fails later runs) #87

New issue

Open

opened 2026-07-26 22:54:59 +00:00 by iggy · 0 comments

iggy commented

2026-07-26 22:54:59 +00:00

Contributor

Symptom. A gated push fast-fails in ~4s with:

project_check_batch_indeterminate: git ["worktree", "add", "--detach",
  "/workspace/cargoless-state/project-check-runs/run-1-4", "origin/dev"]
  exited 128: fatal: ... is a missing but already registered worktree;
  use 'add -f' to override, or 'prune' or 'remove' to clear

The verdict publishes as unknown — it does not wedge a PR, but it burns a gate slot and reads as an unexplained infra flake.

Cause. When the serve pod dies mid-compile (Flux roll, OOM, node eviction), the per-run scratch worktree survives on the PVC and stays registered in .git/worktrees/. The run counter resets on the new pod, so a later run eventually reuses the same run-1-N index and collides with the orphaned registration.

Why git worktree prune does NOT fix it. Prune only clears registrations whose directory is gone. These directories still exist (full of build artifacts), so prune reports nothing to do. Clearing them requires git worktree remove --force (or rm -rf + prune).

Observed 2026-07-26 on the live witness: 4 orphans holding 11 GB — dated Jul 17, Jul 21, and same-day — on a 147G PVC. Normal cleanup works fine (a run that completes removes its own scratch); these were purely restart debris.

Suggested fix. At daemon boot, sweep <state_dir>/project-check-runs/* with git worktree remove --force (then prune). Nothing there can be live at startup by definition, so the sweep is unconditionally safe. Alternative/complement: make the run-dir name unique per process (e.g. include boot id) so a stale registration can never be collided with.

Found while soaking the CGLS-25 serialization work; the orphans predate that change (Jul 17/21 entries), so this is long-standing, not a regression.

**Symptom.** A gated push fast-fails in ~4s with: ``` project_check_batch_indeterminate: git ["worktree", "add", "--detach", "/workspace/cargoless-state/project-check-runs/run-1-4", "origin/dev"] exited 128: fatal: ... is a missing but already registered worktree; use 'add -f' to override, or 'prune' or 'remove' to clear ``` The verdict publishes as `unknown` — it does not wedge a PR, but it burns a gate slot and reads as an unexplained infra flake. **Cause.** When the serve pod dies mid-compile (Flux roll, OOM, node eviction), the per-run scratch worktree survives on the PVC **and stays registered in `.git/worktrees/`**. The run counter resets on the new pod, so a later run eventually reuses the same `run-1-N` index and collides with the orphaned registration. **Why `git worktree prune` does NOT fix it.** Prune only clears registrations whose directory is *gone*. These directories still exist (full of build artifacts), so prune reports nothing to do. Clearing them requires `git worktree remove --force` (or `rm -rf` + prune). **Observed 2026-07-26** on the live witness: 4 orphans holding **11 GB** — dated Jul 17, Jul 21, and same-day — on a 147G PVC. Normal cleanup works fine (a run that completes removes its own scratch); these were purely restart debris. **Suggested fix.** At daemon boot, sweep `<state_dir>/project-check-runs/*` with `git worktree remove --force` (then `prune`). Nothing there can be live at startup by definition, so the sweep is unconditionally safe. Alternative/complement: make the run-dir name unique per process (e.g. include boot id) so a stale registration can never be collided with. Found while soaking the CGLS-25 serialization work; the orphans predate that change (Jul 17/21 entries), so this is long-standing, not a regression.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

triform/cargoless#87

No description provided.

Rows
Columns