capture+signaling: batch R — the worker redials the broker (verified live); the resize GPU abort is gone #108

Merged

triform-admin merged 11 commits from capture/batch-r-audio-resize-redial into main 2026-09-08 13:31:37 +00:00

Conversation 0 Commits 11 Files changed 24 +1130 -101

triform-admin commented 2026-09-08 10:49:10 +00:00

Owner

Copy link

Fixes for the three guest defects found on 2026-09-07, plus a broker fix the first one uncovered. Six lane builds, four live measurement rounds. Every claim below was measured on the k8s standalone stack.

1. Redial — FIXED, VERIFIED LIVE

cb_signaling_reconnect (R7) had been compiled by every lane since M3 and constructed by nothing. main_parts now owns it instead of the bare ws client; it IS-A SignalingTransport so the driver's raw pointer survives a redial. On reconnect the embedder re-offers unless a viewer is still attached; OnGaveUp recycles the process.

Two latent bugs in the wrapper that six months of disuse had hidden: a clean 1000 close from a broker shutting down went terminal (exactly the rollout case it existed for), and nothing distinguished our own Disconnect. Its BUILD.gn TODO had promised tests "in a follow-up commit" since M3 — cb_signaling_reconnect_test.cc now exists and the t7 lane compiles it.

That alone was not enough: with the guest redialing, the attached viewer still lost video, because the broker synthesised a bye to every peer as it shut down. Every socket closing at once is indistinguishable from every peer leaving. hub.shuttingDown suppresses it (TestWS_ShutdownDoesNotSynthesiseByes, watched fail).

Measured with a viewer attached across a broker rollout:

	before	after
attached viewer's framesDecoded	stops	14 → 893, uninterrupted
worker process	killed by liveness ~70 s later	unchanged
dials	1	3
next viewer	after a restart	connects normally

2. Resize — HALF FIXED (abort gone, freeze remains)

Attempt 1 (settle 100 ms, then re-issue) moved the GPU abort from 15 s to 108 ms — which confirms the mechanism beyond doubt: any issue before viz runs its pending callback is the double-issue. Attempt 2 (wait for the ack; nudge with ScheduleFullRedraw once a second) eliminated the abort (33 nudges, no FATAL) but the ack still never came, so the picture freezes and CV2-GPU-DEATH recycles instead.

Both attempts and the three next probes are in the finding. CHROMELESS_VIEWPORT_FOLLOW=0 (already default off) keeps the stack usable meanwhile.

3. Audio — THREE THEORIES REFUTED, not fixed

Each was implemented, built, deployed and measured. The ADM is already stopped at re-arm (it was asked: recording=0). Terminate/Init is unsafe — quit_ is set and never cleared, so the record thread would die for the life of the process. Pre-arming the ADM reached now_recording=1 and was still silent. The finding now records all three and names the next probe: instrument libwebrtc's audio send stream, not the ADM.

Verified

• Lane: 664 targets, zero errors, all eight unit-test binaries green including the new reconnect suite. (One pod hit OutOfmemory at admission — a scheduling rejection, not a compile error.)
• tests/interactive 65/65 against the batch-R guest: no regressions.
• make lint, make lint-cxx, go test ./... in signaling all clean.
• Guest pinned by digest in stack.yaml + guest-release.json; broker image batchr-ff7e944 rolled and pinned.

🤖 Generated with Claude Code

Fixes for the three guest defects found on 2026-09-07, plus a broker fix the first one uncovered. Six lane builds, four live measurement rounds. Every claim below was measured on the k8s standalone stack. ## 1. Redial — FIXED, VERIFIED LIVE `cb_signaling_reconnect` (R7) had been compiled by every lane since M3 and constructed by nothing. `main_parts` now owns it instead of the bare ws client; it IS-A `SignalingTransport` so the driver's raw pointer survives a redial. On reconnect the embedder re-offers unless a viewer is still attached; `OnGaveUp` recycles the process. Two latent bugs in the wrapper that six months of disuse had hidden: a clean 1000 close from a broker shutting down went **terminal** (exactly the rollout case it existed for), and nothing distinguished our own `Disconnect`. Its BUILD.gn TODO had promised tests "in a follow-up commit" since M3 — `cb_signaling_reconnect_test.cc` now exists and the t7 lane compiles it. That alone was not enough: with the guest redialing, the attached viewer **still** lost video, because the broker synthesised a `bye` to every peer as it shut down. Every socket closing at once is indistinguishable from every peer leaving. `hub.shuttingDown` suppresses it (`TestWS_ShutdownDoesNotSynthesiseByes`, watched fail). Measured with a viewer attached across a broker rollout: | | before | after | | --- | --- | --- | | attached viewer's `framesDecoded` | stops | **14 → 893, uninterrupted** | | worker process | killed by liveness ~70 s later | **unchanged** | | dials | 1 | **3** | | next viewer | after a restart | **connects normally** | ## 2. Resize — HALF FIXED (abort gone, freeze remains) Attempt 1 (settle 100 ms, then re-issue) moved the GPU abort from 15 s to **108 ms** — which confirms the mechanism beyond doubt: any issue before viz runs its pending callback is the double-issue. Attempt 2 (wait for the ack; nudge with `ScheduleFullRedraw` once a second) **eliminated the abort** (33 nudges, no FATAL) but the ack still never came, so the picture freezes and `CV2-GPU-DEATH` recycles instead. Both attempts and the three next probes are in the finding. `CHROMELESS_VIEWPORT_FOLLOW=0` (already default off) keeps the stack usable meanwhile. ## 3. Audio — THREE THEORIES REFUTED, not fixed Each was implemented, built, deployed and measured. The ADM is already stopped at re-arm (it was asked: `recording=0`). `Terminate`/`Init` is unsafe — `quit_` is set and never cleared, so the record thread would die for the life of the process. Pre-arming the ADM reached `now_recording=1` and was **still silent**. The finding now records all three and names the next probe: instrument libwebrtc's audio **send stream**, not the ADM. ## Verified - Lane: 664 targets, zero errors, all eight unit-test binaries green including the new reconnect suite. (One pod hit `OutOfmemory` at admission — a scheduling rejection, not a compile error.) - `tests/interactive` **65/65** against the batch-R guest: no regressions. - `make lint`, `make lint-cxx`, `go test ./...` in `signaling` all clean. - Guest pinned by digest in `stack.yaml` + `guest-release.json`; broker image `batchr-ff7e944` rolled and pinned. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

triform-admin added 11 commits 2026-09-08 10:49:10 +00:00

capture: batch R — a resize no longer kills the GPU, audio survives a re-arm, the worker redials the broker — UNVERIFIED ... 071d0828f3

The three guest defects measured live on 2026-09-07 and written up in
docs/findings/. All three are in capture/, so none of this is compiled
here; the lane is the only verification and it has not run yet.

1. RESIZE (viewport-resize-freezes-beginframe-and-crashes-gpu.md).
   Confirmed the mechanism in viz source at the pinned branch:
   Display::Resize sets expecting_root_surface_damage_because_of_resize,
   DisplayScheduler then returns kLate and never runs the pending
   ExternalBeginFrame callback, so our ack never comes; 15 s later the
   stall watchdog re-issues and ExternalBeginFrameSourceMojo's
   !pending_frame_callback_ check aborts the GPU process. Fix:
   CbBeginFrameDriver::NotifyDisplayReconfigured — abandon the in-flight
   frame (epoch bump), cancel the watchdog, re-issue once after 100 ms.
   CbViewportController::Apply calls it before touching the Display;
   main_parts injects the driver and clears it before the driver dies.

2. AUDIO (audio-dies-after-first-rearm.md). The ADM is per-process and
   shared by every re-armed PeerConnection; StopInternal relied on
   AudioState::RemoveSendingStream to stop recording transitively, and
   for every session after the first StartRecording then failed with
   "failed to activate recording" (packets_sent=0 for the rest of the
   process). Fix: CbAudioLifecycle::Rearm calls ADM::StopRecording on
   the ADM's own worker thread — the state InitRecording expects to
   start from — and logs Recording()/RecordingIsInitialized() first, so
   the next run says which of the two theories was right.

3. REDIAL (worker-signaling-no-redial.md). cb_signaling_reconnect (R7)
   has been built by every lane since M3 and constructed by nothing.
   main_parts now owns it instead of the bare ws client; it IS-A
   SignalingTransport so the driver's raw pointer survives redials. Two
   bugs in the wrapper that six months of not being used had hidden:
   a clean 1000 close from a broker shutting down was treated as
   consumer-driven and went terminal (the exact rollout case), and
   nothing distinguished our own Disconnect. Fixed with a
   disconnect_requested_ latch. On reconnect the embedder re-offers
   unless a viewer is still attached; OnGaveUp recycles the process.

   Its BUILD.gn TODO promised tests "in a follow-up commit" that never
   came — CLAUDE.md's rule about a TODO naming its own verification.
   cb_signaling_reconnect_test.cc now covers the close rules, the
   backoff/give-up bound, the schedule reset and pass-through, and the
   t7 lane compiles it.

API pins for everything new: docs/build/chromium-7727-api-pins.md,
"Batch R". make lint and make lint-cxx clean. NOT COMPILED — the lane
run and a live re-verification of all three experiments come next.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

chore(cv2-build): record guest release cr7727-071d0828f342 (batch R) and pin the standalone stack to it ... eb045452f4

Built clean on the t7 lane at 071d082: 664 targets, zero errors, and all
eight unit-test binaries ran green — including cb_signaling_reconnect_
unittests, which had never existed before this batch.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(capture): the resize fix must WAIT for the pending ack, not re-issue after it — UNVERIFIED ... e86956e98d

Batch R's first attempt failed live on the image built from it. Measured
2026-09-08 on cr7727-071d0828f342:

  08:54:15.945  CV2-RESIZE display reconfigured (Cb.setViewport)
  08:54:16.053  FATAL viz_main_impl.cc:342 !has_created_frame_sink_manager_
  08:54:16.132  GPU process exited

108 ms — exactly the 100 ms "settle" delay after which the driver
re-issued. The abort did not go away; it moved 15 s earlier. viz still
holds pending_frame_callback_ for the pre-resize frame, and ANY issue
before that callback runs is the double-issue, whether it comes from the
watchdog at 15 s or from a settle timer at 100 ms.

The driver's own header had the answer for the analogous cold-boot and
warm-restore cases and I did not apply it here: wait. A resize sets
expecting_root_surface_damage_because_of_resize, which makes
DisplayScheduler take a LATE deadline, not no deadline — the ack does
arrive once the renderer paints the new geometry. So
NotifyDisplayReconfigured now stops the watchdog and sets
awaiting_reconfigure_ack_; the watchdog treats that flag exactly like the
pre-first-ack window (wait indefinitely, never re-issue), and the ack
clears it and resumes the cadence. If the ack genuinely never comes,
CV2-GPU-DEATH recycles the guest after 30 s of zero frames, which is the
honest answer rather than an abort.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(capture): pre-arm the ADM on re-arm — the stop theory was wrong, the guest said so — UNVERIFIED ... fbd8e55c94

The first audio attempt asked the ADM what state it was in before
stopping it, and the answer refuted the fix:

  CV2-REARM-AUDIO: ADM before re-arm: recording=0 rec_initialized=0;
                   StopRecording() rc=0
  ...4 ms later:   failed to activate recording

Already stopped, so an explicit stop is a no-op. The 4 ms is the real
clue: AudioDeviceLinuxPulse::StartRecording waits up to TEN seconds on
_recStartEvent, so failing 4 ms in means that wait returned instantly on
an event left set by the previous session, then found _recording false.
Nothing clears that event between sessions. Terminate/Init is not an
escape: Terminate sets quit_ and nothing ever clears it, so the record
thread would exit for the life of the process.

So stop taking that path. AudioState::AddSendingStream only starts
recording when !adm->Recording(); if the ADM is already recording when
the next send stream is added, libwebrtc attaches to the running capture
— the state a first session leaves behind. Rearm now stops, re-inits and
starts the ADM itself on the ADM's own thread, and logs all four return
codes plus the resulting Recording(), so the next run says plainly
whether it worked rather than needing another theory.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(signaling): a broker shutting down must not tell its peers the session ended ... ff7e944fcb

Batch R's redial fix worked — measured 2026-09-08, the worker was back on
the wire in 1 s, kept its pid, and the liveness probe never fired. The
attached viewer still lost video, and the cause was the broker.

On shutdown every socket closes at once, and unregister could not tell
that from every peer leaving, so it synthesised a `bye` to each
counterpart. A bye means "this session is over". A rollout is not that:
the peers are still there and their media never touched this socket. The
viewer's client did what a bye says — tore down — and the worker then saw
ICE fail, waited out its 20 s grace and re-armed. A viewer-visible outage
caused entirely by the broker's own restart.

hub.shuttingDown is set before http.Server.Shutdown; unregister skips the
synthesised bye while it is set. Redial + this is what makes a broker
rollout invisible to an attached viewer.

TestWS_ShutdownDoesNotSynthesiseByes was watched fail with the check
removed. It also had to wait for the departing peer's unregister to run:
the first version passed against the broken broker because collect()'s
window closed first — a test that could not have failed.

Spec: docs/protocols/signaling-envelope.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

chore(cv2-build): record guest release cr7727-fbd8e55c948b (resize waits, ADM pre-armed) and pin the stack ... 860f071f21

Built clean on the t7 lane at fbd8e55; all eight unit-test binaries green.
The first pod hit OutOfmemory at admission (a scheduling rejection, not a
compile error — CLAUDE.md's first trap) and the retry compiled 663 targets
with zero errors.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(capture): waiting for the resize ack also needs a nudge, or the picture never comes back — UNVERIFIED ... 0f336fbd1c

Third measurement, third correction. On the build that only waited
(fbd8e55) the GPU abort was gone — the wait works — but the ack never
arrived either: stall watchdog fired every second for 30 s, zero frames
captured, then CV2-GPU-DEATH recycled the guest. Trading an abort for a
freeze is not a fix.

viz says why. ExternalBeginFrameSourceMojo::MaybeProduceFrameCallback
only dispatches the pending callback when the Display is going to draw,
and a Display that has just been resized, with no fresh root-surface
damage, is not going to draw. Nothing was ever going to complete that
frame.

So while waiting, nudge once a second with Compositor::ScheduleFullRedraw
— SetNeedsRedrawRect(viewport) + SetNeedsCommit. It produces exactly the
damage the scheduler needs to reach a deadline and run the callback it is
holding, and it issues NO BeginFrame, so it cannot be the double-issue
that aborts the GPU process. The ack clears the flag, stops the nudge and
reports how many it took. If it still never comes, CV2-GPU-DEATH still
recycles — unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs(findings): audio after re-arm — three fixes tried, three refuted, and the next probe ... e8440d1e8b

Records what was actually measured rather than leaving the finding
describing a fix that did not work: the ADM is already stopped at re-arm
(it was asked), Terminate/Init would permanently kill the record thread
(quit_ is set and never cleared), and pre-arming the ADM to make
AudioState skip its own start reached now_recording=1 and was still
silent. The 4 ms failure against a ten-second wait remains the sharpest
clue. Next probe named: instrument the audio SEND STREAM, not the ADM.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

chore(cv2-build): record guest release cr7727-0f336fbd1c7d (resize nudge) and pin the stack ... e398cacd36

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs(findings): the redial defect is fixed and verified live ... b8ed6f834c

Viewer attached across a broker rollout: framesDecoded 14 -> 893 without
interruption, dials 1 -> 3, pid unchanged, no liveness kill, next viewer
fine. Took both halves — the guest redialing (R7 wired, two latent bugs in
it fixed, its missing tests written) and the broker not telling its peers
the session ended on the way out.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs(findings): the resize crash is half fixed — abort gone, freeze remains, both attempts measured ... 9b27b71abd

Attempt 1 (settle-then-re-issue) moved the abort from 15 s to 108 ms,
which confirms the mechanism beyond doubt. Attempt 2 (wait + full-redraw
nudge) eliminated the abort — 33 nudges, no FATAL — but the ack still
never came, so GPU-death recycles the guest instead. Names the three next
probes, cheapest first, and records that the gateway kill switch keeps
the stack usable meanwhile.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

triform-admin merged commit 30178cd412 into main 2026-09-08 13:31:37 +00:00

triform-admin referenced this pull request from a commit 2026-09-08 13:31:38 +00:00

capture+signaling: batch R — the worker redials the broker (verified live); the resize GPU abort is gone (#108)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

triform/chromeless!108

No description provided.