CV2 WebRTC video stack → main: boot GPU-crash fix (staging 67%→92%) + the CV2 line #37

Merged

triform-admin merged 26 commits from cv2/gpu-crash-watchdog into main 2026-07-06 04:15:58 +00:00

Conversation 0 Commits 26 Files changed 23 +1328 -83

triform-admin commented 2026-07-06 03:33:02 +00:00

Owner

Copy link

CV2 WebRTC video stack → main (durable landing)

Brings the full CV2 native-WebRTC guest line to main so it reaches prod (main is 21 commits behind; prod still serves the pre-CV2 c61404a4 rootfs).

Headline fix (validated on staging): the boot GPU-process crash that permanently blacked ~1/3 of fresh WebRTC sessions. The CbBeginFrameDriver stall-watchdog re-issued a BeginFrame into the warm-restore unbound GPU-channel window → viz double-CreateFrameSinkManager → viz_main_impl.cc:342 CHECK → GPU exit → dead capturer. Fix: the watchdog now waits out the reconnect (kWatchdogFiresBeforeReissue=15) before re-issuing.

Measured on staging (N=12, byte-verified served binary): 8/12 → 11/12 (67% → 92%).

Also included (the CV2 line this fix sits on): warm-snapshot Cb.startNativeSession, dcheck-off (guests SIGABRT'd on assertions), capturer-count hold (~50% cold-guest starvation), WALL1/WALL2 answer-buffer + WS-reassembly, idle-refresh hold-and-repeat, the f9d79b1 boot-window watchdog guard, and build/depot_tools/kaniko fixes.

Blast radius: capture/, signaling/, build/, infra/ (guest launch + supervisord) only.

Residual (fast-follow PR): ~1/12 permanent-GPU-unbind boot (guest self-detect → re-pin, in design).

🤖 Generated with Claude Code

## CV2 WebRTC video stack → main (durable landing) Brings the full CV2 native-WebRTC guest line to `main` so it reaches prod (main is 21 commits behind; prod still serves the pre-CV2 `c61404a4` rootfs). **Headline fix (validated on staging):** the boot GPU-process crash that permanently blacked ~1/3 of fresh WebRTC sessions. The `CbBeginFrameDriver` stall-watchdog re-issued a BeginFrame into the warm-restore unbound GPU-channel window → viz double-`CreateFrameSinkManager` → `viz_main_impl.cc:342` CHECK → GPU exit → dead capturer. Fix: the watchdog now waits out the reconnect (`kWatchdogFiresBeforeReissue=15`) before re-issuing. **Measured on staging (N=12, byte-verified served binary):** 8/12 → **11/12** (67% → 92%). Also included (the CV2 line this fix sits on): warm-snapshot `Cb.startNativeSession`, dcheck-off (guests SIGABRT'd on assertions), capturer-count hold (~50% cold-guest starvation), WALL1/WALL2 answer-buffer + WS-reassembly, idle-refresh hold-and-repeat, the f9d79b1 boot-window watchdog guard, and build/depot_tools/kaniko fixes. Blast radius: `capture/`, `signaling/`, `build/`, `infra/` (guest launch + supervisord) only. Residual (fast-follow PR): ~1/12 permanent-GPU-unbind boot (guest self-detect → re-pin, in design). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

triform-admin added 24 commits 2026-07-06 03:33:02 +00:00

feat(cv2-warm): Cb.startNativeSession CDP method for warm-snapshot cold-start ... a2f77e0d68

Extracts the native WebRTC signaling bring-up (ws_client + offerer driver +
DataChannel host + input/cursor/clipboard/audio+video transceivers) out of the
inline PreMainMessageLoopRun env path into a reusable
MainParts::StartNativeSession(NativeSessionConfig), and adds a Cb.startNativeSession
CDP method that invokes it at runtime with per-session params.

This lets the isolator restore a WARM cb-chromium golden snapshot (chromium
already launched + first-paint-ready, signaling env UNSET → idle CDP-only
target) and inject per-session signaling AFTER restore over CDP — eliminating
the ~30s in-guest chromium cold-start that the config-drive boot path forces.

Behavior is byte-identical when WEBRTC_SIGNALING_* env is set: the boot path
calls StartNativeSession from LoadConfigFromEnv()+LoadIceConfigFromEnv() exactly
as before. When env is unset, the worker no longer early-returns into the idle
state via a now-deleted `return RESULT_CODE_NORMAL_EXIT` — it falls through to
the same return at the end of PreMainMessageLoopRun (which means "stay alive in
the message loop"), so it idles as a warm CDP target awaiting the CDP call. A
snapshot frozen in that state bakes ZERO WebRTC network state (ws_client_/
offerer_driver_/dc_host_ all nullptr).

H1: StartNativeSession wraps its body in ScopedAllowBaseSyncPrimitivesForTesting
because the signaling-thread BlockingCall (audio transceiver) would otherwise
DCHECK-FATAL under the per-task DisallowBaseSyncPrimitives active in a CDP
HandleCommand task. No-op on the env-boot path. Mirrors the existing
ScopedAllowBlockingForTesting in CbDevToolsManagerDelegate::CreateBrowserContext.

The CDP handler parses params via crdtp CBOR->JSON->base::Value, reuses the env
path's ParseIceServersJson / ParseIceTransportPolicy / BuildDefaultIceServers /
SummariseIceServers so the two paths are semantically identical, and maps an
already-started session to an INVALID_STATE ServerError. No BUILD.gn change
(cb_devtools_agent.cc + main_parts.cc share the embedder source_set, which
already deps cb_ice_config + cb_signaling_ws_client).

Companion isolator change (VMM_WEBRTC_WARM_RESTORE, default off) ships after
this binary is live. Design: memory cv2-warm-snapshot-cold-start-design.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(cv2-warm): raw_ptr<AudioDeviceModule> for the rawptr plugin ... 983834ac36

The chromium-rawptr plugin rejects a bare webrtc::AudioDeviceModule* member
(main_parts.h:455). Change adm_for_audio_lifecycle_ to raw_ptr<> + include
base/memory/raw_ptr.h, matching CbAudioLifecycle::adm_debug_ (the consumer)
and the rest of the embedder's non-owning pointer members.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(cv2-warm): use JSONReader::ReadDict for branch-heads/7727 base API ... 264d36d07f

The pinned chromium (M147) has no base::Value::Dict spelling reachable here and
JSONReader::Read has no default options arg. Switch to JSONReader::ReadDict(json)
which parses + extracts the top-level object in one call and returns
std::optional<Dict>; bind with auto so the dict type name is never written.
CBOR->JSON via crdtp::json::ConvertCBORToJSON unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(cv2-warm): pass JSON_PARSE_RFC to ReadDict (no default options arg) ... 82988a4780

ReadDict(json, options, max_depth=...) in branch-heads/7727 has no default for
options. Pass base::JSON_PARSE_RFC (strict; input is machine-generated from
CBOR). Verified DictValue::FindString/FindBool + all crdtp cbor encoders +
JSON_PARSE_RFC against the pinned tree — this was the last API arg-count gap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

obs(cv2-ice): route guest WebRTC ICE logs to the firecracker serial console ... 9cb4d016d0

The Gate-2 'video lands' blocker is conclusively localized to the cb-chromium
guest's cb_offerer emitting ZERO ICE candidates (proven via broker envelope
directions: ice-from-guest=0 while panel ICE, offers, answers all flow). The
cb_offerer_driver.cc CV2-ICE OnIceCandidate/OnIceGatheringChange LOG(INFO)
instrumentation already exists (commit ffcdfdc) but was UNREADABLE from outside
the microVM: chromium was launched with no --enable-logging (LOG suppressed),
and even when emitted supervisord routed stderr to an in-rootfs file
(/var/log/supervisor/chromium.err.log) the host cannot reach.

Two changes make the guest's ICE-gathering state diagnosable from the worker
host WITHOUT a debug build:
  1. launch-chromeless.sh: add --enable-logging=stderr --v=
     (default 0) --vmodule=cb_offerer_driver=1,cb_ice_config=1,cb_devtools_agent=1
     so the CV2-ICE lines are actually emitted (overridable via env).
  2. supervisord.phase2.conf: chromium stderr_logfile -> /dev/console, which
     firecracker captures to the worker host's vmm-work/<vm>/serial.log.

After rebake, one Gate-2 run's serial.log shows whether the guest reaches
ICE gathering (OnIceGatheringChange kGathering/kComplete), whether
OnIceCandidate fires (typ host/srflx/relay), or whether SetLocalDescription
never completes — pinpointing the (a) gathering-never-starts / (b) net-cant-
gather / (c) candidate-dropped-pre-wire fork and completing the RCA the 8
landed physics/portal fixes (D5-D13) could not, because the terminal blocker
is guest-internal. Observability-only; no behavior change to the WebRTC path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(capture): idle-refresh hold-and-repeat so idle pages stream video ... d95478c4f5

THE DEFECT (byte-proven on staging 2026-06-25): portal WebRTC video
produced ZERO frames for IDLE / sporadic-animation content (e.g.
animejs.com between animations) — guest serial VERDICT=RENDERER-STARVED,
frames_received +0 — while continuously-damaging content (scrolling,
the portal UI) streamed fine at ~29fps (VERDICT=PRODUCING).

ROOT CAUSE: viz's FrameSinkVideoCapturer is a PULL consumer —
OnFrameCaptured fires ONLY when the captured renderer commits a new
damaging CompositorFrame. CbBeginFrameDriver issues 30fps external
BeginFrames and forces the root Display to redraw, but those ticks do
NOT reach an idle renderer's cc::Scheduler (client_needs_begin_frame_
false → the support never subscribes to our BeginFrameSource). The
driver's own header (cb_begin_frame_driver.h:125-134) flags this as
architecturally unavoidable from the browser process and defers the
static-page cure to "the encoder/track-source layer [must]
hold-and-repeat the last frame" — which never existed. So an idle page
streamed 0 fps.

THE FIX (capturer.{h,cc}): an idle-refresh DEADLINE timer in the
capturer (the proper, non-diagnostic owner of the producer Remote — NOT
the driver's diag_capturer_ handle, which is documented diagnostic-only).
Every delivered frame re-arms a base::OneShotTimer one period out. An
animating page delivers frames faster than the period, so the deadline
never fires → ZERO RequestRefreshFrame calls → the PRODUCING path is
bit-for-bit unchanged. Only when natural production stops for a whole
period does the deadline fire: we call producer_->RequestRefreshFrame()
(re-delivers the LAST composited surface with a fresh capture timestamp,
no renderer repaint), which arrives as a normal OnFrameCaptured —
advancing frames_received and re-arming the deadline into a steady
hold-and-repeat cadence until the page paints again.

Opt-in via SetIdleRefreshPeriod(); base::TimeDelta() (default) leaves it
OFF so existing callers/tests are unaffected. Wired ON at 10fps in
cloud_browser_browser_main_parts.cc step 5b (>2.9x margin under the
~29fps producing inter-frame, so producing pages issue zero refreshes).
New FrameSinkCapturerStats.idle_refreshes_requested counter lets a log/
metric scrape attribute frames_received deltas to refresh vs natural
paint.

Tests (capturer_test.cc, new MOCK_TIME fixture): disabled-by-default
never refreshes; idle renderer gets steady per-period cadence; ANIMATING
page (30fps) is NEVER refreshed (the safety invariant); refresh fires
only during the idle gap and snaps back on resume; Stop() cancels.

BAKE-GATED: this is cb-chromium C++ — needs a rootfs rebuild to deploy,
then re-validate framesDecoded>0 on animejs.com at the client.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

build(cv2): repoint build-job to live cb-build-runner + decouple ephemeral request ... 2ba180b525

Two build-pipeline blockers fixed so the idle-refresh-capturer fix can bake:

BLOCKER A — build-runner image GC-pruned: the
registry.triform.cloud/chromeless/chromeless-build-runner repo path was
pruned from the registry (NAME_UNKNOWN on every tag). The canonical,
byte-identical runner now lives at cloud-browser-webrtc/cb-build-runner
:0.1.4 (CV2-rehome rename; config blob verified identical 2026-06-29 —
same Debian-bookworm base, depot_tools --depth 200, sccache v0.8.2,
builder uid 1000). Re-point both the bootstrap initContainer and the
build container to it. Chosen over a fresh rebuild because this is the
exact image that produced every live cr7727 tag — no depot_tools main
version-skew.

BLOCKER B — triform-8 ephemeral-storage 96% RESERVED: the manifest set
only an 8Gi ephemeral LIMIT, which K8s mirrors into an implicit 8Gi
REQUEST. triform-8 is ~96% ephemeral-reserved (but only ~6% used — 87Gi
truly free), so the 8Gi request made the Pod unschedulable. The build's
real I/O is on hostPath (/work, /sccache), so its ephemeral footprint is
log spillover (tens of MiB). Add an explicit 1Gi request, keep the 8Gi
limit for burst. Root-cause fix (overprovisioned scratch), not a bypass.

Also points OUR_REPO_REF at cv2/idle-refresh-capturer (commit d95478c).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

build(cv2): bootstrap tree depot_tools + drop pre-broken pcf_unittests ... 45b146f341

Two further build-pipeline fixes found by driving the build on
cb-build-runner:0.1.4:

1. depot_tools python3 bootstrap as uid 1000. The cb-build-runner image
   ships /opt/depot_tools ROOT-owned; the build container runs uid 1000
   with allowPrivilegeEscalation=false (no sudo). depot_tools' cipd
   self-bootstrap does 'mktemp -p .' inside /opt/depot_tools → Permission
   denied → empty tmp path → 'curl ... -o ' ('option -o is badly used')
   → python3 never installed → STEP 5 'gn gen' dies on
   python3_bin_reldir.txt not found. Fix: bootstrap the Chromium tree's
   own vendored //third_party/depot_tools (builder-writable, branch-7727
   version-matched) and prepend it to PATH so build.sh's bare gn/autoninja
   resolve to it. Proven as uid 1000: bootstrap RC=0, gn 2341, gn gen made
   28496 targets, autoninja+ninja 1.12.1 resolve. (The pruned
   chromeless-build-runner image evidently shipped /opt/depot_tools
   writable or pre-bootstrapped.)

2. Drop cloud_browser_pcf_unittests from the target list. It has
   PRE-EXISTING breakage unrelated to this branch: cloud_browser_pcf_test
   .cc:112 dynamic_cast under -fno-rtti, :129 calls the dropped
   TestAudioDeviceModule::CreateTestAudioDeviceModule webrtc API (last
   touched by cv2-91 127715a). autoninja -k1 aborts the whole graph on it,
   blocking the worker binary. Validation of THIS fix is unaffected — the
   5 idle-refresh tests live in cloud_browser_framesink_capturer_unittests
   (retained).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

build(cv2): link gfx::Size PrintTo into framesink_capturer_unittests ... 6bda1e1a50

The cloud_browser_framesink_capturer_unittests suite failed to LINK:
  ld.lld: error: undefined symbol:
    gfx::PrintTo(gfx::Size const&, std::ostream*)

RetargetReappliesResolutionConstraints does
EXPECT_EQ(gfx::Size(1280,720), producer_.last_resolution()), so gtest's
value pretty-printer references gfx::PrintTo(const Size&, ostream*). That
symbol is DEFINED in //ui/gfx:test_support (geometry/test/geometry_util
.cc:482), but the test target only depped //ui/gfx/geometry (which merely
DECLARES it in size.h:99). Add //ui/gfx:test_support to deps.

Latent since the gfx::Size EXPECT_EQ landed (6dcc1b27, 2026-06-14) —
surfaced now because this is the first pipeline run that links the full
suite (prior cr7727 builds shipped only the worker binary). NOT
introduced by the idle-refresh fix (d95478c). //ui/gfx:test_support is
testonly=true with no visibility restriction; this is a test() target so
the dep is clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

build(cv2): fix test teardown dangling-raw_ptr crash + skip in-pod CDP step ... e1e63cd95a

Driving the build surfaced two more issues, both orthogonal to the
idle-refresh fix's runtime logic:

1. capturer_test.cc TearDown ordering — partition_alloc FATAL. The base
   AND idle-refresh fixtures both did 'capturer_.reset(); delivered_
   .clear();'. Each delivered media::VideoFrame holds a BufferHandleScope
   whose dtor runs on_done_metric_ = BindRepeating(++s->buffers_done,
   &stats_) — a raw ptr into the capturer's stats_ (capturer.cc:281,
   pre-existing 7ff92c25). Destroying the capturer (and stats_) FIRST,
   then releasing a still-queued frame, fires that closure on freed memory
   → 'Detected dangling raw_ptr in unretained' FATAL → tests CRASHED.
   The two idle-refresh tests that legitimately leave frames queued
   (AnimatingPageIsNeverRefreshed, RefreshOnlyDuringIdleGap) were the
   first to retain frames through teardown and exposed it. Fix: clear
   delivered_ BEFORE capturer_.reset() in both fixtures so the Done()-ack
   closures run while stats_ is alive. Test-only; production releases each
   frame via libwebrtc while the capturer lives.

2. SKIP_CDP_VALIDATION=1. STEP 10 does an in-pod 'kubectl apply' of the
   CDP smoke job, but the cb-build-runner image has no kubectl → exit 127
   → the step dies and burns a backoffLimit retry + full rebuild even
   though the worker was already built+staged at STEPS 6-9 (every
   incarnation here restarted at STEP 10). Skip it; the binary is
   validated by the in-build unit suites + the downstream animejs
   re-validation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(cv2-signaling): tolerate duplicate cross-pod answer (D18b crash fix) ... 220f81cb50

Physics' cross-pod answer leg re-delivers the same answer ~6ms apart.
HandleAnswerEnvelope previously FailWithReason'd on the 2nd (state was
kSettingRemote), tearing down from a posted-task ctx → FATAL DCHECK
!tls_base_sync_primitives_disallowed → guest SIGABRT → dead guest can't
answer client STUN checks → respR=0 → ICE failed → framesDecoded=0.

Drop dup/late answer idempotently when already kSettingRemote/kIceInFlight.
RCA byte-proven 2026-06-30 (guest serial cb_offerer_driver.cc:865 +
cb_signaling_ws_client.cc:632 dup envelope). Pairs with the idle-refresh
capturer fix already on this branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(cv2-signaling): buffer early answer arriving in kSettingLocal (D18b gap) ... 2331706fc2

The prior dup-answer tolerance (220f81c) only guarded the too-LATE window
(kSettingRemote/kIceInFlight). On the cross-pod physics path the answer can
arrive too EARLY — before our own SetLocalDescription completes (kCreatingOffer/
kSettingLocal, earlier in the enum than kAwaitingAnswer) — and fell through to
FailWithReason -> OnFailed -> cb_audio_lifecycle teardown -> FATAL
tls_base_sync_primitives DCHECK -> guest SIGABRT -> respR=0 -> no video.
Measured deterministic 0/5 on animejs cross-pod (agent ab352ae0 RCA).

Buffer the early answer (it is the legit first answer, not a redundant dup)
and drain it from HopHandleSetLocalDescriptionComplete once we reach
kAwaitingAnswer. Pairs with idle-refresh + the kSettingRemote/kIceInFlight
dup-tolerance already on this branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(cv2-webrtc): hold WebContents capturer-count to end ~50% cold-guest renderer starvation ... 40aa088705

THE residual ~50% framesDecoded=0 defect (RCA 2026-06-30, all alternatives
falsified by test: content-independent example.com==animejs, concurrency-
independent sequential-also-starves, NOT signaling/crash/capture-binding).

Mechanism (source-proven): the capture path drove external BeginFrames + called
WasShown() but NEVER took a WebContents capturer count (IncrementCapturerCount
absent from all of capture/). web_contents.h: 'embedders must increment the
capturer count while capturing ... renderers will be configured to produce
compositor frames regardless of their backgrounded or on-screen occlusion
state.' Without it, ~50% of cold offscreen guests applied hidden-rendering
optimizations → renderer never subscribed to the external BeginFrame source →
0 CompositorFrames (serial: issued=147 acked=147 frames_received+0
VERDICT=RENDERER-STARVED). Warm-forked guests inherited a producing snapshot →
worked; cold guests raced the subscription → ~50% starved permanently.

Fix: take initial_web_contents_->IncrementCapturerCount(gfx::Size(), false,
true, true) right after WasShown()/Focus(), holding the returned
ScopedClosureRunner as a member for the worker's lifetime (released before the
WebContents via reverse-order member destruction). On top of the live idle-
refresh + dup-answer(kSettingRemote/IceInFlight/kSettingLocal) fixes (this is
their branch, HEAD 2331706).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(build): bootstrap depot_tools in runner image (python3_bin_reldir.txt) ... 7c70791d26

The GC-pruned chromeless-build-runner:0.1.4 had to be rebuilt; a fresh
depot_tools clone is NOT bootstrapped, so gn gen fails with
'python3_bin_reldir.txt not found. need to initialize depot_tools by
running gclient'. Run 'gclient --version' at image-build time to trigger
ensure_bootstrap (creates python3_bin_reldir.txt + the vpython3 shims),
and assert the file exists so a future bootstrap regression fails the
image build, not the chromium build.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(cv2-webrtc): include callback_helpers.h in main_parts.h for ScopedClosureRunner ... 2707804e73

The capture_keepalive_handle_ member uses base::ScopedClosureRunner, declared
in base/functional/callback_helpers.h — the .h only included callback.h, so
every TU including cloud_browser_browser_main_parts.h would fail to compile.
The .cc already had the include; the .h needs it too (member type must be
complete at class definition).

fix(cv2-webrtc): disable dcheck_always_on — guests SIGABRT on assertions, 0 frames ... 6893e91a25

Proven in staging: 7/7 cb-chromium guests across the isolation fleet
FATAL-crashed on a non-fatal Blink assertion (DCHECK failed:
default_svg_style_->... at css_default_style_sheets.cc:185) while
rendering normal pages. Root cause: args.gn sets is_official_build=false
with no dcheck_always_on override, and Chromium defaults
dcheck_always_on=!is_official_build → DCHECKs ship ACTIVE+fatal in the
'release' worker build. The renderer aborts on assertions that production
Chrome tolerates, producing 0 frames and tearing down the WebRTC
FrameSink session (portal: 'native FrameSink capture failed to start',
WS close 1006, framesDecoded=0).

Setting dcheck_always_on=false stops the guest renderer aborting on
debug-only assertions; is_official_build stays false for build speed.

This sits on top of the already-deployed IncrementCapturerCount
renderer-starvation fix (40aa088) and GPU-ICD fix (2707804d) — the
DCHECK crash killed the session before either could take effect.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

chore(cv2): CB_UA_RESOURCE renderer probe + kaniko ephemeral request + build-job ref pin ... a66f84c3db

Three pre-build changes for the provenance-proven rebuild:

1. cloud_browser_content_client.cc: log CB_UA_RESOURCE (resource id +
   byte size) on the first GetDataResourceString call per process and on
   EVERY empty resolution. Blink's UA stylesheets (svg/mathml/media-
   controls) come through this exact call from the renderer; an empty
   return is the css_default_style_sheets.cc:185 FATAL-DCHECK precursor.
   Deployed binaries repeatedly turned out stale (pre-ContentClient
   wording at cb_devtools_agent.cc:123) despite fixed source, so the
   resource-path truth must be observable on the guest serial console.

2. chromeless-kaniko-push.yaml: explicit 2Gi ephemeral-storage REQUEST
   decoupled from the 8Gi limit — an implicit 8Gi request fails
   admission on triform-8 (~95% ephemeral-reserved).

3. build-job-x264-t7.yaml: OUR_REPO_REF cv2/wave-3-integration →
   cv2/idle-refresh-capturer in BOTH env sites (init clone + build
   container; documented mismatch hazard).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix(build): 512Mi ephemeral request on t7 build job (99%-reserved node admission) ... 877d5b9a39

Same class as the kaniko-push fix: an 8Gi limit with no explicit request
mirrors an implicit 8Gi request; triform-7 has 0.92Gi ephemeral headroom
(87.1Gi requested / 88Gi allocatable) so admission fails with
OutOfephemeral-storage. Heavy I/O is on the /work hostPath; 512Mi is the
schedulability floor, the 8Gi limit still governs burst.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix(build): kaniko ephemeral request 2Gi → 512Mi (97%-reserved node admission) ... 27fc637b2e

t7 headroom is ~2.06Gi (88.2Gi requested / 92.3Gi allocatable when the
build job holds its 512Mi) — the 2Gi kaniko request lost admission by
~90MB. Kaniko's real ephemeral use is a few hundred MB of layer scratch;
512Mi is the schedulability floor, the 8Gi limit keeps burst.

Also: freed t7 root disk (99%→31%) by pruning 57 unreferenced blobs
(~80GB) from /opt/triform/isolation-artifacts/sha256 — the worker
artifact cache accreted one 1.8-3.2GB rootfs per bake all campaign with
no GC; kept the 12 blobs referenced by live manifests (c61404a4 prod,
2707804d + 39f6d4eb staging).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix(cv2-signaling): WALL2 WS frame-reassembly — arrival path owns receiving_message_ ... e3994cf236

Cherry-pick of the cb_signaling_ws_client.cc half of b2b5450c03
(fix/cv2-walls-guest, never merged into this lineage). Byte-confirmed
live on the 877d5b9 staging fleet: paced-run failures show
'CONTINUATION frame with no message open' (2 per guest) followed by
'ws Send(ice) failed' — the guest offers, then its signaling WS dies
before ICE goes out → client remoteC=0/respR=0 → no video (10/12
paced runs).

Mechanism (from b2b5450's RCA): FinishFrame reset receiving_message_
on the DECODE timeline, which lags the ARRIVAL timeline. A message
that already opened on the arrival path got its open clobbered when an
earlier message finished decoding, so the next CONTINUATION tripped
'no message open' and killed the WS. The arrival path (OnDataFrame
sets ) is the single authority.

The WALL1 half of b2b5450 (dup-answer tolerance) is already covered
here by 220f81c + 2331706.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix(capture): watchdog boot-window guard — don't re-issue before the first-ever ack ... f9d79b1316

The stall watchdog re-issued a BeginFrame when no ack arrived within 1s,
assuming 'the pending callback was dropped'. On cold microVM boots the
real cause is different: the viz external-begin-frame controller is not
bound yet (GPU channel still initializing, routinely >1s), and
ui::Compositor is holding our FIRST issue stashed in
pending_begin_frame_args_ for replay-on-bind. Re-issuing in that window
creates the exact double-issue the issue path promises never happens and
corrupts the bind/create sequence — observed live on the e3994cf staging
fleet as the GPU process dying on
 (viz_main_impl.cc:342),
~1-2 of 12 boots, presenting as ICE-connected-but-0-frames (the last
residual failure mode after the WALL2 fix: light 6/6, heavy 5/6).

Guard: before the first-ever ack, a watchdog fire re-arms and waits (the
stashed frame replays on bind and restarts the chain by itself). After
first ack the controller is bound for the process lifetime, so the
existing abandon-and-re-issue recovery stays as-is (including across
Stop()/Start(), where a missing ack is a genuine stall again).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix(cv2-webrtc): warm-restore stall-watchdog guard (.cc) 23d7504a89

fix(cv2-webrtc): warm-restore stall-watchdog guard (.h) 041884f1ce

fix(cv2-webrtc): raise stall-watchdog re-issue threshold 2->15 (warm-restore rebind tail) ea2058c1d0

triform-admin added 1 commit 2026-07-06 03:39:34 +00:00

merge-resolve: cloud_browser_browser_main_parts.h — keep callback_helpers include + main PR#36 CV2-WARM additions 8f4d709647

triform-admin added 1 commit 2026-07-06 04:14:15 +00:00

cv2(gpu-crash): relocate callback_helpers.h include to resolve add/add conflict with main ... d2b71d387b

Functionally order-independent; placed after base/timer/timer.h so main PR #36 CV2-WARM raw_ptr.h auto-merges and this add lands as its own clean hunk. No logic change; resolves the sole textual conflict blocking PR #37->main.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

triform-admin merged commit 45b68996de into main 2026-07-06 04:15:58 +00:00

triform-admin referenced this pull request from a commit 2026-07-06 04:16:00 +00:00

Merge pull request 'CV2 WebRTC video stack → main: boot GPU-crash fix (staging 67%→92%) + the CV2 line' (#37) from cv2/gpu-crash-watchdog into main

triform-admin referenced this pull request 2026-07-10 07:52:53 +00:00

cv2(gpu-death): compose guest GPU-death fix onto the deployed capstats+rearm lineage (deploy-ready superset) #42

triform-admin referenced this pull request from a commit 2026-07-30 04:00:29 +00:00

fix(build): restore unittest targets to the t7 lane (10-week silent skip)

triform-admin referenced this pull request 2026-07-30 04:01:02 +00:00

fix(build): restore unittest targets to the t7 lane (10-week silent skip) #45

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

triform/chromeless!37

No description provided.