ci: make CI executable + close the drift loops it never caught #44
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "agent/oss-ci-verification-loops"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
This repo's CI has never been green. 171 of 175 runs failed — zero successes across June–July 2026. It was executing the whole time; it just failed on every trigger, so nothing ever caught drift. That single fact explains most of what this PR fixes: the dead
infra/Dockerfile, the never-readconfig.js, the unpullable Helm sidecar, two live silent-failure bugs inCb.startNativeSession, and an entirely unimplemented security control.Fixing the code without fixing the loop just resets the clock, so this does both.
What
1. CI can actually pass. Forgejo git-clones every
uses:in a job before running any step, so one unresolvable action kills the job regardless of itsif:(logs showskipping post step for 'actions/checkout@v4'; step was not executed). Seven actions 404 ondata.forgejo.org; all replaced with portablerun:steps or gated.github-scriptalone caused 56 failures,codeql-actionanother 38.2. Three host-runnable checks — no Chromium tree, no Docker, ~13s via
make verify:workflow_actions_lint.py— everyuses:must resolve on the mirrorcxx_include_lint.py— Chromium symbols used without their header (capture/only compiles inside a 4–8h build, so this class otherwise surfaces hours later)test_cxx_include_lint.py— 9 self-tests so a broken linter fails loudlyCI now also runs the client's 207 vitest assertions, which were never gated.
3. Region scoping (
aud) enforced.processRegionwas set at startup and never read;Claimshad noAudfield. A token minted foreu-west-1was accepted by aus-east-1server. The test asserting this has been failing silently the entire time.4. Two live
Cb.startNativeSessionbugs.signalingUseTlswas silently dropped (→wss://at a plaintext broker); array-formiceServersfell through to Google public STUN, so the relay harness believed it was on provisioned TURN and wasn't. Neither path fails quietly now.5. Runnable for people who aren't triform.
SIGNALING_URLis finally translated into theWEBRTC_SIGNALING_*vars the peer reads — previously only the k8s controller could start a session, sodocker runbooted a peerless worker. Plus bring-your-own-image compose, a Helm default that made every install fail, and a README that described an abandoned architecture.Verification
make verifygreen: 207 client + all signaling/integration tests, all linters, all workflow YAML, every action resolving.⚠️ The C++ is not compiled. No Chromium tree was available. Every API was checked against in-tree precedent on
branch-heads/7727, which caught a real defect (BindOnce/FROM_HEREwithout their headers) — now enforced by the include linter. Treatcapture/changes as unverified until the build lane runs.build/chromeless-build.shStep 7 is now fatal by default; it previously loggedWARN: continuingand shipped anyway, and the opt-in flag was set by nothing, anywhere.This repo's CI has never been green. Across June-July 2026: 171 of 175 runs failed, zero successes. It was executing the whole time -- Forgejo does run .github/workflows -- and failing on every trigger. Two root causes, both confirmed from production logs. 1. Unresolvable actions. Forgejo resolves actions from data.forgejo.org and git-clones EVERY `uses:` in a job before executing ANY step, so a single missing action fails the whole job regardless of its `if:`. The logs show "skipping post step for 'actions/checkout@v4'; step was not executed" -- the job died before checkout. Seven actions 404 there: actions/github-script -> tools/ci/file_regression_issue.py (56 failures, largest bucket) github/codeql-action -> whole job gated to github.com (38 failures) hadolint/hadolint-action -> container `run:` lycheeverse/lychee-action -> container `run:` anchore/sbom-action -> upstream syft installer softprops/action-gh-release -> tools/ci/publish_release.py actions/attest-build-provenance -> removed (GitHub-only API) A job-level `if:` IS evaluated before resolution (verified against this Forgejo instance: 5229 jobs record status=skipped), which is what makes the CodeQL gate work where a step-level `if:` would not. 2. The image-building jobs built infra/Dockerfile, unbuildable since the M7 native-peer migration deleted capture/streamer-page/. Those jobs are now opt-in on a supplied CHROMELESS_IMAGE: a from-source Chromium build (4-8h) cannot run on a CI runner, and this repo publishes no images, so they report a notice and pass rather than failing forever. Adds three host-runnable checks, none needing a Chromium tree or Docker: tools/lint/workflow_actions_lint.py every `uses:` must exist on the mirror. Calibrated twice against reality: the mirror answers 403 for actions that EXIST and 404 for missing ones (so `>=400` was wrong), and the first regex missed subpath actions like github/codeql-action/init@v3 -- which is how those 38 failures hid from the linter itself. tools/lint/cxx_include_lint.py catches Chromium symbols used without their header. capture/ only compiles inside a full Chromium checkout, so this class of defect otherwise surfaces hours later in the build lane. Motivating case: Cb.shutdown shipped using base::BindOnce and FROM_HERE with neither header included. The first draft's 20 rules produced 46 findings on code that builds, so the table was cut to 13 high-confidence rules plus measured umbrella relationships. tools/lint/test_cxx_include_lint.py 9 self-tests pinning both directions -- catches the real defect, stays silent on transitive includes, comments, string literals, and reports true line numbers. `make verify` runs all of it plus every unit and integration suite in ~13s. CI also now runs the client's 14 vitest suites (207 assertions), which were never gated -- only `npm run typecheck` was wired. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>initRegion() set processRegion at startup and NOTHING ever read it, and Claims had no Aud field at all. A token minted for eu-west-1 was accepted by a us-east-1 server. Region scoping was documented, tested, and entirely unimplemented. TestRegionAware_AudExcludesRegion has been asserting the 1008 close for this the whole time and failing silently -- it was one of five failures sitting on clean HEAD, invisible because CI never passed. * Claims.Aud, with an UnmarshalJSON that accepts the RFC 7519 4.1.3 bare-string form as well as an array, so a spec-legal third-party token isn't reported as malformed claims. * regionPermitted(), opt-in by design: no aud claim -> permitted; server region unspecified -> permitted (a server that doesn't know its own region can't meaningfully enforce, and failing closed would break every single-region deployment the moment someone minted a scoped token); otherwise the server's region must appear in aud. * enforcement in verifyToken with a region_mismatch auth-failure metric. * dev issuer honours ?aud=a,b so the integration test can mint one. Also deletes tests/integration/input_loop_test.go: it builds a binary from capture/input-bridge/, a sidecar retired in M7 R4 whose source has zero tracked files. Four more of the five failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Both bugs were LIVE and both failed silently, which is why they survived. BUG-1: the handler read `useTls`, but callers write `signalingUseTls` -- every other signaling param carries that prefix, and cv2-fcdirect.mjs:113 sends it under that name with its own comment documenting it. The unknown key was dropped, use_tls defaulted true, and the guest dialled wss:// at a plaintext broker, surfacing minutes later as an opaque TLS error. Both spellings are now accepted. BUG-2: iceServers was read with FindString(), which returns nullptr on a JSON list. An array-form caller therefore fell through to BuildDefaultIceServers() = stun:stun.l.google.com:19302. The fcdirect relay harness believed it was running on provisioned TURN and was running on Google public STUN. Now accepts list, dict, and the legacy JSON-string form (the isolator stores ice_servers_json as a string and forwards it verbatim). The wider fix is that this path no longer fails quietly: an iceServers param that was SUPPLIED but yields no usable servers is a hard ServerError rather than a silent substitution; absence logs a warning; and iceTransportPolicy=relay with zero TURN servers logs an error, because that combination can never connect. Cb.shutdown: wires the quit closure that WillRunMainMessageLoop has been parking and nothing has ever run (the TODO at main_parts.cc:927 names this exact method). It posts rather than runs inline so HandleCommand's response lands before the loop quits -- otherwise the caller races a socket close instead of seeing {"shuttingDown":true}. Quitting unwinds into PostMainMessageLoopRun's LIFO teardown, so the peer gets a `bye` and the WS closes 1000; SIGTERM skips all of that. signaling: iceReplayMaxAge 60s -> 300s, configurable via CHROMELESS_ICE_REPLAY_MAX_AGE_S and clamped to a 600s TURN-allocation ceiling. The old value was justified as "longer than any realistic Phase-1 demo session-start latency", an assumption microVM cold-boot invalidates: past 60s every buffered guest candidate ages out, the peer sees remoteCandidates=0, and it sits in checking forever as 0x0 video. Physics independently converged on 300s for exactly this reason. NOT COMPILED. There is no Chromium tree on the machine this was written on; capture/ needs the 4-8h build lane. Every API used was checked against in-tree precedent on branch-heads/7727, which caught a real defect (base::BindOnce + FROM_HERE without base/functional/bind.h or base/location.h) -- now enforced by tools/lint/cxx_include_lint.py. Treat the C++ as unverified until the build lane runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>The superseded-run-coalescer cronjob cancelled all 8 jobs of the previous run. Its dead_pr arm matches PRs by index alone: WHERE ('refs/pull/' || pr.index || '/head') = r.ref with no pr.base_repo_id = r.repo_id predicate, so chromeless PR #44 (open) matched cargoless #44 and tf-multiverse #44 (both closed/merged) and was treated as dead. 11 open PRs instance-wide are currently exposed to this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>The e2e job reported 'CHROMELESS_IMAGE not set - e2e skipped' correctly, then failed anyway: the 'docker compose down' teardown ran unconditionally under `if: always()`. infra/compose.yaml declares CHROMELESS_IMAGE as REQUIRED (${VAR:?msg}), so EVERY compose subcommand fails interpolation when it is unset -- including `down`. The teardown therefore exited 1 and failed a job that had already decided to skip. Caught by the first e2e run that ever reached this code: the workflow had been dying at action-resolution before today. Swept the other workflows for the same class (a risky step ungated inside a probe-guarded job); e2e.yml was the only instance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>THE root cause of cold-boot frozen video. A cold-boot cb-chromium guest starts on about:blank; capture is armed on that page's FrameSinkId; then the navigation to real content swaps the RenderViewHost (and its FrameSinkId), but nothing re-points the capturer at the new sink. It stays bound to the dead pre-nav sink -> VERDICT=RENDERER-STARVED, frames_encoded=0, no video. Warm-restored golden sessions dodge this (they resume already-on-content with no about:blank->nav transition), which is why cold-boot was the sole frozen path -- and why the isolator cold-boot fallback, though correct at the allocation layer, produced a connected-but-frozen stream. Fix: make capture self-healing at the layer that owns both the observer and the capturer. CbActiveWebContentsResolver already observes the captured WebContents and receives RenderViewHostChanged, but is content-layer glue with no capturer handle (the FrameSink capturer is deliberately content-agnostic -- FrameSinkIds only). So the resolver gains an optional re-arm closure (SetRecaptureOnRvhSwapCallback) that CloudBrowserBrowserMainParts supplies; on an RVH swap of the active capture the resolver POSTs it (the new RWH's view/FSID may not be the WC's live primary synchronously at RenderViewHostChanged -- a same-turn UI PostTask runs after the swap settles). The owner (RearmCaptureAfterRvhSwap) re-resolves WC->RWHV->RWH->GetFrameSinkId, forces the fresh view SHOWING (the post-nav RWH boots HIDDEN -> renderer throttled, same gating Cb.startFrameSinkCapture applies), calls cb_track_source_->StartCapture on the new sink, and refreshes the resolver's active target. capturer.cc:138-164 confirms StartCapture on a NEW FrameSinkId re-targets (ChangeTarget), not a no-op -- so the re-arm is effective. For a terminal nav whose fresh RWHV isn't attached by the task turn (no later swap to retrigger), the owner re-posts itself up to 5x/50ms (bounded) rather than strand the capturer. Stale "Start is a no-op" comments in cb_devtools_agent.cc + cb_framesink_video_track_source.{h,cc} that predated the retarget path are corrected so this fix can't be mistakenly reverted as dead code. Layering preserved (capturer stays content-agnostic; re-arm lives in build-integration where content deps already exist). Optional callback: unset -> legacy invalidate-only behaviour, so tests/no-capturer builds are unaffected. Adversarially reviewed (lifetime/UAF, re-entrancy, threading, StartCapture-idempotency, includes, Show() walk) -- no correctness bugs. Composes with the isolator cold-boot fallback (now a COMPLETE fix, not frozen) and the per-language warm-slot fix. Guest change -- needs a cb-chromium rootfs bake to deploy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>