build: the lane was green about files it never compiled — fix + two guards #60
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "c8/three-unbuilt-tests"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this is
Three commits, one thread: the x264-t7 build lane was green about a smaller
set of files than anyone believed it covered, and this makes that impossible
to repeat.
1. Add the three test targets nothing had compiled (
cd1b3b6)capture/**/BUILD.gndeclares seventest()targets. Three of them(
adm,input_dispatch,pointer_state) appeared in zero buildmanifests. They had been declared, reviewed, merged, and never compiled by
anything for roughly ten weeks.
2. The first compile found a real defect (
9b22f70)Adding them was not a formality. The very first build failed:
CbAudioTestRecorderdeclared itselfpublic webrtc::AudioTransportandimplemented two of that interface's three pure virtuals. The class was
abstract, so the two
test::CbAudioTestRecorder recorder;declarationscould not compile — and nothing had ever tried.
The header's own
TODO(M55-R2-test-audio-device-header)predicted this downto the mechanism: "confirm the same header still satisfies AudioTransport in
this branch when the test first goes through the chromium build pod." It did
not, and the first trip through the build pod is exactly what found it.
The fix is a silence-renderer, same contract and same rationale as the
existing
NeedMorePlayData— M5.5's measurement is capture-side, so therender pull only has to be safe. One trap earned a comment:
PullRenderDatareports geometry in bits per sample and frames, where
NeedMorePlayDatauses bytes and samples. Copying the sibling's arithmeticwould
memsetthe wrong length.3. Two guards so neither half recurs (
9131b30)tools/lint/build_targets_lint.py— everytest()target must be builtby at least one lane. Fails in both directions: declared-but-unbuilt is the
ten-week gap; built-but-undeclared is a typo or rename that
gnwould rejecthours into a job, caught in 40ms instead.
The rule is deliberately "at least ONE lane", not "every lane" —
pcf_unittestsis x264-t2-only on purpose and the encoder tests areprofile-gated. Demanding uniformity would produce exactly the false positives
CLAUDE.md warns cost more trust than a missed defect costs time.
Two things worth calling out about the self-tests:
block does not count as built. The real manifests document all seven in
comments, so a parser that scooped those up would pass on the exact tree
this lint exists to fail — green, sitting on top of the gap.
(
_unittestvs_unittests) showed direction 2 skipping the likeliesttypo of all, because the predicate was a suffix match. Now a substring, so
both arms fire together on a misspelling and the reader sees a near-miss
pair rather than hunting for a manifest entry that is sitting right there.
infra/k8s/chromeless-build/fire-build.sh— the other half is notvisible to a repo lint, because the divergence is between a file and a
cluster object. Firing a build by hand means copying a manifest, adjusting
name/ref/targets, applying; every step can silently desync, and the Job runs
to completion either way, reporting a verdict about whatever the live spec
said.
That cost three wrong results in one afternoon: a manifest
sed-derived froma previous job's copy (branch had six targets, live Job had three, reported
as "all six compile"); the same class again; and a job still pinned to
OUR_REPO_REF=mainwhile the commit under test was on a branch.The fix in all three was identical — read the live object back — so the
script does that before the job burns 30 minutes rather than after. Manifest
comes from
git show <ref>:<path>.OUR_REPO_REFis rewritten at both sitesand it aborts on fewer than two (an init/build mismatch silently runs an old
build script; cost two attempts on 2026-05-17). Post-apply, live targets and
ref are diffed against the ref's manifest; mismatch deletes the Job and exits
non-zero. The ref argument is mandatory — defaulting to
mainis preciselyhow you test
mainwhile believing you tested your branch.Verification
make verifyclean, self-tests pass. The guards were tested against the livecluster, not just read:
deleted the Job, exited 1
ten-week gap and named all three targets
The tampered-run Job was removed and the in-flight build was left untouched.
Status of the C++
PullRenderDatais compile-verified only in the negative: the t7 laneproduced the error this fixes. The fix itself is in a build that is still
running as of this writing (
chromeless-build-c8c,NINJA_KEEP_GOING=0soone pass collects every remaining drift error rather than one per cycle).
Two of the three newly-added targets have never been compiled by anything, so
further drift in them is likely — that result will be posted here.
cloud_browser_adm_unittests compiled clean (561/562 edges) and then failed at link with three undefined symbols: ld.lld: error: undefined symbol: webrtc::metrics::HistogramAdd(...) >>> referenced by audio_device_buffer.cc:199 in libaudio_device_buffer.a ld.lld: error: undefined symbol: webrtc::metrics::HistogramFactoryGetEnumeration(...) ld.lld: error: undefined symbol: webrtc::SystemTimeNanos() >>> referenced by time_utils.cc:161 in libtimeutils.a Chromium does not build libwebrtc's own versions of these. It REPLACES them with implementations in webrtc_overrides that forward to //base: metrics.cc -> //third_party/webrtc_overrides:metrics rtc_base/system_time.cc -> //third_party/webrtc_overrides:system_time The cloud_browser passthrough re-exported plenty of HEADERS but neither of these implementations, so nothing put them on a link line. They go in `deps`, not `public_deps` — no cloud-browser TU includes them; they only have to be present at link. Why it never showed up before: cloud_browser_worker links the whole content/browser stack, which drags both in transitively. Only a dep-narrow target can see the gap, and the only dep-narrow targets in this tree are the unit tests — one of which had been declared but built by no lane for ~10 weeks. This is the SAME SHAPE as the //base-in-private-deps defect found earlier today: production is fine because it links everything, the deficiency is invisible until something links a small subset, and nothing was linking a small subset. Two independent instances of one structural gap in one day is the argument for `make lint-build-targets`, which now makes "declared but built by nothing" impossible to reach. Patch hygiene: the hunk header and the commit-message diffstat both encode line counts, and both were updated (82 -> 117). Verified with `git apply --check` against a clean tree — a stale count makes the patch fail to apply at STEP 3, hours before any of this would be reached. Not yet compiled: the re-fire is next. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Build result: second real defect found, same structural shape
The
PullRenderDatafix worked —cloud_browser_adm_unittestsnowcompiles (561 of 562 edges done). It then failed at link:
Chromium does not build libwebrtc'''s versions of these three. It replaces
them with implementations in
webrtc_overridesthat forward to//base(
metrics.cc,rtc_base/system_time.cc). The cloud_browser passthroughre-exported plenty of headers but neither implementation, so nothing put them
on a link line.
Fixed in
3c5fbd8by adding both to:webrtc_test_passthrough— indeps, notpublic_deps, because no cloud-browser TU includes them; theyonly need to be present at link.
Why this matters beyond the fix
This is the same structural shape as the
//base-in-private-deps defectfound earlier today:
transitively and is fine
Two independent instances of one structural gap, surfaced within hours of each
other, both by the same act of compiling code nothing had compiled. That is
the argument for
make lint-build-targets: it makes "declared but built bynothing" unreachable, so this class cannot accumulate silently again.
Patch hygiene note
patches/0003-*encodes line counts in two places — the hunk header(
@@ -0,0 +1,N @@) and the commit-message diffstat. Both had to move 82 →117. A stale count fails at STEP 3 of the build, hours before anything here
would be reached. Verified with
git apply --checkagainst a clean tree.Status
Re-fired as
chromeless-build-c8d— using this branch'''s ownfire-build.sh, which verified the live Job spec matches3c5fbd8andlists all nine targets before starting. Still
NINJA_KEEP_GOING=0.The two remaining never-compiled targets (
input_dispatch,pointer_state) have not linked yet either, so more of this class ispossible.
✅ Both fixes confirmed by the build lane
chromeless-build-c8d(fired via this branch'''s ownfire-build.sh,live spec verified against
3c5fbd8):All four previously-unbuilt test binaries compile and link. That is the
first time
cloud_browser_adm_unittests,cloud_browser_input_dispatch_unittestsand
cloud_browser_pointer_state_unittestshave ever been produced by anylane.
Worth separating which fix did what, since it was not obvious in advance:
input_dispatch/pointer_statelinked with no new work — the//base→public_depschange (PR #57, merged earlier today) was alreadysufficient for them. Their long absence from the lane was very likely caused
by that defect, exactly as suspected.
adm_unittestsneeded both of today'''s fixes:PullRenderDatatocompile, and the
:metrics/:system_timepassthrough deps to link.So the ten-week gap was hiding two independent defects, of two different
kinds (an abstract class and a link-time gap), in code that had been reviewed
and merged.
Build is still running (worker at 263/654). STEP 7 will execute all six test
binaries — that result will be posted here.
STEP 7 ran all six binaries — 89 tests, first execution ever for four of them
89 tests executed, all passing. Including C4'''s portal-dialect tests,
which now have run on real hardware rather than only under vitest:
The one remaining failure is the build image, not the code
The ADM test links the real libwebrtc Linux audio device module, which drags
in the X client libraries. The build container installs compilers and codec
headers but no runtime X libs — enough to compile everything, not enough
to run this one binary.
That gap was invisible for the same reason as everything else in this PR: for
~10 weeks STEP 7 ran only two binaries, and neither touches X.
Worth noting the failure SHAPE, because it is misleading: the binary exits
127 before
main(), so STEP 7 reportsunit tests failedwhile zero testshave run. That reads exactly like a code defect and is not one. (Same family
as the "green by skipping" traps this branch has been chasing — here the
verdict is red for a reason unrelated to the assertions.)
Fixed by adding the runtime libs to the manifest'''s apt line. Only
libx11-6is strictly required today; the rest of the standard Chromiumheadless set is included because the next test target that links more of the
browser will need it, and each discovery costs a 12-minute build. Package
names verified against
debian:bookworm-slimbefore firing, rather thanfinding a typo 12 minutes in.
With libX11 present, cloud_browser_adm_unittests finally started — and immediately hit the next layer of the same gap: Can't load libpulse.so.0 : cannot open shared object file failed to initialize PulseAudio M55-R2-VERDICT: {"verdict":"FAIL","reason":"CreateCloudBrowserNative AudioDeviceModule returned nullptr — likely patches/0003 audio_device_impl widening needed (see R1 TODO)"} That reason string is wrong in this environment, and fixing that is the whole point of this commit. cb_audio_device_module_test.cc is a deliberate RED-first test whose file bottom carries an escalation table keyed on WHICH failure it observes: ADM nullptr -> widen patches/0003 first RecordingDevices()==0 -> escalate to choice (a), a libpulse dynamic loader no samples / low RMS -> escalate to (a) Without a libpulse client library the test can only ever report the first row, so the table cannot discriminate and the RED is unusable as evidence in either direction. Worse, it actively points at the wrong fix: someone acting on that verdict would go widen patches/0003 to solve a missing .so. The test needs a reachable SERVER, not just the library — it drives the real libwebrtc Pulse backend — so the daemon is started too. Placement is load-bearing and was verified in a throwaway pod rather than guessed: as root: "This program is not intended to be run as root (unless --system is specified)" -> refuses as builder: Server String: /tmp/pulse-.../native Server Protocol Version: 35 -> live Hence the start lives INSIDE the `su builder` block, not in the root preamble above it. Same uid matters twice: the daemon refuses root, and the client locates the server through that user's runtime dir. Best-effort by design (`|| echo WARN`): a missing audio daemon must never fail a compile lane. If it doesn't come up the ADM test still runs and still reports RED, just with the pre-existing ambiguity. To be explicit about what this does NOT do: the ADM test will still be RED. It is designed to be until the M5.5 audio work lands. This makes its RED mean what the escalation table says it means. All three package names verified against debian:bookworm-slim first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>The ADM verdict now discriminates — and it moved
With
libpulse0+ abuilder-owned PulseAudio daemon present, the M5.5 testreports something different, which was the entire point of that commit:
Before:
ADM construction returned nullptr→ escalation table row 1 → "widenpatches/0003audio_device_impl".Now: the ADM constructs,
Init()succeeds,RegisterAudioCallback()succeeds, and
InitRecording()returns -1. That is a different row entirely.The old verdict was pointing at the wrong fix — someone acting on it would have
gone patching gn deps to solve a missing
.so.MonitorSourceDeliversToneRmsnever ran (gtest stops the fixture at the firstASSERT_), so the RMS check — the one the spec calls load-bearing — is stillunmeasured.
Status of this test: RED by design, and that is correct
cb_audio_device_module_test.ccis a deliberate RED-first test with anescalation table in its file bottom. It is supposed to fail until the M5.5
audio work lands. Nothing in this PR claims to make it green.
What changed is that its RED is now usable as evidence: it names a real
next step (why does the Pulse backend reject
InitRecording()on theserver-default source — no source in the build container? permissions? the
choice-(a) dynamic-loader escalation?) instead of a fabricated one.
Everything else passes
105 tests across six binaries, four of which no lane had ever built.
Open question for review
STEP 7 is fatal-on-failure (deliberately — that was PR #56's point). With a
RED-by-design test now in the lane, the lane cannot go green until M5.5 lands.
Options, in my order of preference:
a real audio server exists (the runtime image already runs PulseAudio under
supervisord). It is an integration test wearing a unit test's clothes.
permanently-red lane is how the last ten-week gap survived.
CHROMELESS_TESTS_NONFATAL=1— rejected. That is exactly the "temporarilynon-blocking became permanent" failure PR #56 removed.
I have not picked one; it changes what the lane means, so it wants a decision
rather than a default.
The lint caught a live regression, within hours, from a PR I merged myself
Merging current
maininto this branch — routine, to keep the green CImeaningful — produced this on the first
make verify:C3 (#55, merged earlier today) added a
test()target and put it in no buildmanifest. That is precisely the defect this PR exists to prevent, reintroduced
within hours of the guard being written, by a change I reviewed and merged.
The previous instance of this class took ten weeks to surface and only did
so because someone went looking. This one took one
make verify, and it is abetter argument for the lint than any example I could have constructed
deliberately.
Fixed by adding the target to the x264-t7 lane — now 7 of 8 (
pcf_unittestsstays t2-only by design, which the lint deliberately permits). The lane's
inventory comment is updated too; it enumerates every declared test target and
would otherwise drift from the truth it documents, which is a small version of
the same failure.
Coverage now:
Branch is now current with main and re-running CI.