fix(release): retain guest source ancestry after the audio squash #116

Merged
triform-admin merged 18 commits from agent/atlas-release-ancestry-20260911 into main 2026-09-11 10:36:27 +00:00

The recorded guest source 9253939694 was integrated through squash PR #113, but lost its ancestry on main. That makes the unchanged Triform provenance guard reject the published image.

This ordinary merge restores the already-integrated PR head f15c6e60. Its tree is exactly main afbf22ef: 4f97e1f814. Independently reconstructing PR #113 from d4725c70 and f15c6e60 yields 1814285080, exactly its squash result 22296115. No file, image, digest, release record or running workload changes. No known-gap exception is added.

Validation: make verify passed (including 345 client tests, Go modules/integration and Python/shell checks); the guest source is an ancestor of this head and the tracked-file diff from main is empty. Chromium was not rebuilt; this changes Git ancestry only. Merge using a merge commit, preserving both parents; squashing this repair would recreate the defect. CI must pass before merge.

The recorded guest source 92539396945c was integrated through squash PR #113, but lost its ancestry on main. That makes the unchanged Triform provenance guard reject the published image. This ordinary merge restores the already-integrated PR head f15c6e60. Its tree is exactly main afbf22ef: 4f97e1f8147ccee8423d39196d62ce97323ffbd5. Independently reconstructing PR #113 from d4725c70 and f15c6e60 yields 18142850802f74063eb226d644255596a6664ac4, exactly its squash result 22296115. No file, image, digest, release record or running workload changes. No known-gap exception is added. Validation: make verify passed (including 345 client tests, Go modules/integration and Python/shell checks); the guest source is an ancestor of this head and the tracked-file diff from main is empty. Chromium was not rebuilt; this changes Git ancestry only. Merge using a merge commit, preserving both parents; squashing this repair would recreate the defect. CI must pass before merge.
Measured on the live guest rather than reasoned about, and the first
thing it overturns is a line in this very document.

The finding records theory 3's pre-arm as "SUCCEEDED ...
StartRecording=0 now_recording=1 — and the session was still silent".
The guest says StartRecording=-1, now_recording=0, on all 231
occurrences. So theory 3 was never refuted by "the pre-arm worked and
audio failed anyway" — the pre-arm has never worked, and the real
question was always why StartRecording fails after a successful
InitRecording.

The answer: the record thread is GONE. Init spawns two threads
(rec + play, both truncated to "webrtc_audio_mo" in /proc); the live
process has one. A thread ends only when RecThreadProcess returns
false, which happens at exactly one line — `if (quit_) return false`
— and quit_ is written only by Terminate() and never cleared.

So Terminate() ran on the shared ADM after the first session, and
every later StartRecording signals _timeEventRec and waits for a
thread that no longer exists.

That also explains the 4 ms this document called its sharpest clue.
The 10-second Wait is not timing out: it returns, finds _recording
false, and takes the SECOND error path — which logs the same string as
the timeout path, which is exactly why the two were indistinguishable.

Two theories are now refuted by source rather than opinion:

- The stale-_recStartEvent theory cannot hold: Event() is
  Event(false,false), i.e. AUTO-reset (rtc_base/event.cc:39), so a
  successful Wait consumes the signal.
- My own ordering theory — that RemoveSendingStream's StopRecording
  undoes the pre-arm — is plausible from source and refuted by the log
  ordering: the failure is already inside the pre-arm, 40us before our
  line, and before the teardown that would trigger it.

Still open: WHO calls Terminate(). Nothing in capture/ does. The
ADM's destructor does, and the PCF holds it by scoped_refptr and is
built once, so a plain refcount drop should not happen. The next probe
is written down and is a pointer-identity check, not another fix
attempt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
THE cause of one-shot audio, found by measurement and fixed upstream
rather than worked around a fifth time.

AudioDeviceLinuxPulse::Terminate() sets quit_ = true and joins the
record and playout threads. Init() spawns a fresh pair and NEVER
resets quit_, so both new threads evaluate `if (quit_) return false;`
on their first wakeup and exit. Init() returns OK with no threads.

StartRecording() then signals _timeEventRec and waits on
_recStartEvent for a record thread that no longer exists. The wait
returns, _recording is still false, and it takes its SECOND failure
path — which logs the same "failed to activate recording" string as
the 10-second timeout path. That is why the 4 ms failure this defect
was named after looked like a stale event: two different failures
wearing one message.

Measured on the live worker, not inferred: exactly ONE
"webrtc_audio_mo" thread where Init() spawns two, and StartRecording
returning -1 in microseconds having just had InitRecording return 0.

This also corrects TWO things I asserted earlier today:

- I claimed quit_ had latched and the thread was gone for good. Wrong:
  the surviving thread's tid CHANGES between samples, so Init() is
  running again — the threads are respawned and immediately die, which
  is a different fault with a different fix.
- I claimed RemoveSendingStream's StopRecording undoes the pre-arm.
  Plausible from source, refuted by log ordering: the failure is
  already inside the pre-arm, 40us before the teardown that would
  cause it.

Why a patch rather than a workaround: quit_ is private, written only
by Terminate(), and cleared by nothing. No public API can reach it, so
none of the four in-tree attempts could have worked — which the
finding now records.

Verified the patch APPLIES, against a copy of the real 2304-line file
pulled from the pinned tree. Worth recording that the first version
was hand-written and `git apply` rejected it as "corrupt patch at line
92" — the hunk header counts were wrong. Regenerated from a real `git
diff` and it applies cleanly. Also checked Init() takes no other lock
(no double-lock) and that Terminate() at :218 uses the identical
MutexLock idiom.

UNVERIFIED: the lane has not run. The test is tests/interactive twice
against ONE worker process — the second run is the real one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lane rejected 0006:

  error: sha1 information is lacking or useless (...pulse_linux.cc).
  error: could not build fake ancestor
  Patch failed at 0001 pulse ADM: reset quit_ in Init()

Not a corrupt diff. `git diff` emitted an `index <sha>..<sha>` line
naming blob hashes from the scratch repo I generated it in; those
hashes mean nothing in the Chromium tree, so `git am` tried to
reconstruct an ancestor from them and gave up.

None of 0002, 0003 or 0005 carry an index line. Mine did because I
generated it with plain `git diff` in a throwaway repo rather than
matching the series' existing shape — the convention was right there
in three files and I did not check it.

My verification was also wrong in a way worth recording: I tested with
`git apply --check`, which PASSED. build.sh uses `git am
--keep-non-patch`, which does not. Testing the wrong command is the
same class of mistake as a silent no-op mutation — the check ran, it
just was not the check that mattered.

Now verified through `git am --keep-non-patch` against a copy of the
real 2304-line file, and patches/README.md records both traps so the
next patch author does not pay for them again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lane rejected 0006 twice with

  error: sha1 information is lacking or useless (...pulse_linux.cc).
  error: could not build fake ancestor

and my first diagnosis — a stray `index` line — was WRONG. Removing it
changed nothing, which is the useful half: it refuted the theory in one
cycle.

The real cause: DEPS clones sub-repos INTO src/, each with its own
.git. third_party/webrtc is one (DEPS:3012). `git am` runs from src/,
whose index does not track that path, so --3way has no blob to build an
ancestor from. Patches 0002/0003/0005 never hit this because they touch
content/ and third_party/webrtc_overrides/, both genuinely in the src
repo. 0006 is the first patch to reach into a DEPS checkout.

apply-patches now reads the target path out of each patch and applies
from whichever checkout owns it, stripping the sub-repo prefix into a
temp copy rather than rewriting the patch — the patch stays readable
against the chromium tree it documents.

Proven with a control arm, not just a green run: a local fixture with
two nested git repos reproduces the lane's failure exactly when applied
from src/, and applies cleanly when routed. Both arms recorded.

Two verification mistakes of mine that this cost, both now in the
README:

- I checked with `git apply --check`, which passes on patches `git am`
  rejects. Testing the wrong command is the same class as a silent
  no-op mutation: the check ran, it just was not the check that
  mattered.
- I then "verified" against a test repo where a previous `git am` had
  already COMMITTED the change — so I was diffing a patched file
  against a pristine tree and reading the mismatch as evidence about
  the patch. An md5 against the real tree is what caught it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(build): reset the sub-repo too — otherwise the NEXT fire fails identically
Some checks failed
CodeQL / Analyze go (pull_request) Has been skipped
Public security / secrets (pull_request) Has been skipped
CodeQL / Analyze javascript-typescript (pull_request) Has been skipped
CI / Docs link check (pull_request) Successful in 14s
native-peer-gate / native-peer-gate-scaffold (permissive) (pull_request) Successful in 52s
CI / Container smoke test (pull_request) Successful in 1m40s
native-peer-gate / native-peer-gate-strict (M7 gate) (pull_request) Successful in 12s
Public security / ownership (pull_request) Successful in 6s
CI / Lint (pull_request) Has been cancelled
E2E / docker-compose + Playwright (pull_request) Has been cancelled
cd0f2b966d
Found by testing the fix rather than trusting it. Four consecutive
apply-patches rounds against a fixture with the real pinned M147
pulse source: round 1 applied, round 2 died with

  error: sha1 information is lacking or useless (...pulse_linux.cc)
  error: could not build fake ancestor

— the SAME two lines as the wrong-repo bug the previous commit fixed,
from a completely unrelated cause, and it left .git/rebase-apply behind
to poison round 3.

This is not a retry-only concern. apply-patches runs on EVERY lane fire
and the chromium tree is a hostPath that outlives the Job, so the fire
after the one that first lands 0006 hits it. Had I only watched the
current build go green, the next person to fire the lane would have
inherited an error message pointing at a bug that was already fixed.

The upfront recovery resets ${CHROMIUM_SRC} to the LKGM base and does
nothing to third_party/webrtc, whose HEAD keeps whatever the last run
committed. Reset it by authorship: walk down to the first commit not
authored by us, which is exactly the DEPS-pinned revision whatever
gclient chose — no second sha to drift from DEPS. A tree with none of
our commits resets to HEAD, a no-op.

Failing loudly beats failing confusingly: if every commit looks like
ours, apply-patches now dies with a message naming the cause instead of
applying a patch onto itself and reaching "fake ancestor" ~30 min later.

Two bugs the harness caught in my own code, neither visible from a
green run:

- The log said "reset v8" while resetting third_party/webrtc. ${sub}
  outside the for-loop holds the LAST iterated value, not the matched
  one. Three sites; all now use strip_prefix, set at the match.
- My first harness sliced build.sh mid-`if`, so `git am` never ran and
  three "rounds" proved nothing about applying. The rounds now run the
  real block, sliced to its closing line and asserted to contain the am.

Verified: 4/4 rounds rc=0, converging on one commit and one applied
hunk; the loud-failure path fires when the base is authored by us.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs(roadmap): the prescribed audio fix was the bug — and a line was duplicated
Some checks failed
CodeQL / Analyze javascript-typescript (pull_request) Has been skipped
CodeQL / Analyze go (pull_request) Has been skipped
Public security / secrets (pull_request) Has been skipped
CI / Docs link check (pull_request) Successful in 12s
CI / Container smoke test (pull_request) Successful in 29s
native-peer-gate / native-peer-gate-scaffold (permissive) (pull_request) Successful in 11s
native-peer-gate / native-peer-gate-strict (M7 gate) (pull_request) Successful in 14s
Public security / ownership (pull_request) Successful in 7s
CI / Lint (pull_request) Successful in 4m25s
E2E / docker-compose + Playwright (pull_request) Has been cancelled
7c87a9b88d
Two defects in the Track 2 "Also:" paragraph.

The line was duplicated verbatim on main (roadmap-ga.md:200-201), so the
list read twice.

The substantive one: the roadmap prescribed "tear the pulse ADM down on
re-arm so the second viewer gets audio". Teardown-and-reinit is exactly
what triggers the defect — Terminate() sets libwebrtc's quit_, Init()
never clears it, and the fresh record thread exits on its first wakeup.
Following this instruction would have reproduced the symptom more
reliably, then sent the next person hunting in capture/ for a bug that
is in third_party/webrtc. The "audio PrepareForTeardown hook on an
inbound bye", listed on the same line, does the same thing on a
different schedule.

Left as a correction with its reasoning rather than a silent deletion:
the wrong fix is plausible enough that someone would propose it again.

Fixed in patches/0006 (PR #113).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The status table already listed 3=cancelled; what it did not say is that
the API reports those as `failure`, so pushing again while a run is in
flight produces a "CI Lint failed" alert about your own push.

Cost a detour today: the alert fired, the log ended in `Job failed`, and
the failing step above it was 345/345 client tests PASSING. The tell is
`context canceled` with no failing step.

Worth the row because it is not an edge case — 811 of this repo's jobs
are cancelled against 223 real failures, so a status read as failure is
more often 3 than 2.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two corrections to my own finding.

It said "the fix shape depends on the answer and is NOT knowable yet",
pending an answer to who calls Terminate(). That was wrong, and I wrote
patches/0006 without going back to fix it. Reading the pinned source
settles the fix without the caller: Terminate() clears _initialized and
sets quit_; Init() passes its _initialized early-out, spawns both
threads, and never resets quit_. Init() is therefore not idempotent
after a Terminate() for ANY caller, and fixing that is correct whoever
calls it. Recorded the placement reasoning too — the reset goes AFTER
the early-out so an already-initialised Init() cannot clear the latch
out from under a shutdown in progress.

The caller question stays open, demoted from blocker to loose end: a
patch that survives Terminate() is not the same as knowing why
Terminate() runs.

The second correction is the more useful one. cb_audio_lifecycle.cc:206
already said, in a comment predating the whole investigation:

  Terminate() is not an escape either: it sets quit_ and NOTHING ever
  clears it (grep says one write, no reset), so a Terminate/Init cycle
  would kill the record thread for the life of the process.

The defect, named exactly, in the tree, before the symptom was
diagnosed. The author reached the right conclusion and routed around it
rather than patching libwebrtc — reasonable — but it means the pre-arm
was deliberately designed not to call Terminate(), which sharpens the
open question about who does.

Cited it in the finding so the next person to consider a Terminate/Init
cycle gets the answer in advance instead of re-deriving it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs(findings): the defect is readable off a running pod in ten seconds
All checks were successful
CodeQL / Analyze go (pull_request) Has been skipped
CodeQL / Analyze javascript-typescript (pull_request) Has been skipped
Public security / secrets (pull_request) Has been skipped
CI / Docs link check (pull_request) Successful in 11s
CI / Container smoke test (pull_request) Successful in 43s
native-peer-gate / native-peer-gate-scaffold (permissive) (pull_request) Successful in 28s
Public security / ownership (pull_request) Successful in 4s
native-peer-gate / native-peer-gate-strict (M7 gate) (pull_request) Successful in 25s
E2E / docker-compose + Playwright (pull_request) Successful in 2m7s
CI / Lint (pull_request) Successful in 5m52s
803702942a
Init() spawns two audio-module threads; the live unpatched worker has
one. That is the bug itself, not a downstream symptom, and it needs no
session, no viewer and no bytesReceived reading — measured on pid 23 of
the 9h-old worker today.

Better oracle than the interactive suite's audio check, which goes red
for a silent ADM and for every unrelated transport fault alike: 2
threads with no audio is a different bug, and now distinguishable.

Includes the process-selection recipe, because getting it wrong returns
a plausible number rather than an error. `pgrep -f cb-chromium` matches
nothing (the binary is /usr/local/bin/chromeless); a loose cmdline match
finds the shell doing the searching, which shows up as three consecutive
calls returning three different pids.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chore(guest): record cr7727-92539396945c — the pulse ADM fix
Some checks failed
Public security / ownership (pull_request) Waiting to run
CodeQL / Analyze go (pull_request) Has been skipped
CodeQL / Analyze javascript-typescript (pull_request) Has been skipped
Public security / secrets (pull_request) Has been skipped
CI / Docs link check (pull_request) Successful in 7s
native-peer-gate / native-peer-gate-scaffold (permissive) (pull_request) Has been cancelled
native-peer-gate / native-peer-gate-strict (M7 gate) (pull_request) Has been cancelled
E2E / docker-compose + Playwright (pull_request) Has been cancelled
CI / Container smoke test (pull_request) Has been cancelled
CI / Lint (pull_request) Has been cancelled
141d259d62
Lane green at 9253939: patches applied via the new sub-repo routing,
671 targets compiled, 8 test targets built and linked.

  image  registry.triform.cloud/chromeless/chromeless:cr7727-92539396945c
  digest sha256:c6603ff6dd65eb5d12f1eae20880a6a70d7c61b389da28dcb1c8a619fd6e76a9

Digest read back from the registry with `crane digest` and compared to
what kaniko reported, then checked against ^sha256:[0-9a-f]{64}$ — an
empty digest is accepted by `kubectl set image` and lands the worker in
InvalidImageName, which cost a deploy earlier in this work.

Tagged 9253939 rather than branch head: the two commits since are docs
plus the sub-repo RESET in build.sh, which changes how patches are
applied, not what they apply to. The 4-round fixture shows routing and
routing+reset converge on the identical tree, so the binary is the same
either way — but the tag names the commit whose build script actually
ran.

Both pin files in one commit. Committing only the json is what made main
drift before; lint-deploy-pin and lint-guest-release both clean.

The image is NOT yet known to fix the defect. That needs two interactive
runs against one worker process.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs(findings): match the thread name exactly — grep -c audio counts three
Some checks failed
CodeQL / Analyze javascript-typescript (pull_request) Has been skipped
CodeQL / Analyze go (pull_request) Has been skipped
Public security / secrets (pull_request) Has been skipped
CI / Docs link check (pull_request) Successful in 5s
CI / Container smoke test (pull_request) Successful in 37s
native-peer-gate / native-peer-gate-strict (M7 gate) (pull_request) Successful in 20s
Public security / ownership (pull_request) Successful in 6s
native-peer-gate / native-peer-gate-scaffold (permissive) (pull_request) Successful in 34s
CI / Lint (pull_request) Has been cancelled
E2E / docker-compose + Playwright (pull_request) Has been cancelled
6934a6f6b0
The recipe I added an hour ago used `grep -c audio` over
/proc/<pid>/task/*/comm. The browser also carries "AudioEncoderQue" and
"AudioDeviceBuff", so case-insensitively that reads 3 — and would keep
reading 3 after the fix.

The case-sensitive form gave the right answer, 1, but only because it
accidentally excluded two capitalised names. Right answer, wrong reason:
it would start lying the moment a lowercase audio thread appeared, and
an oracle that moves for reasons unrelated to the defect is not one.

Now `grep -hx webrtc_audio_mo`. Ran the documented command verbatim
against the live pod: returns 1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs(findings): name the acceptance test precisely, including how it can lie
Some checks failed
Public security / ownership (pull_request) Waiting to run
CodeQL / Analyze go (pull_request) Has been skipped
Public security / secrets (pull_request) Has been skipped
CodeQL / Analyze javascript-typescript (pull_request) Has been skipped
CI / Docs link check (pull_request) Successful in 6s
CI / Lint (pull_request) Has been cancelled
CI / Container smoke test (pull_request) Has been cancelled
E2E / docker-compose + Playwright (pull_request) Has been cancelled
native-peer-gate / native-peer-gate-scaffold (permissive) (pull_request) Has been cancelled
native-peer-gate / native-peer-gate-strict (M7 gate) (pull_request) Has been cancelled
6d88383dd9
The section said "run twice against the same worker process" and named
only the e2e spec. Three gaps, each of which can produce a false pass:

- The interactive suite is what actually runs against the cluster and has
  the equivalent check; it was not mentioned. Both drivers listed now.
- Nothing said to verify the browser pid is UNCHANGED between runs. The
  worker serves one session and supervisord respawns it in ~15 s, so run
  2 can land on a fresh process and pass as a first viewer while looking
  like a re-arm — the precise false pass the two-run method exists to
  avoid.
- No way to tell "the fix failed" from "audio broke for another reason".
  The thread count separates them: 2 threads with no audio bytes is a
  different bug and should not be charged to this finding.

Also states that a boot-time reading is 2 on patched and unpatched alike,
so it is a sanity check and never the verification.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs(findings): the resize confounder is already blocked two ways — check, don't assume
All checks were successful
CodeQL / Analyze go (pull_request) Has been skipped
CodeQL / Analyze javascript-typescript (pull_request) Has been skipped
Public security / secrets (pull_request) Has been skipped
CI / Docs link check (pull_request) Successful in 5s
CI / Container smoke test (pull_request) Successful in 55s
CI / Lint (pull_request) Successful in 2m59s
Public security / ownership (pull_request) Successful in 5s
native-peer-gate / native-peer-gate-strict (M7 gate) (pull_request) Successful in 48s
native-peer-gate / native-peer-gate-scaffold (permissive) (pull_request) Successful in 4m43s
E2E / docker-compose + Playwright (pull_request) Successful in 7m48s
8ee96d20b0
The section told the reader to block */api/viewport in the test browser.
On the cluster today that is unnecessary: the interactive suite never
calls the route, and the deployed gateway runs
CHROMELESS_VIEWPORT_FOLLOW=0, which answers it 501. Both verified.

Kept the instruction for hand-rolled drivers that do drive the client's
resize path, and added the thing that actually matters — confirm the env
var before trusting a run, because it flips back to 1 when the resize fix
lands and the protection silently disappears.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bytesReceived==0 is a symptom shared by a silent ADM and by every
transport fault there is, so on its own it cannot say which you have.
The thread count can, and it reads the SEND side directly instead of
inferring it from the receive side.

AudioDeviceLinuxPulse::Init() spawns exactly two threads, both truncated
by the kernel to "webrtc_audio_mo". They end only by returning false,
which happens in exactly one place: `if (quit_) return false;`. So ONE
thread is the quit_ latch, precisely, and TWO with no audio bytes is a
different bug that should not be charged to that finding.

This is the check that was missing: the live worker read 1 for weeks
while every suite run was green about audio it never asserted.

Three things that make this check honest rather than decorative,
each verified against the live pod rather than assumed:

- The command is sent verbatim from the suite and returns '2' on the
  patched worker. Escaping through python -> kubectl -> sh is the part
  most likely to be silently wrong.
- Negative control: the same probe returns 1 for AudioDeviceBuff and 0
  for a nonexistent name. It reports what is there, not a constant.
- Matches the comm name EXACTLY. `grep -ci audio` also counts
  AudioEncoderQue and AudioDeviceBuff and would read 3 in every state.

Also fixed a scope bug in my own first draft: ns/dep are locals of
suite_downloads and suite_permissions, not module globals, and this
check lives in suite_stats — it would have raised NameError on the first
run. Caught by walking the function's assignments, not by reading it.

Process selection needs the same care as the thread name: the binary is
/usr/local/bin/chromeless, so `pgrep -f cb-chromium` finds nothing and a
loose match finds the shell doing the searching.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(deploy): discover the TURN relay — the hardcoded IP is dead
Some checks failed
CodeQL / Analyze go (pull_request) Has been skipped
CodeQL / Analyze javascript-typescript (pull_request) Has been skipped
Public security / secrets (pull_request) Has been skipped
CI / Docs link check (pull_request) Successful in 5s
CI / Container smoke test (pull_request) Successful in 22s
native-peer-gate / native-peer-gate-scaffold (permissive) (pull_request) Successful in 10s
native-peer-gate / native-peer-gate-strict (M7 gate) (pull_request) Successful in 12s
Public security / ownership (pull_request) Successful in 5s
CI / Lint (pull_request) Has been cancelled
E2E / docker-compose + Playwright (pull_request) Has been cancelled
9e6013683d
TURN_IP=95.217.200.179 sat directly beneath a comment block explaining
how to tell "the relay moved" from "the credential expired". By
2026-09-10 that address refused 3478 outright — coturn had moved to
triform-7 — so deploy.sh would have deployed a dead relay and reported
success, which is precisely the failure its own comment describes.

Now: discover the node from the cluster, then PROVE the answer by
dialling 3478 before using it. A relay that is merely NAMED is the exact
failure mode the comment warns about — the stack comes up, negotiates,
and shows no video, with no error anywhere.

Two rounds of getting the selector wrong, both of which returned a
plausible node rather than an error:

- "first Running coturn pod" is a coin flip. There are two deployments,
  `coturn` on triform-7 and `coturn-2` on triform-8, both answering
  3478, so BOTH halves of the flip pass a reachability test. If the two
  ever hold different static-auth secrets, the wrong pick refuses every
  allocation while looking healthy. (Checked: both currently run
  lt-cred-mech + use-auth-secret with realm=triform.cloud, so either
  would work today. Luck, not a guarantee, and invisible from the pod
  list.)
- Every name-prefix pattern I tried still matched coturn-2, including
  ^coturn-[0-9a-f]+- — "2" is a hex digit. Selecting on the `app=` label
  is exact; verified it resolves to 116.202.84.152, the address the
  working live config actually uses.

TURN_NS / TURN_DEPLOY / TURN_IP override for an external relay.

Found because `kubectl apply -f stack.yaml` on its own silently dropped
the worker's WEBRTC_ICE_SERVERS — stack.yaml does not define it,
deploy.sh sets it afterwards — leaving the guest on public STUN with no
relay. The guest log said `server urls=[stun:stun.l.google.com:19302]`
and the client gave up after ~35 s without a frame, which the interactive
harness reported as "the worker already served a session". It had served
none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test(interactive): "never decoded a frame" has a third cause — no TURN relay
All checks were successful
CodeQL / Analyze go (pull_request) Has been skipped
CodeQL / Analyze javascript-typescript (pull_request) Has been skipped
Public security / secrets (pull_request) Has been skipped
CI / Docs link check (pull_request) Successful in 7s
CI / Container smoke test (pull_request) Successful in 22s
native-peer-gate / native-peer-gate-scaffold (permissive) (pull_request) Successful in 11s
Public security / ownership (pull_request) Successful in 5s
native-peer-gate / native-peer-gate-strict (M7 gate) (pull_request) Successful in 13s
E2E / docker-compose + Playwright (pull_request) Successful in 3m13s
CI / Lint (pull_request) Successful in 4m5s
5d7f8e62fa
The error listed two causes and asserted "NEITHER is a bug in what you
are testing". A missing relay is a third, it IS a deployment fault, and
its wording sent me to check session slots while the guest log was
saying `server urls=[stun:stun.l.google.com:19302]` — public STUN, no
relay, no traversal, client gives up ~35 s in with zero frames.

All three look identical from the client: zero frames after the full
timeout. So the message now leads with the guest's own log, which
separates them in one grep, and lists the ICE cause first.

The trap worth naming: the worker carries its OWN copy of
WEBRTC_ICE_SERVERS, stack.yaml does not define it, and deploy.sh sets it
AFTER applying the manifest. So `kubectl apply -f stack.yaml` on its own
— which is what you reach for to change an image pin — silently strips
the relay from a working deployment. That is how I hit it.

An expired credential presents identically from the client side, so the
message says so rather than implying an empty env var is the only shape.

Both printed commands run as written against the live pod; with the
relay restored the same grep now reads `servers=2` and names the TURN
address.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs(findings): FIXED AND VERIFIED — audio survives the re-arm
All checks were successful
CodeQL / Analyze javascript-typescript (pull_request) Has been skipped
CodeQL / Analyze go (pull_request) Has been skipped
Public security / secrets (pull_request) Has been skipped
CI / Docs link check (pull_request) Successful in 5s
native-peer-gate / native-peer-gate-scaffold (permissive) (pull_request) Successful in 33s
CI / Container smoke test (pull_request) Successful in 49s
native-peer-gate / native-peer-gate-strict (M7 gate) (pull_request) Successful in 33s
Public security / ownership (pull_request) Successful in 5s
CI / Lint (pull_request) Successful in 3m9s
E2E / docker-compose + Playwright (pull_request) Successful in 4m11s
f15c6e60bb
Three consecutive viewer sessions against ONE browser process (pid 23
before and after each), each receiving ~22 kB of audio at the client.
Every previous second session received zero.

Guest-side ledger for that process: 6 re-arms, 6 pre-arms reporting
StartRecording=0 now_recording=1, zero StartRecording=-1, zero "failed
to activate recording". Both webrtc_audio_mo threads alive throughout.

Status banner at the top; the body stays as it is. The refuted theories
and the corrections to its own premise are the useful part of this file,
and deleting them would leave only the answer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(release): preserve the guest source ancestry from PR 113
All checks were successful
CodeQL / Analyze javascript-typescript (pull_request) Has been skipped
CodeQL / Analyze go (pull_request) Has been skipped
Public security / secrets (pull_request) Has been skipped
CI / Docs link check (pull_request) Successful in 18s
CI / Container smoke test (pull_request) Successful in 34s
native-peer-gate / native-peer-gate-strict (M7 gate) (pull_request) Successful in 11s
Public security / ownership (pull_request) Successful in 5s
native-peer-gate / native-peer-gate-scaffold (permissive) (pull_request) Successful in 1m33s
CI / Lint (pull_request) Successful in 4m22s
E2E / docker-compose + Playwright (pull_request) Successful in 6m42s
fa94cffe01
PR 113 squashed the audio-rearm branch into 22296115. Reconstructing its merge from d4725c70 and f15c6e60 yields the exact squash tree 1814285080. Merge that already-integrated branch normally so recorded guest source 92539396 is again reachable from main. The result tree equals current main afbf22ef (4f97e1f814): no source, release record, artifact pin or runtime changes. This repairs the provenance rejection in tf-multiverse task 583900 without adding a known-gap exception.
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
triform/chromeless!116
No description provided.