docs(conformance): the README advertised two things the kit does not do #53

Merged

triform-admin merged 1 commit from fix/conformance-readme-media into main

2026-07-30 15:58:43 +00:00

triform-admin commented

2026-07-30 15:52:07 +00:00

Owner

Self-review of what I merged in #51. The conformance README advertised two things the kit does not do.

1. A media role that doesn't exist. The suites table lists media / media-sanity, but run.mjs's ROLES map is {signaling, cdp, full}. So --role=media exits 2 with "must be one of signaling | cdp | full" — a reader following the table hits a usage error on their first command.

2. "Cb.* presence" attributed to the cdp suite. It doesn't check that. The only mentions of Cb.* in cdp-surface.mjs are comments explaining why /json/protocol — the natural way to enumerate them — is deliberately not probed, because it CHECK-FATALs the worker.

Fixed by making the doc match the code, not by adding hand-wave checks to match the doc. The media suite is now described as not-implemented, with why (it needs a real PeerConnection against the running deployment — a bigger lift that belongs with wiring verification/'s boot harness into a live media path) and what happens meanwhile (usage error, not a silent no-op).

A conformance kit whose own README overstates its coverage is precisely the failure mode it exists to catch. Worth fixing before anyone runs it.

node --test conformance/__tests__/*.test.mjs still 23/23.

Self-review of what I merged in #51. The conformance README advertised two things the kit does not do. **1. A `media` role that doesn't exist.** The suites table lists `media` / `media-sanity`, but `run.mjs`'s `ROLES` map is `{signaling, cdp, full}`. So `--role=media` exits 2 with *"must be one of signaling | cdp | full"* — a reader following the table hits a usage error on their first command. **2. "`Cb.*` presence"** attributed to the cdp suite. It doesn't check that. The only mentions of `Cb.*` in `cdp-surface.mjs` are comments explaining why `/json/protocol` — the natural way to enumerate them — is deliberately *not* probed, because it CHECK-FATALs the worker. Fixed by making the doc match the code, not by adding hand-wave checks to match the doc. The media suite is now described as not-implemented, with **why** (it needs a real PeerConnection against the running deployment — a bigger lift that belongs with wiring `verification/`'s boot harness into a live media path) and **what happens meanwhile** (usage error, not a silent no-op). A conformance kit whose own README overstates its coverage is precisely the failure mode it exists to catch. Worth fixing before anyone runs it. `node --test conformance/__tests__/*.test.mjs` still 23/23.

triform-admin added 1 commit

2026-07-30 15:52:08 +00:00

docs(conformance): the README advertised two things the kit does not do

CodeQL / Analyze javascript-typescript (pull_request) Has been skipped

Details

CodeQL / Analyze go (pull_request) Has been skipped

Details

E2E / docker-compose + Playwright (pull_request) Successful in 35s

Details

CI / Docs link check (pull_request) Successful in 47s

Details

native-peer-gate / native-peer-gate-scaffold (permissive) (pull_request) Successful in 1m1s

Details

CI / Container smoke test (pull_request) Successful in 1m49s

Details

native-peer-gate / native-peer-gate-strict (M7 gate) (pull_request) Successful in 2m33s

Details

CI / Lint (pull_request) Successful in 5m14s

Details

f77c7e85e8

Self-review of what I just merged in #51. The suites table claimed:

  * a `media` role backed by a `media-sanity` suite. Neither exists —
    run.mjs's ROLES map is {signaling, cdp, full}, so `--role=media` exits 2
    with "must be one of signaling | cdp | full". A reader following the
    table hits a usage error on their first command.
  * that the cdp suite asserts "`Cb.*` presence". It does not. The only
    mentions of Cb.* in cdp-surface.mjs are comments explaining why
    /json/protocol — the natural way to enumerate them — is deliberately
    NOT probed (it CHECK-FATALs the worker).

Both fixed by making the doc match the code, not by adding hand-wave checks.
The media suite is described as not-implemented, with WHY (it needs a real
PeerConnection against the running deployment, which is a bigger lift and
belongs with wiring verification/'s boot harness into a live media path) and
what happens meanwhile (usage error, not a silent no-op).

A conformance kit whose own README overstates its coverage is the exact
failure it exists to catch. Worth fixing before anyone runs it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>