feat(conformance): a kit that answers "does MY deployment conform?" (Track 7) #51

Merged

triform-admin merged 2 commits from track7/conformance-kit into main 2026-07-30 15:43:27 +00:00

Conversation 1 Commits 2 Files changed 9 +1189

triform-admin commented 2026-07-30 14:59:56 +00:00

Owner

Copy link

Track 7 of the OSS-usability plan.

tests/ answers "does this repo's code work". Nothing answered "does your deployment satisfy the contract" — a different question, for a different person, and the one that matters now that this project is meant to work for people with none of the triform infrastructure.

node conformance/run.mjs --target=ws://localhost:8080 --role=signaling
node conformance/run.mjs --target=http://localhost:9222 --role=cdp --json

Zero dependencies (node builtins only), no reliance on this repo's fixtures, a Chromium build, or triform.

Decisions worth reviewing

• Every check names its spec clause. The registry refuses to register a check without a spec field. verification/ can assume the reader has the tree open; this can't, so "FAIL" without a citation is useless to the audience.
• required vs advisory is enforced in the verdict algebra, not by convention. Advisory failures are reported loudly but don't set the exit code — otherwise "recommended" quietly means "mandatory".
• A required ERROR counts as FAIL. A check that couldn't run hasn't demonstrated conformance; treating "couldn't tell" as "fine" is how green stops meaning anything.
• All-SKIP reports NOTHING_RAN, not PASS. A run where nothing applied proved nothing.
• The negative checks are the point. Anyone can relay a well-formed offer. The load-bearing half of cb_wire_envelope.h is what a conforming peer must refuse: sdp_offer (the historic v0 tag), bye carrying data, a missing from.
• /json/protocol is asserted NOT probed. Enumerating Cb.* that way is what a naive kit would do — and on this worker it CHECK-FATALs the browser. The check that looks most natural is the one that kills the thing under test, so the hazard is encoded as a SKIP carrying the reason, putting CLAUDE.md's trap where someone debugging a SIGABRT will find it.

Wire rules were read off cb_wire_envelope.cc's Decode/DecodeData, and the registration handshake off signaling/server.go:505-560 — the code is what a real peer does.

Verified, not assumed

• 23/23 self-tests (node --test conformance/__tests__/*.test.mjs)
• Run against a real worker (cr7727-f9e46a1434b5) on a runner's dind, inside the container's network namespace:

✓ PASS  devtools-reachable
✓ PASS  page-target-exists
· SKIP  protocol-endpoint-hazard (advisory)
verdict=PASS

…and the worker was still healthy afterwards (4/4 supervisord programs), since the hazard endpoint was never touched.

• Exit codes checked directly rather than through a pipe (which masks them): 0 help, 2 usage error, 2 bad role, 1 required-check failure.

CI wiring

Two places, for two reasons:

• Lint runs the kit's own tests — no network, no image, always gating.
• smoke runs the kit against the image it just proved boots. If the reference implementation can't pass its own conformance suite, the suite is wrong and CI should say so before a third party does.

Not included

The signaling-wire suite ships but is unexercised against a live broker — I had no two-peer session to drive in this environment. Its rules are read from the C++ decoder rather than guessed, but I'd rather say that plainly than imply a verification I didn't do. The media-sanity suite named in the README is not implemented yet.

Track 7 of the OSS-usability plan. `tests/` answers *"does this repo's code work"*. Nothing answered *"does **your** deployment satisfy the contract"* — a different question, for a different person, and the one that matters now that this project is meant to work for people with none of the triform infrastructure. ```bash node conformance/run.mjs --target=ws://localhost:8080 --role=signaling node conformance/run.mjs --target=http://localhost:9222 --role=cdp --json ``` Zero dependencies (node builtins only), no reliance on this repo's fixtures, a Chromium build, or triform. ## Decisions worth reviewing - **Every check names its spec clause.** The registry *refuses* to register a check without a `spec` field. `verification/` can assume the reader has the tree open; this can't, so "FAIL" without a citation is useless to the audience. - **required vs advisory is enforced in the verdict algebra**, not by convention. Advisory failures are reported loudly but don't set the exit code — otherwise "recommended" quietly means "mandatory". - **A required ERROR counts as FAIL.** A check that couldn't run hasn't demonstrated conformance; treating "couldn't tell" as "fine" is how green stops meaning anything. - **All-SKIP reports `NOTHING_RAN`, not PASS.** A run where nothing applied proved nothing. - **The negative checks are the point.** Anyone can relay a well-formed `offer`. The load-bearing half of `cb_wire_envelope.h` is what a conforming peer must *refuse*: `sdp_offer` (the historic v0 tag), `bye` carrying `data`, a missing `from`. - **`/json/protocol` is asserted NOT probed.** Enumerating `Cb.*` that way is what a naive kit would do — and on this worker it CHECK-FATALs the browser. The check that looks most natural is the one that kills the thing under test, so the hazard is encoded as a SKIP carrying the reason, putting CLAUDE.md's trap where someone debugging a SIGABRT will find it. Wire rules were read off `cb_wire_envelope.cc`'s `Decode`/`DecodeData`, and the registration handshake off `signaling/server.go:505-560` — the code is what a real peer does. ## Verified, not assumed - **23/23** self-tests (`node --test conformance/__tests__/*.test.mjs`) - **Run against a real worker** (`cr7727-f9e46a1434b5`) on a runner's dind, inside the container's network namespace: ``` ✓ PASS devtools-reachable ✓ PASS page-target-exists · SKIP protocol-endpoint-hazard (advisory) verdict=PASS ``` …and the worker was **still healthy afterwards** (4/4 supervisord programs), since the hazard endpoint was never touched. - Exit codes checked directly rather than through a pipe (which masks them): `0` help, `2` usage error, `2` bad role, `1` required-check failure. ## CI wiring Two places, for two reasons: - **Lint** runs the kit's own tests — no network, no image, always gating. - **smoke** runs the kit *against* the image it just proved boots. If the reference implementation can't pass its own conformance suite, the suite is wrong and CI should say so before a third party does. ## Not included The `signaling-wire` suite ships but is **unexercised against a live broker** — I had no two-peer session to drive in this environment. Its rules are read from the C++ decoder rather than guessed, but I'd rather say that plainly than imply a verification I didn't do. The `media-sanity` suite named in the README is not implemented yet.

triform-admin added 1 commit 2026-07-30 14:59:56 +00:00

feat(conformance): a kit that answers "does MY deployment conform?" ... 7a3f0475ef

Track 7. tests/ answers "does this repo's code work". Nothing answered "does
YOUR deployment satisfy the contract" — a different question, for a different
person, and the one that matters as soon as this project is usable by people
with none of the triform infrastructure. A third party running their own
broker, their own build of the worker, or a reimplementation of either had no
way to check short of reading C++.

    node conformance/run.mjs --target=ws://localhost:8080 --role=signaling
    node conformance/run.mjs --target=http://localhost:9222 --role=cdp --json

Zero dependencies (node builtins only) and no reliance on this repo's
fixtures, a Chromium build, or triform. `npm install` before you can ask "am
I conforming?" is a barrier this kit should not have.

Design decisions worth stating:

* EVERY check names the spec clause it enforces. verification/ can assume the
  reader has the tree in front of them; this cannot. "FAIL" without a
  citation is unusable by the audience, so the registry REFUSES to register a
  check with no `spec` field.

* required vs advisory, enforced in the verdict algebra rather than by
  convention. Advisory failures are reported loudly and do NOT set the exit
  code — otherwise "recommended" silently means "mandatory" and the
  distinction the README advertises is a lie.

* a required ERROR counts as FAIL. A check that could not run has not
  demonstrated conformance, and treating "couldn't tell" as "fine" is how a
  green result stops meaning anything.

* an all-SKIP run reports NOTHING_RAN, not PASS. A run where nothing was
  applicable proved nothing; a caller seeing PASS would reasonably believe
  otherwise.

* the negative checks are the point. Anyone can relay a well-formed `offer`.
  The load-bearing half of cb_wire_envelope.h is what a conforming peer must
  REFUSE — `sdp_offer` (the historic v0 tag), `bye` carrying `data`, a
  missing `from`. A deployment that accepts them works right up until it
  meets a peer that assumes the documented behaviour.

* the /json/protocol check asserts the endpoint is NOT probed. Enumerating
  Cb.* that way is what a naive kit would do, and on this worker it
  CHECK-FATALs the browser — the check that looks most natural is the one
  that kills the thing under test. Encoding the hazard as a SKIP with the
  reason attached puts CLAUDE.md's trap where someone debugging a SIGABRT
  will actually find it.

The wire rules were read off cb_wire_envelope.cc's Decode/DecodeData, and the
registration handshake off signaling/server.go:505-560, rather than
paraphrased from prose — the code is what a real peer does.

VERIFIED, not assumed:
  * 23/23 self-tests pass (node --test conformance/__tests__/*.test.mjs).
  * Run against a REAL worker (cr7727-f9e46a1434b5) on a runner's dind,
    inside the container's network namespace — verdict=PASS, both required
    CDP checks green, and the worker still healthy afterwards (4/4
    supervisord programs) since the hazard endpoint was never touched.
  * Exit codes checked directly rather than through a pipe: 0 help,
    2 usage error, 2 bad role, 1 required-check failure.

CI wiring, in two places for two reasons:
  * Lint runs the kit's own tests — no network, no image, always gating.
  * smoke runs the kit AGAINST the image it just proved boots. If the
    reference implementation cannot pass its own conformance suite, the
    suite is wrong and CI should say so before a third party does.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

triform-admin commented 2026-07-30 15:12:59 +00:00

Author

Owner

Copy link

Update: the signaling-wire suite is no longer unexercised. I ran the real Go broker (signaling/, auth disabled) on :18080 and drove the suite against it.

✓ PASS  relays-valid-offer              spec: cb_wire_envelope.h:15
✓ PASS  rejects-off-contract-tags       spec: cb_wire_envelope.h:40
✓ PASS  relays-ice-end-of-candidates    spec: cb_wire_envelope.h:25
✓ PASS  preserves-from-field            spec: cb_wire_envelope.h:110
✗ FAIL  rejects-bye-with-data (advisory)   spec: cb_wire_envelope.h:27
        expected: `bye` relayed without a `data` field, or not relayed
        observed: relayed with data={}
verdict=PASS

Two things worth noting.

The kit found a real divergence on its first live run. cb_wire_envelope.h:27 says bye omits data entirely — "not present, not null, not {}" — and cb_wire_envelope.cc's DecodeData enforces it (if (data) return nullopt, rejecting the whole envelope). The Go broker relays bye verbatim, data included. So a peer that sent bye {data:{}} through this broker would have it silently dropped by the far side, which then waits out its own timeout instead of seeing the disconnect.

It is correctly advisory, not a bug to fix. I checked both real emitters before deciding: client/src/session.ts:63 types bye as data?: undefined, and cb_offerer_driver.cc:1186 explicitly leaves the field unset. Neither in-repo peer can trip this, and a relay-only broker that never parses data is a legitimate design — failing a deployment for it would make "required" mean "matches our broker's implementation choices" rather than "satisfies the contract". The advisory classification did exactly its job: reported loudly, verdict=PASS, exit 0.

This is the case the kit exists for — a contract clause that no in-repo test covers, because no in-repo peer violates it. A third party writing a broker from the spec would learn this from one command instead of from a stalled session.

CI on this PR is green across Lint (23/23 conformance self-tests confirmed executed in the log), Docs, both CodeQL, E2E, and both native-peer gates.

**Update: the `signaling-wire` suite is no longer unexercised.** I ran the real Go broker (`signaling/`, auth disabled) on :18080 and drove the suite against it. ``` ✓ PASS relays-valid-offer spec: cb_wire_envelope.h:15 ✓ PASS rejects-off-contract-tags spec: cb_wire_envelope.h:40 ✓ PASS relays-ice-end-of-candidates spec: cb_wire_envelope.h:25 ✓ PASS preserves-from-field spec: cb_wire_envelope.h:110 ✗ FAIL rejects-bye-with-data (advisory) spec: cb_wire_envelope.h:27 expected: `bye` relayed without a `data` field, or not relayed observed: relayed with data={} verdict=PASS ``` Two things worth noting. **The kit found a real divergence on its first live run.** `cb_wire_envelope.h:27` says `bye` omits `data` entirely — *"not present, not null, not `{}`"* — and `cb_wire_envelope.cc`'s `DecodeData` enforces it (`if (data) return nullopt`, rejecting the whole envelope). The Go broker relays `bye` verbatim, `data` included. So a peer that sent `bye {data:{}}` through this broker would have it silently dropped by the far side, which then waits out its own timeout instead of seeing the disconnect. **It is correctly advisory, not a bug to fix.** I checked both real emitters before deciding: `client/src/session.ts:63` types `bye` as `data?: undefined`, and `cb_offerer_driver.cc:1186` explicitly leaves the field unset. Neither in-repo peer can trip this, and a relay-only broker that never parses `data` is a legitimate design — failing a deployment for it would make "required" mean *"matches our broker's implementation choices"* rather than *"satisfies the contract"*. The advisory classification did exactly its job: reported loudly, `verdict=PASS`, exit 0. This is the case the kit exists for — a contract clause that no in-repo test covers, because no in-repo peer violates it. A third party writing a broker from the spec would learn this from one command instead of from a stalled session. CI on this PR is green across Lint (23/23 conformance self-tests confirmed executed in the log), Docs, both CodeQL, E2E, and both native-peer gates.

triform-admin added 1 commit 2026-07-30 15:23:02 +00:00

fix(ci): the conformance step bind-mounted into a daemon that cannot see it ... ea13f01ebd

The step I added in the previous commit failed on its first CI run:

    Error: Cannot find module '/conf/run.mjs'

`-v "${PWD}/conformance:/conf:ro"` resolves the bind SOURCE on the DAEMON,
which on this host is a sibling dind with its own filesystem. It cannot see
the job's checkout, so docker silently created an empty directory at that
path and node found nothing to run.

This is the identical trap e2e.yml's "Probe daemon filesystem sharing" step
exists for -- documented in this same file two commits ago, and then walked
into anyway. The e2e job needs compose bind-mounts and so must SKIP when the
filesystem is not shared; this step only needs to get ~15 KiB of JS into a
container, which `docker cp` does through the daemon's own API regardless of
whether any filesystem is shared. So: create -> cp -> start, and the step
works on a shared-fs runner and a sibling-dind runner alike.

The smoke test itself passed in that run (PASS, 17121-byte PNG); only the new
step failed.

Verified before pushing this time -- the exact create/cp/start sequence, run
on the same runner dind against cr7727-f9e46a1434b5:

    devtools up after 4s
    ✓ PASS  devtools-reachable
    ✓ PASS  page-target-exists
    · SKIP  protocol-endpoint-hazard (advisory)
    verdict=PASS
    EXIT=0

Also adds the runner container to the cleanup trap; the previous version
leaked it when the kit exited non-zero.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

triform-admin merged commit ff52fa5955 into main 2026-07-30 15:43:27 +00:00

triform-admin referenced this pull request from a commit 2026-07-30 15:43:29 +00:00

Merge pull request 'feat(conformance): a kit that answers "does MY deployment conform?"' (#51) from track7/conformance-kit into main

triform-admin referenced this pull request from a commit 2026-07-30 15:51:58 +00:00

docs(conformance): the README advertised two things the kit does not do

triform-admin referenced this pull request 2026-07-30 15:52:08 +00:00

docs(conformance): the README advertised two things the kit does not do #53

triform-admin referenced this pull request from a commit 2026-07-30 15:58:43 +00:00

Merge pull request 'docs(conformance): the README advertised two things the kit does not do' (#53) from fix/conformance-readme-media into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

triform/chromeless!51

No description provided.