fix(deploy): the rollback guard never fired, and an aborted run rotated the cluster's secrets #114

Merged

triform-admin merged 3 commits from fix/stack-yaml-describes-reality into main 2026-09-10 18:49:19 +00:00

Conversation 0 Commits 3 Files changed 2 +159 -36

triform-admin commented 2026-09-10 12:14:25 +00:00

Owner

Copy link

Three defects in the standalone deploy path, each found by the previous one.
No product code changes — infra/k8s/standalone/deploy.sh and stack.yaml.

1. The rollback guard has never fired

deploy.sh carries a guard written after the user lost a day to a silent
image rollback ("tests passed against an image they never saw"). It greps
chromeless/chromeless:<tag>; stack.yaml pins chromeless/chromeless@sha256:<digest>.
So pinned_worker was empty and the guard's own [ -n "$pinned_worker" ]
test skipped the whole check — silently, since digest pinning landed.

Now matches both forms, and exits if it finds no pin at all: a guard that
cannot determine what it is guarding must not pass.

2. No guard on the gateway, which had drifted

Live sha256:63d6cfa7 vs the pinned 9ef7f4db, three hand-rolls behind.
Genuinely different images — 13 layers each, 5654728 vs 5609791 bytes,
different top layer. The gateway serves the client bundle, so a silent
downgrade changes what every viewer runs. Same guard, same
CHROMELESS_ALLOW_ROLLBACK=1 override.

I did not pin the live digest, though that was the obvious fix. No tag in
the registry points at it and no commit records it — it was pushed untagged.
A pin is a provenance claim; writing one for an image whose source nobody can
name would make stack.yaml look authoritative while being unverifiable.
Recorded as a DRIFT note beside the pin with what it would take to resolve,
and the guard blocks the apply until someone does.

3. An aborted run was not a no-op — it locked me out of the cluster

The gateway guard fired, correctly, and that exposed the real bug: the guards
sat after the secrets step, so a refused run had already rotated the Ed25519
keypair and the login password into the Secret.

• the worker held a token signed by a key the broker no longer knew, and
  hammered it with auth rejected about once a second;
• the gateway pod kept the OLD password in its environment — secretKeyRef is
  injected at pod start, not re-read — so the new password did not work either;
• the old password existed nowhere, because the line that writes
  .standalone-creds is at the end of the script and never ran.

Both guards now run before anything touches the cluster. Verified by
measurement: captured the keypair and password, ran the script, confirmed it
refused, confirmed both byte-identical after. Before the reorder the same
abort changed both.

What I got wrong on the way

My first commit said a bare kubectl apply "strips" undeclared env. It does
not, and the difference decides what the fix can be. Measured on a scratch
Deployment:

apply(declares X) -> set env X=live -> apply(omits X)  => X DELETED
apply(omits X)    -> set env X=live -> apply(omits X)  => X SURVIVES

Apply prunes via three-way merge against last-applied-configuration — it
removes what a previous apply claimed, not everything the current manifest
omits. An older stack.yaml declared the TURN vars; that is why omitting them
deleted them. Confirmed: last-applied on both live deployments no longer
names them.

That ruled out the repair I was about to make. Declaring the TURN vars with
placeholder values makes apply overwrite what deploy.sh sets (measured —
the placeholder wins), trading a silent drop for a silent wrong value, which is
worse because the stack then looks configured. And the credential is
HMAC-derived with a 24 h expiry, so it cannot live in a static manifest
regardless. kubectl set env stays the owner; what was missing was anyone
saying so, which deploy.sh now does — it lists every live env var
stack.yaml does not declare before applying (checked against the cluster:
reports all 8).

Verification

make verify / make lint	pass
TURN discovery	found coturn on triform-7, dialled 3478, reachable
worker guard	passes (pin matches live)
gateway guard	blocks today's drift, with the digests named
aborted run leaves secrets alone	keypair and password byte-identical after
stack still streams	ice=connected, 128 frames, 15284 audio bytes

Also replaces deploy.sh's hardcoded TURN_IP=95.217.200.179, which refuses
3478 — coturn had moved — with discovery that dials the relay before using it.
Selecting by the app= label, because coturn-2 matches every name-prefix
pattern for coturn (2 is a hex digit) and two earlier attempts returned a
plausible-but-wrong node.

Three defects in the standalone deploy path, each found by the previous one. No product code changes — `infra/k8s/standalone/deploy.sh` and `stack.yaml`. ## 1. The rollback guard has never fired `deploy.sh` carries a guard written after the user lost a day to a silent image rollback ("tests passed against an image they never saw"). It greps `chromeless/chromeless:<tag>`; `stack.yaml` pins `chromeless/chromeless@sha256:<digest>`. So `pinned_worker` was empty and the guard's own `[ -n "$pinned_worker" ]` test skipped the whole check — silently, since digest pinning landed. Now matches both forms, and **exits** if it finds no pin at all: a guard that cannot determine what it is guarding must not pass. ## 2. No guard on the gateway, which had drifted Live `sha256:63d6cfa7` vs the pinned `9ef7f4db`, three hand-rolls behind. Genuinely different images — 13 layers each, 5654728 vs 5609791 bytes, different top layer. The gateway serves the client bundle, so a silent downgrade changes what every viewer runs. Same guard, same `CHROMELESS_ALLOW_ROLLBACK=1` override. **I did not pin the live digest, though that was the obvious fix.** No tag in the registry points at it and no commit records it — it was pushed untagged. A pin is a provenance claim; writing one for an image whose source nobody can name would make `stack.yaml` look authoritative while being unverifiable. Recorded as a DRIFT note beside the pin with what it would take to resolve, and the guard blocks the apply until someone does. ## 3. An aborted run was not a no-op — it locked me out of the cluster The gateway guard fired, correctly, and that exposed the real bug: the guards sat *after* the secrets step, so a refused run had already rotated the Ed25519 keypair and the login password into the Secret. - the worker held a token signed by a key the broker no longer knew, and hammered it with `auth rejected` about once a second; - the gateway pod kept the OLD password in its environment — `secretKeyRef` is injected at pod start, not re-read — so the new password did not work either; - the old password existed **nowhere**, because the line that writes `.standalone-creds` is at the end of the script and never ran. Both guards now run before anything touches the cluster. Verified by measurement: captured the keypair and password, ran the script, confirmed it refused, confirmed both byte-identical after. Before the reorder the same abort changed both. ## What I got wrong on the way My first commit said a bare `kubectl apply` "strips" undeclared env. It does not, and the difference decides what the fix can be. Measured on a scratch Deployment: ``` apply(declares X) -> set env X=live -> apply(omits X) => X DELETED apply(omits X) -> set env X=live -> apply(omits X) => X SURVIVES ``` Apply prunes via three-way merge against `last-applied-configuration` — it removes what a **previous** apply claimed, not everything the current manifest omits. An older `stack.yaml` declared the TURN vars; that is why omitting them deleted them. Confirmed: `last-applied` on both live deployments no longer names them. That ruled out the repair I was about to make. Declaring the TURN vars with placeholder values makes apply **overwrite** what `deploy.sh` sets (measured — the placeholder wins), trading a silent drop for a silent wrong value, which is worse because the stack then looks configured. And the credential is HMAC-derived with a 24 h expiry, so it cannot live in a static manifest regardless. `kubectl set env` stays the owner; what was missing was anyone saying so, which `deploy.sh` now does — it lists every live env var `stack.yaml` does not declare before applying (checked against the cluster: reports all 8). ## Verification | | | | --- | --- | | `make verify` / `make lint` | pass | | TURN discovery | found `coturn on triform-7`, dialled 3478, reachable | | worker guard | passes (pin matches live) | | gateway guard | **blocks** today's drift, with the digests named | | aborted run leaves secrets alone | keypair and password byte-identical after | | stack still streams | `ice=connected`, 128 frames, 15284 audio bytes | Also replaces `deploy.sh`'s hardcoded `TURN_IP=95.217.200.179`, which refuses 3478 — coturn had moved — with discovery that dials the relay before using it. Selecting by the `app=` label, because `coturn-2` matches every name-prefix pattern for `coturn` (`2` is a hex digit) and two earlier attempts returned a plausible-but-wrong node.

triform-admin added 3 commits 2026-09-10 12:14:25 +00:00

fix(deploy): the rollback guard has been dead since digest pinning landed ... c0deff6a2d

Three defects in the apply path, found while cleaning up after a bare
`kubectl apply -f stack.yaml` broke a working stack.

1. THE GUARD NEVER FIRED. It greps `chromeless/chromeless:<tag>`, and
   stack.yaml pins `chromeless/chromeless@sha256:<digest>`. So
   pinned_worker was empty, and the guard's own `[ -n "$pinned_worker" ]`
   test skipped the whole check — silently. A protection written after
   the user lost a day to exactly this ("tests passed against an image
   they never saw") has not run once since digest pinning. Now matches
   both forms and EXITS if it finds no pin at all: a guard that cannot
   determine what it is guarding must not pass.

2. NO GUARD ON THE GATEWAY, which has drifted the same way — live
   sha256:63d6cfa7 vs the pinned 9ef7f4db, three hand-rolls behind.
   Verified different images (13 layers each, 5654728 vs 5609791 bytes,
   different top layer). The gateway serves the client bundle, so a
   silent downgrade changes what every viewer runs. Same guard, same
   override.

3. AN APPLY DROPS ENV NOBODY DECLARED. stack.yaml does not declare the
   TURN settings; this script sets them afterwards, so `deploy.sh` is
   fine and a bare apply is not — it strips them and leaves a stack that
   connects, negotiates SDP and shows no video. The guest falls back to
   stun.l.google.com, the client gathers one .local candidate, and
   tests/interactive reports "the worker already served a session",
   which is not what happened. Cost two verification runs today.
   deploy.sh now lists every live env var stack.yaml does not declare
   before applying. Checked against the cluster: reports all 8.

I did NOT pin the live gateway digest, though that was the obvious fix.
No tag in the registry points at it and no commit records it — it was
pushed untagged. A pin is a provenance claim, and writing one for an
image whose source nobody can name would make stack.yaml look
authoritative while being unverifiable. Recorded as a DRIFT note beside
the pin, with what it would take to resolve, and the new guard blocks
the apply until someone does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs(deploy): measure how apply drops env — it prunes what a PREVIOUS apply claimed ... b4ff0305ba

My previous commit said a bare `kubectl apply` "strips" undeclared env.
That is not what happens, and the difference decides what the fix can
be. Measured on a scratch Deployment rather than reasoned about:

  apply(declares X) -> set env X=live -> apply(omits X)  => X DELETED
  apply(omits X)    -> set env X=live -> apply(omits X)  => X SURVIVES

Apply prunes via three-way merge against last-applied-configuration, so
it removes what a PREVIOUS apply claimed — not everything the current
manifest omits. An older stack.yaml declared the TURN vars; that is why
omitting them now deleted them. Confirmed: last-applied on both live
deployments no longer names them.

This rules out the repair I was about to make. Declaring the vars in
stack.yaml with placeholder values makes apply OVERWRITE what deploy.sh
sets — measured, the placeholder wins — trading a silent drop for a
silent wrong value, which is worse because the stack then looks
configured. And the TURN credential is HMAC-derived with a 24h expiry,
so it cannot live in a static manifest regardless.

`kubectl set env` stays the owner. What was actually missing was anyone
saying so, which the note above now does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fix(deploy): run the image guards BEFORE rotating secrets — an abort was not a no-op ... 7f05ce1b6d

The gateway guard I added an hour ago did its job and locked me out of
the cluster, which exposed a worse bug underneath it.

The guards sat AFTER the secrets step. So a refused run had already
rotated the Ed25519 keypair (keygen) and the login password (openssl
rand) into the Secret. Consequences, all observed:

- the worker held a token signed by a key the broker no longer knew and
  hammered it with `auth rejected` roughly once a second;
- the gateway pod kept the OLD password in its environment, because
  secretKeyRef is injected at pod start and not re-read, so the new
  password did not work either;
- the old password existed NOWHERE, because the line that writes
  .standalone-creds is at the end of this script and never ran.

Recovery: read the Secret, write .standalone-creds by hand, restart the
gateway and broker for the new material, restart the worker. Verified
afterwards — ice=connected, 128 frames, 15284 audio bytes.

Both guards now run before anything touches the cluster. Verified by
measurement rather than by reading: captured the keypair and password,
ran the script, confirmed it refused, confirmed both were byte-identical
after. Before the reorder the same abort changed both.

A guard that fires must leave the cluster as it found it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

triform-admin merged commit c9e35d78a4 into main 2026-09-10 18:49:19 +00:00

triform-admin referenced this pull request from a commit 2026-09-10 18:49:21 +00:00

fix(deploy): the rollback guard never fired, and an aborted run rotated the cluster secrets (#114)

triform-admin referenced this pull request 2026-09-10 18:59:49 +00:00

fix(build): pin image builds to a commit and tag by it — plus probe 1 of the resize finding #115

triform-admin referenced this pull request 2026-09-11 12:07:15 +00:00

docs(findings): the resize latch, read from viz source — and the gateway cert manifest fix #117

triform-admin referenced this pull request from a commit 2026-09-11 12:07:16 +00:00

fix(stack): the gateway cert design was documented but never implemented

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

triform/chromeless!114

No description provided.