SaSame

We pinned the orchestrator's verification runtime to Node 22 today (#4895). This is the environment the CI/verification pipeline runs on — the step that checks a release before it's allowed to close out. Up until now it was floating, tracking whatever "current" resolved to on the runner. That's fine until it isn't: a minor version bump upstream changes behavior slightly, a verification run that passed yesterday behaves differently today, and now you're debugging a runtime drift instead of debugging the actual release. Pinning trades that away on purpose. Verification for a given release should mean the same thing every time it runs — same runtime, same result, no surprises from the platform underneath. If we want to move to a newer Node later, that's a deliberate decision with its own changelog entry, not something that happens silently because a base image updated. The tradeoff is we now own keeping that pin current. Security patches, EOL timelines, all of it becomes something we have to actively track instead of getting for free. Worth it for a system whose whole job is closeout and release integrity — but it's a real cost, not a free lunch.

Closeout #4888 failed on its first pass. Not the release itself — the release had already promoted — but the orchestrator's post-promotion closeout step, the part where it self-certifies that the release is actually done and writes the evidence that lets the next release bind to it. That step errored out mid-run, which meant #4888 was promoted but not closed, and nothing downstream could safely build on it until we recovered the closeout by hand and re-ran it against the file caps it's supposed to satisfy. That recovery surfaced two more edge cases we hadn't handled cleanly. #4891's closeout had to admit an exact schema transition — not a compatible-with or a diff summary, the literal before/after schema — while staying under the same file cap that had just bitten #4888. Fitting an exact transition record into a fixed budget is a different problem than fitting a log line into one, and our tooling wasn't originally built to distinguish the two. #4890's closeout needed to map Exchange verification coverage as evidence — basically proving which verification paths actually ran, not just that the release passed. That's a new category of closeout evidence we hadn't required before this batch. None of these were failures of the release logic. They were gaps in the tooling that certifies the release logic — the part of the pipeline that has to prove its own work before anything downstream is allowed to trust it. Fixing that meant the orchestrator could go back and re-satisfy the caps instead of us quietly loosening them. Small batch, but it's the kind of thing that tells you more about the pipeline than a clean release does.

Our bounded testnet money-flow e2e test was failing intermittently, and for a while we did what everyone does with intermittent blockchain tests: assumed it was flaky and moved on. It wasn't. Two separate things were happening on Base Sepolia. First, receipt reorgs — a transaction gets a receipt, then the chain reorgs and the receipt is no longer valid, so the next lookup fails even though the tx is fine. Second, plain transient tx-lookup misses — the node just hasn't indexed it yet when we ask. Neither of these is actually nondeterminism in our code. They're normal testnet behavior that a naive "get receipt once, assume it's final" flow doesn't tolerate. The orchestrator was treating a temporary miss or a reorg as a hard failure instead of a retryable condition. We added retry logic for both cases — retry on reorg'd receipts, retry on lookup misses — instead of papering over it with longer sleeps or just rerunning the whole test suite until it passed. The interesting part is what surfaced once the flakiness stopped hiding things: child USDC balance evidence that had been getting lost in the noise the whole time. The money-flow test was actually working correctly, we just couldn't see the evidence because the test infra kept throwing it away before it could be recorded (#4880). Small reminder to ourselves: "flaky test" is often a diagnosis of laziness, not of the test.