Our settlement stopped for seventeen hours and every check stayed green
Part 1 of two. This one is the engineering: what broke, and what the system does differently now. Part 2 is what it means if you are the person who would have to answer for it.
This happened on our public demo, which has no customers and no customer money. That is what a demo under continuous simulated load is for.
The short version
On 25 July our settlement stopped and nothing told us for seventeen hours.
- Trading never stopped. Orders matched, trades printed, the book looked alive. Money just stopped moving.
- $9.15 million of maker collateral froze, across 161,347 holds that could never be released.
- The settlement bridge retried 1,891 times. Every retry started from the beginning, so it made exactly zero progress, forever.
- Every health check was green the whole time. They were not lying. They were answering the wrong question.
- The cause was five links up the chain from where we first looked, and our first write-up blamed the wrong thing.
- The fix took nothing off the shelf. Storage per node went from 1.2 GB to about 50 MB. No bigger box, no moving the ledger off RAM, no latency paid.
Below: how a single scheduler bug became seventeen hours, why the safety gate made it worse, and the two rules we now hold every check to.
Both of our clusters snapshot themselves and reclaim their own disk. That machinery was in the assets engine's build plan from 9 July, and the matching engine had been using it for weeks.
The scheduler that drives it could only see one cluster.
Not a bug in the snapshot code, which worked. When the assets engine became a first-class cluster, the HTTP layer learned to route per cluster and the scheduler did not. So the assets engine had never taken a snapshot, on any day, since it existed, and nobody noticed because nothing had run out of anything yet.
The chain
It surfaced on 25 July as "the bots have run out of money." They hadn't. Settlement had been dead for seventeen hours.
- A restart went badly. At 02:43 a routine roll left one node unhealthy on both clusters.
- A safety gate started refusing. We had a rule that the cluster will not snapshot unless every node is healthy, because snapshotting past a lagging node can strand it permanently. It ran every five minutes, saw a degraded node, and correctly refused. For sixteen and a half hours. Both clusters: zero snapshots. The matching engine was blocked by the gate; the assets engine was never scheduled in the first place.
- So the logs grew. A Raft log is only truncated after a snapshot, and these live in
/dev/shm, which is RAM. Assets nodes reached 1.2 GB each. Shared memory hit 96% full. - The bridge could not get memory. Replaying the trade journal means mapping a buffer. It needed 192 MB; 100 MB was left.
- So it looped. Connection timed out after fifteen seconds, and each fresh attempt walked the recording chain from the beginning again. 1,891 times, zero trades forwarded.
Restarting the bridge would never have helped. The memory was gone before the bridge asked.
Our first write-up blamed the loop. The loop was real, and it was the last link, not the cause.
The check that watched it happen
We had a canary. It ran every minute. It checked settlement. It stayed green for seventeen hours.
It asked two questions: is the bridge halted, is it connected. No and yes. Both true the whole time, while the thing did no work at all. It measured that a component was up, never that work was happening.
One check did go red, half an hour in: order-book depth on Bitcoin. Every resting buy order locks collateral, and with settlement stopped nothing was ever released, so market makers ran out of buying power first where a bid locks the most dollars. Dogecoin, where a bid locks pennies, traded normally for hours. By the time we looked: 161,347 outstanding holds, $9.15 million of maker collateral frozen.
What changed
The scheduler sees every cluster. It now iterates the same cluster map the request handlers route through, so a new cluster cannot be invisible to it. Deterministic order, and one cluster failing does not stop the loop. Per-cluster results are on the status endpoint: last success, seconds since, consecutive failures, last error.
The guard moved from snapshot to purge. This is the design mistake worth the most. A snapshot is additive, and it is exactly what a recovering member needs. Reclamation is the destructive half. Guarding both because one member is unhealthy is what turned a degraded node into sixteen hours of unbounded log growth on two clusters. Now snapshots always run, only reclamation is skipped, and it says so out loud rather than silently declining.
Reclamation exists on both engines, as the same code. Aeron snapshots do not truncate the log; they add a recording. Giving disk back is a separate operation, and the assets engine had none. It got one, ported from the matching engine's, and both now run it automatically after every snapshot.
The result is the part we did not expect. Assets nodes went from 1.2 GB to about 50 MB, shared memory from 94% to 55%. The morning after the outage, the two obvious fixes on the table were a bigger box and moving the ledger off RAM onto disk. Both would have worked, and both would have been wrong: it was never a capacity problem, it was that nothing ever gave anything back. The ledger is still in RAM and we are not paying the latency for it.
Health checks measure progress, not state. A healthy bridge live-follows in one epoch forever, so the question became "is the epoch counter climbing while forwarded trades stay flat", which is the exact signature of spinning rather than settling. Three consecutive bad probes before alarming, so a benign reconnect stays quiet. If the counters cannot be read at all it reports "forward progress unverified" rather than green, because a check that cannot see is not a check that passes. And the healthy output now carries real numbers, because a green light with nothing behind it is what got us here.
Why we knew to look for a pattern
We write up every incident in enough detail to be useful six months later. The point is not the individual reports. It is that a pattern is invisible one at a time.
Reading back through them after this one, the same failure had happened three times in three weeks, in three different components: a bridge live-following a dead journal for two hours while reporting healthy, a halt flag that latched with nobody watching it, and this. Three unrelated bugs, one shared root: the check confirmed the component was alive, not that the work was getting done.
No single write-up caught that. Reading them together did. So it went into our standing engineering rules rather than into one patch: every check on the money path must verify that a counter is advancing. Not that a process is running, not that a connection is open. That a number went up. It has since been applied to every health check we have.
There is a second rule, less comfortable, from the safety gate that was correct a thousand times in a row while doing this damage: a check that refuses to act is still making a decision, and if nothing escalates when it keeps refusing, it has quietly become the failure.
What we built next
None of the above is a feature. It is the absence of bugs we shipped, plus two rules we did not have written down.
The part that is genuinely new came after: the ledger's state now leaves the machine. Every snapshot and the log behind it ships to object storage with a checksum, and nothing local is deleted until the remote copy is confirmed. That is part 2, told from the point of view of the person who would have to answer for an outage like this one.