Four questions an exchange CTO asks, and what our answers actually are

Part 2 of two. Part 1 is the outage and the engineering. This one is the same work from the other side of the table: what you would want to know before running money on it.

The short version

Anyone evaluating an exchange stack asks the same four questions. They are not architecture questions, they are questions about a bad day. Ours, answered in one line each:

• "A node dies at 3am." Three-node Raft, so one member dying changes nothing. A wiped member is the harder case, and Aeron will not catch one up from a snapshot — we measured that, and reseeding is scripted.
• "Is there a copy independent of that machine?" Yes. Every snapshot and the log behind it ships to object storage every five minutes, checksummed, and nothing local is deleted until the remote copy is confirmed.
• "Can you prove you can restore it?" No. We have never performed a restore. The design is there and the engine half is now tested; the drill is not done.
• "An incident three weeks ago?" Each bundle carries a snapshot plus every message after it, so any past position is reachable at one-message granularity. Ninety-day window.
• "Have you had an outage?" Seventeen hours of dead settlement, on the demo. Part 1 is the full account.

The third answer is the one worth pressing on, for us or for anyone else you evaluate. Everything below is the long form, including four limits stated as limits.

"A node dies at 3am. What happens?"

Nothing, for a single node. Both the matching engine and the ledger run three-node Raft clusters, so one member can die without stopping trading or settlement. That is table stakes and not interesting.

The interesting version is the one we actually hit: what happens when the machine is rebuilt, or when a node comes back with nothing.

That used to depend on a file on the box. The ledger's durable position, and the time it became durable, were recorded locally. Lose the disk, lose the knowledge of what was safe. Now both are recovered from object storage at boot: a gateway restarted a minute ago can still tell you when the ledger was last provably safe, because it asks the archive rather than its own memory.

One thing we measured the hard way and will state plainly: Aeron will not catch a wiped member up from a snapshot. An empty member joins at position zero, the election tries to truncate the log to zero, and the leader has already reclaimed that far. So a wiped member has to be reseeded from a healthy one, and that has to be a first-class automatic operation rather than a runbook. Ours is scripted and has been exercised; it is not yet one button.

"Is there a copy of the balances independent of that machine, and can you prove you can restore it?"

The first half has a concrete answer. Every snapshot, and the log behind it, ships to object storage with a SHA-256 that the storage service itself validates on write. The manifest is uploaded last, deliberately: it is the commit marker, so a half-finished upload is never mistaken for a complete one, and the durable position only advances after that has happened.

Reclamation is bounded by that number. Nothing local is deleted until the remote copy is confirmed. In the first round after the archive went live, 1.08 GB of local copies were released once storage had confirmed them, and both clusters stayed healthy throughout.

The storage side is boring on purpose: EU region, public access blocked, versioning on, encrypted at rest, TLS required, unencrypted writes rejected. The role the box uses to write cannot delete. Deletion is the lifecycle policy's job, not the application's, which means a compromised box cannot erase the archive.

The second half of the question is the honest part. We have never performed a restore. Not once. The pieces are there, and one that was missing until this week is now in place: a bundle carries the descriptors that say how to read its own files, which it previously did not, meaning the earlier bundles are data without instructions. What has not happened is standing a ledger back up from one and comparing it to what it should be.

Designing a capability is not the same as proving it, and we would rather say so than let the design imply the drill.

"How would you investigate an incident from three weeks ago?"

This is where the archive is worth more than the word "backup" suggests.

Each bundle carries the range from the previous bundle's position to its own snapshot. So a contiguous run of bundles is a snapshot plus every message that followed it. To reach any past position: restore the newest snapshot below it, replay the log up to it. The granularity is one message.

That is point-in-time recovery, not backup, and it is the right frame for what we built. We are not keeping copies. We are keeping the ability to stand the ledger up as it was at a chosen moment, on the exact build that produced it, because the bundle records the commit that wrote it and that build is still addressable in the artifact store.

Four limits, stated as limits:

• The window is 90 days, not forever. Build artifacts are kept 180, so a build never expires before a bundle that names it. Those two numbers are coupled on purpose, not chosen independently.
• A gap makes everything above it unreplayable. The code refuses to create one silently: uploads run oldest-first and stop at the first failure, and capture refuses to continue if the position ever moves backwards. But if a gap ever existed, recovery would be limited to below it.
• There is no time-to-position index. Log entries carry timestamps, so it is buildable, but today "take me back to 15:42" means finding the position by hand.
• Replay reproduces the past only if the engine is deterministic given a snapshot and a log. Until this week that was an assumption. It is now a test.

"Have you had an outage? What changed afterwards?"

Yes. Seventeen hours of dead settlement on our demo system, described in part 1. No customers, no customer money, which is the entire reason we run a demo under constant load.

What changed is in that post. What is worth repeating here is the shape of the answer, because it is the answer we would want from a vendor.

The thing that made the outage last seventeen hours instead of five minutes was not the bug. It was that every dashboard was green while nothing worked. So the rule now is that every check on the money path has to verify a counter advanced, not that a process is alive. That rule was applied to every check we have, not just the one that failed.

And it caught something almost immediately, which is the part worth telling. A deployment quietly installed a four-day-old build of the ledger and ran it for forty minutes while all three nodes reported healthy. They were right: the process was running fine. It was simply the wrong code. The fix was not the missing wiring, it was removing the fallback underneath it, so a configured artifact store with nothing behind it is now an error rather than a quiet local rebuild.

That has now happened three times in this codebase in different forms: something is not registered, a plausible default sits underneath, and an omission becomes a wrong answer instead of an error. Three instances makes it a lesson rather than an anecdote, and it is why "refuse rather than fall back" is a rule here.

What we would rather you take from this

Not losing money is the job, not a feature, so there is nothing to congratulate us for in most of the above. The claim we are willing to make is narrower and, we think, more useful:

The ledger's state leaves the machine on a five-minute cadence, checksummed, addressed to the commit that produced it, and nothing local is deleted before the remote copy is confirmed. The engine that would replay it is now tested to produce byte-identical state from a snapshot plus a log, at every position, on every scenario we have. The restore drill itself is the next thing we build, and until we have run it, we are describing a design rather than a demonstrated recovery.

If you are evaluating any exchange stack, including ours, that last distinction is the one worth pressing on.