Open Exchange v0.5.0-beta: the durability release
Open Exchange v0.5.0-beta is tagged across all five repositories.
v0.3 was about proving the engine survives failure. v0.4 was about the money itself. v0.5 is about what happens to the state when the machine goes away: whether the ledger exists anywhere other than the box it was produced on, and whether you can prove the copy is the same ledger.
Snapshots that take themselves
A cluster that never snapshots looks perfectly healthy right up until it isn't. Ours ran that way for a long time, and it was the root cause underneath our seventeen-hour outage: no snapshot meant no way to reclaim disk, and the disk filled.
Snapshots are now taken by log volume rather than by clock, every one is made durable automatically, and the archive behind them is reclaimed instead of growing forever. Both clusters do it. Nobody has to remember.
A bundle is a snapshot plus the log behind it
A snapshot on its own is a photograph. What you actually want is the photograph and everything that happened after it, so you can move forward from that point to any later one.
That pair is now captured as a single unit, a bundle, and shipped to object storage off the machine. The retention window is ninety days. Within it, the state is recoverable to a position, not just to a backup.
The bug that would have made all of that lie
Here is the part worth reading twice.
The ledger's snapshot wrote accounts and holds in hash table order. That order depends on which users were touched first and which holds happened to be released along the way. So a node replaying from the beginning and a node resuming from a snapshot wrote different bytes for an identical ledger.
Nobody's balance was ever wrong. The ledger was fine. What was broken was our ability to check it: the way you verify a bundle is to replay it and compare the snapshot it produces. That comparison would have reported divergence for a ledger that had not diverged. An alarm that goes off when nothing is wrong is worse than no alarm, because you learn to ignore it.
The fix is one ordering rule, the format did not change, and old snapshots still load. The interesting part is how long it hid: our whole-corpus replay test passed without the fix. Twenty-four hand-written scenarios were not big enough to grow a hash table and shrink it again. It only failed once someone wrote a scenario shaped like production. That is now the rule for this kind of test: write it, then temporarily undo the fix, and watch it fail.
Every binary has an address
Nothing compiles on the demo box any more. Builds are published by commit sha, downloaded, checksum verified, and swapped in. The reason is not tidiness. A bundle records which build produced it, and that record is worthless if the build only ever existed in somebody's working directory.
We learned the shape of that the hard way in the same week. One deployment path was wired to the new artifact store and one was not, so a rolling update quietly fell back to compiling locally, from a four day old checkout, on the live money ledger. All three nodes reported healthy, and they were right: the jar ran fine. It was simply the wrong jar. Healthy does not mean running the code you think you shipped, and the fallback that made it possible is gone: a cluster with no artifact store configured now stops rather than guesses.
The money path got twenty four times faster
The full money round trip, where a fund lock is placed and committed before any order reaches the matching engine, went from about 4,400 orders per second to over 120,000. It was not saturated there either.
The cause was not the ledger's arithmetic, which is already allocation free. It was that the engine sent its results one message at a time, inline, on the thread that applies state, and waited on a slow reader. One backed up subscriber could stop everything. Results are queued and sent in batches now, and each subscriber says what it actually reads instead of receiving everything.
Separately, an order's round trip through the matching engine dropped from 16.4 ms to 757 µs. Almost all of that was a timer: the per order facts headed for the OMS were riding a flush interval designed to conflate a price feed, which has no business batching them.
The number we will not publish
While measuring all this we made the two available numbers impossible to confuse.
One is ingress publication: the time to encode an order and hand it to the local driver. It never leaves the machine. No network, no consensus, no matching, no reply. It is well under a microsecond, and it is not a system latency.
The other is the committed round trip: order in, replicated, committed, applied, answered. On the same run, on the same orders, the two differ by roughly 880x.
The load generator now reports both, labelled, in the same report. If no round trip could be matched it says the number is absent rather than printing a zero, and it warns against quoting the flattering one in its place. Sub microsecond numbers are easy to produce and easy to misread; we would rather make that mistake impossible for ourselves than trust ourselves not to make it.
Designed, not demonstrated
Reading a bundle back is fully specified and not yet written. Until it is, everything above is a durable copy that has never been opened. We would rather say that than let "point in time recovery" imply a restore we have not performed.
What beta still doesn't mean
No external security audit has happened yet. That is still the biggest single item between here and anything holding real funds. Open Exchange remains infrastructure: KYC, custody, fiat rails, and licenses are yours. That line stays in every post until it stops being true.
Try it
The quick start goes from a fresh clone to a running exchange on one Linux box in a few minutes. The release notes carry the full changelog, per repo. If you build something on it, or try to break it, we want to hear either way.