I recently stumbled upon a note from an old work laptop—a digital fossil from a previous life in engineering. It was a brain-dump about a specific, maddening problem: transactions arriving out of order.
Reading it now, the note is less about a bug and more about a moment in digital history. It captures the era when we were evolving from the "batch world" of hourly FTP files to a "streaming world" where "real-time" meant per-second HTTP pushes. We were migrating from legacy RDBMS systems that processed static files to what I think of as organic ETL—data that flowed and transformed as it arrived, rather than in scheduled chunks.
In that transition, we hit a wall: the struggle between sequential expectations and distributed reality.
We were dealing with a payment provider that sent us batches of transactions every second. On paper, it was a stream. In reality, it was chaos. Not only were messages within a batch occasionally shuffled, but transactions could actually drift across batches.
The real pain point was the dependency.
In a perfect world, the sequence looks like this:
But in the wild, we observed sequences like:
Accrual 2 $\rightarrow$ Void 1 $\rightarrow$ Accrual 1
If your system assumes a linear timeline, the Void 1 arrives and looks for an Accrual 1 to cancel. It finds nothing. It either throws an error or creates a "negative balance" that doesn't make sense. Then, seconds later, Accrual 1 arrives, and suddenly the customer has points they should have already lost.
When you first encounter this, your instinct is to force the world to be orderly. I spent a lot of time thinking through the "standard" fixes, and in doing so, I realized that the "Naive Delay" isn't actually naive—it's the foundation of the entire financial industry.
The banking world is essentially built on a giant, quantized delay. We call it "settlement." Whether it's the end of the day, the week, or the quarter, the industry simply waits long enough for the chaos to settle before finalizing the state.
As we moved from big batch jobs to smaller batches, and finally to modern streaming, we were just shrinking that quantization window. The question was: how small can that window be before the system breaks?
At the time, Redis was "all the rage," but not just as a cache—it was viewed as a powerful data-structure server.
I remember thinking about building a purpose-built database for this task: a delayed-processing stream. The idea was to use Redis' priority queues to insert records with expiration-timestamp-based priorities. You'd poll for unprocessed items only up to a specific timestamp, effectively using the queue as a buffered-sorting process.
It was a brilliant piece of engineering for the problem at hand, but it still felt like we were fighting the stream. We were trying to reconstruct a linear timeline from a source that was inherently non-linear.
The next logical step was retries. If a Void arrives without an Accrual, fail it and retry in 10 seconds. Use a Finite State Machine (FSM) to track "Pending Voids."
But this turned a data problem into a state management nightmare. Where do you persist the "Pending" state if the server crashes? How long do you back off? You introduce a massive window for eventual consistency issues and potential fraud. You're no longer processing data; you're managing a complex web of "maybe" states.
The breakthrough came when I stopped trying to sort the stream and started thinking about CRDTs (Conflict-free Replicated Data Types)—specifically, the concept of a PN-Counter (Positive-Negative Counter).
The core insight of a CRDT is commutativity. In mathematics, addition is commutative: $A + B$ is the same as $B + A$.
If we treat our transactions not as a sequence of commands to be executed, but as a set of facts to be summed, the order of arrival no longer matters.
Instead of updating a "Current Balance" table in real-time, we moved to an append-only ledger:
Accrual, Void, Adjustment) is written to a ledger exactly as it arrives. No sorting, no delaying.Void exists in the ledger but the corresponding Accrual hasn't arrived yet, the sum simply reflects that. The moment the Accrual is inserted, the next read automatically corrects the balance.Looking back at that old note, the lesson is clear: Whenever you find yourself fighting the order of events in a distributed system, you are likely fighting the wrong battle.
Trying to reconstruct a linear timeline from a distributed source is a recipe for fragility. By shifting the logic from imperative updates ("Subtract 10 points now") to declarative state ("Here is a list of all changes that have occurred"), the problem disappears.
The system becomes more robust, the audit trail is built-in by default, and the "Out-of-Order" bug becomes a non-issue.