Persistence

The engine holds the market in memory because that is the only way to answer with all of it in one request. Memory does not survive a restart, and for a while neither did the history.

What a restart used to cost

Everything the ledger knew, plus a bill.

The engine would come back empty, backfill a couple of hours from the chain to have anything to show, and start counting again from zero. Every wallet's record began at the moment of the last deployment. Twelve thousand blocks of RPC per restart, every restart, to rebuild a shallower version of what had just been thrown away.

There was a second failure hiding inside the first one. Resuming the wallet ledger without also rebuilding the price book left hours old wallets pointing at tokens the book had never seen, which made every row on the board undrawable. The two halves have to come back together or neither works.

What is on disk now

Storage is node:sqlite, which ships inside Node 22, so this costs zero dependencies.

Aggregates only. Raw swaps are never stored, because the chain is already the archive and re reading it is always possible.

Token identities survive even a deliberate reset, because a token's symbol and decimals are a fact about the chain rather than a fact about our window.

Retention is ninety days

Hour buckets older than about a day get folded into day rows. Day rows older than ninety days are dropped. Both happen on the same sweep the engine was already running, and the operation is idempotent, so running it twice changes nothing.

The horizon is configurable rather than compiled in.

The engine publishes the instant of the oldest row it still holds, and the terminal prints it as DATA SINCE on the bottom bar. That number grows as the engine accumulates history and it stops growing when it reaches the horizon. It is the honest answer to how far back the board can see.

The size was measured, not estimated

At the time of writing the database was 290 MB, which works out to roughly 250 bytes per row, on a box with 25 GB free.

A full ninety days of rolled up ledger projects to about 4 GB in the worst case. The box does not need a bigger disk, and knowing that meant the retention window could be chosen on what is useful rather than on what fits.

Two rules that keep it correct

Both are pinned by tests, and both were proven to bite by breaking them on purpose first.

The flush queues consume. Taking the dirty set empties it. If a flush then fails, the cost is one interval of work rather than a double write on the retry. The tempting alternative, reading without clearing, quietly writes some rows twice.

The daily fold must add, never overwrite. Merging an hour bucket into a day row has to accumulate onto whatever is there. An overwrite silently halves the history and passes every shallow test, because the row exists, the schema is right, and the number is plausible. That is the exact profile of a bug that lives in production for months.

Verified by killing it

Not by a graceful shutdown. kill -9, no clean exit, no final flush.

process killed → back up in under 25s
8,001 wallets rehydrated from disk
market book warmed with 28,033 trades over 11,925 blocks in 7s
confirmation lag 0
top wallet before $16,518.97 · after $16,518.97

Four promises proven at once. The supervisor restarts it, the ledger survives on disk, the book refills from the chain, and nothing is counted twice.

That last line is the one that matters. An identical top wallet across a hard kill is the only convincing evidence that a resume is not silently double counting, because a double count produces larger numbers that still look perfectly reasonable on their own.

What this changed upstream

The persistence is why the board can honestly claim a full day.

The ledger rehydrates twenty four hours from disk while the price book only holds what it has re read since the restart. When the two disagree, the screen reports the shorter one and under claims. Labelling a day of ledger as three hours costs nothing. The reverse gets a leaderboard quoted and then corrected.