Persist It When It Happens: The Data You Can't Fetch Twice

Some of the most important facts in a trading system — what actually filled, at what price, in what order — are only available for a short window after they occur. Why you write them down at event time, and how to test that you did.

The report you can’t regenerate

Imagine a weekly review script that rebuilds last week’s activity from a service’s “recent history” endpoint. For months it works. Then the service is restarted mid-week, or the endpoint’s retention window turns out to be shorter than assumed, and the review quietly shows half a week of data. No error. The totals are smaller, the charts are shorter, and nothing looks broken — because nothing, from the script’s point of view, is broken.

The underlying mistake is treating a live, expiring view of events as if it were a durable record of them. They look identical on the day you write the code. They diverge the first time something restarts.

Two kinds of source

It helps to sort every data source a system touches into two buckets.

Durable sources keep answering the same question the same way tomorrow, next month, next year: an archived price history, a ledger exported by a vendor, your own append-only table.

Ephemeral sources answer well now and degrade afterwards. A session-scoped query that only covers the current connection. A “recent items” list with an unstated cap. An in-memory object that is the only copy of a value. A cache whose entries are replaced on refresh. A vendor endpoint that returns today’s state and has no notion of yesterday’s.

Ephemeral sources are not bad. They are often the fastest and the most detailed. The failure is architectural: a pipeline that needs a fact later but only ever asks an ephemeral source for it later.

Why this failure is quiet

Three properties make it hard to catch.

It degrades instead of failing. A durable source that goes away raises an error. An ephemeral source that has aged out of its window usually returns a valid, shorter answer. Short answers pass type checks, schema checks, and most eyeballing.

It works in testing. Development sessions are short. Anything you tested ran minutes after the events it queries, well inside every retention window. The bug only exists on the timescale of real operation.

The missing data is unevenly distributed. What gets lost is usually the oldest and the most unusual — exactly the events (the odd partial, the retried request, the day something restarted) you most want when explaining a discrepancy later.

The rule: write at event time, to a store you own

The fix is small and unglamorous. When an event of lasting interest happens, the handler that receives it writes a record to storage you control before doing anything else with it.

on_event(e):
    append_record(own_store, e)     # first: durable, append-only, timestamped
    update_in_memory_state(e)       # then: whatever the live system needs
    notify(e)                       # then: alerts, dashboards, etc.

A few details matter more than they look:

• Record what was received, not just what you concluded. Store the raw payload plus your receipt time. If a later version of your parsing is smarter, you can reprocess; if you only stored the conclusion, you can’t.
• Append-only. Corrections are new rows that reference old ones. An in-place update destroys the evidence of what the system believed at the time.
• Record the partial and the odd. A single logical action can arrive as several events. If you store only the summary, you cannot later tell a clean event from a messy one.
• Include a source and a version. Which feed said it, and which code interpreted it. This is what lets you explain a disagreement between two records a month later.
• Keep it boring. A flat file or a small table beats an elaborate schema. The goal is that it still exists and still parses after everything else has changed.

Derived numbers need their inputs

The same logic applies one level up. If a reported figure is computed from several raw events, persist the raw events, not only the figure. A stored total that can’t be reconstructed from stored components is a number you must trust on faith; a stored total alongside its components is a number you can audit.

This is also what makes reconciliation possible. Comparing your system’s own record against an independent outside record is one of the strongest checks available — but it requires that your side of the comparison was captured independently of the thing you’re comparing against, at the time, rather than regenerated from the same outside source on the day you check.

Testing that you did it

Because the failure only appears over time, tests need to simulate time.

• Restart test. Run a scenario, kill the process mid-way, start a fresh one, and confirm the record of everything before the kill is still complete and readable.
• Expiry test. Replace the ephemeral source with a stub that returns nothing for anything older than a few seconds. The review path should still produce the full answer from your own store.
• Gap detection. Give records a sequence or compare counts against an expected total, so a silent hole becomes a loud alert instead of a smaller report.
• Round-trip test. Periodically rebuild a past period only from your own store and compare it with what the live system said at the time. A mismatch is a finding either way.

What this generalises to

A short self-audit:

• For each report or review job, can you name the source it reads — and whether that source is durable or ephemeral?
• If the process or connection restarted at the worst moment, which facts exist only in memory or only in someone else’s short-lived view?
• Do event handlers write to your own store before they do anything that could throw?
• Do you store raw inputs alongside derived values, so any figure can be rebuilt?
• Would a missing hour of data raise an alert, or just produce a slightly shorter chart?
• Has anyone ever deliberately tried to rebuild last month using only what you saved?

The data you can’t fetch twice is the data most worth writing down once.