Design Every Job to Run Twice: Idempotency in Trading Systems
Schedulers retry, laptops wake from sleep, and humans re-run scripts 'just to be sure.' A job that does something different the second time is a latent incident. What idempotency means for order and data pipelines, and the test that proves it.
The second run
Picture a nightly job that reads a list of pending signals and submits one order per signal. It works for months. Then one evening the machine sleeps mid-run, the scheduler’s retry logic fires on wake, and the job starts again from the top. The first run had already sent half its orders. The second run sees the same signals, finds nothing marking them as handled, and sends them again.
Nothing in that story is exotic. No bug in the strategy, no bad data. The only defect is an assumption buried in the design: this job will run exactly once per day. Almost nothing in real infrastructure guarantees that. Schedulers retry on failure, processes get killed and restarted, a person re-runs a script after a confusing log line, and two triggers occasionally overlap. A system that is only correct when every job runs exactly once is correct by luck.
What idempotent actually means
An operation is idempotent if running it twice leaves the world in the same state as running it once. Not “it probably does nothing the second time” — the same state, provably.
Some operations are naturally idempotent. “Set the stop level for this position to X” can be applied a hundred times and the result is identical. Others are not: “add one to the counter,” “append a row,” “submit an order.” The engineering task is to turn the second kind into the first kind, usually by changing what the operation says, not by hoping it is called once.
The shift is from commands to desired states:
- “Submit an order for these shares” becomes “ensure exactly one order exists for this signal.”
- “Append today’s bar” becomes “make the stored bar for this date equal to this value.”
- “Send an alert” becomes “ensure an alert with this identity has been sent.”
The second phrasing of each can be repeated safely, because it describes where the system should end up rather than a step to take.
Three patterns that do the work
Deterministic identity. Every unit of work gets an ID derived from its content, never from the clock or a random generator at run time. If the identity of “the order for signal S on date D” is a pure function of S and D, then a retry produces the same identity, and anything downstream can recognise it as a duplicate. A random ID generated inside the retry silently turns every retry into a brand-new piece of work.
Check, then act — against the durable record. Before doing something with an outside effect, look up whether its identity has already been recorded as done. The record has to be somewhere that survives the process: a file or database written to disk, or the external system’s own state. An in-memory set of “already handled” items is wiped by exactly the restart you are defending against.
Write the intent before the effect. The dangerous window is between “did the thing” and “recorded that I did the thing.” If a crash lands in that window, the next run sees no record and repeats the effect. Recording intent first (“about to do X, identity Y”) and marking it complete afterwards makes that window visible: a run that finds an intent without a completion knows it must reconcile with the outside world before deciding, instead of blindly repeating.
The quieter version: data pipelines
Duplicate side effects get the attention, but the same flaw corrupts data without anyone noticing. A job that appends fetched rows to a history table is not idempotent: run it twice and every row exists twice. Downstream code that computes a moving average or a count now silently weights those days double, and nothing throws an error. The numbers are merely wrong, in a way that looks plausible.
The fix is the same shift from commands to states. Key each row by its natural identity — for example, instrument plus timestamp — and write with replace-on-conflict semantics. Re-running the fetch then rewrites the same rows with the same values. A related and underrated benefit: if a data vendor later revises a value, the re-run corrects it instead of leaving a stale copy beside the new one.
One subtlety worth naming: idempotent does not mean “ignore the second run.” If the second run fetches a different value for the same key, overwriting is usually right for market data corrections but wrong for something like a recorded fill, which is a historical fact. Deciding which category a field belongs to is a design decision, and it deserves to be written down rather than discovered during an incident.
The test that proves it
Idempotency is easy to believe in and cheap to test, which makes skipping the test hard to excuse. Three tests cover most of the ground:
- Run twice, compare. Execute the job against a fixed input, snapshot the resulting state, execute it again, and assert the state is unchanged. This catches the plain double-run.
- Crash in the middle. Inject a failure after the effect but before the completion record, then run the job again. The assertion is that the final state matches a clean single run — no duplicates, no gaps. This is the test that exercises the dangerous window.
- Overlap. Start two instances at the same time against the same input. Without some form of locking or a uniqueness constraint in the durable store, this reveals who wins the race — and whether anyone does.
None of these needs market data or a broker connection. A stubbed outside world that records every call it receives is enough: the assertion is simply “the stub saw each unique request exactly once.”
What this generalises to
Worth auditing your own pipeline for:
- For each scheduled job, can you state what happens on a second run in one sentence?
- Are work identities derived from content, or from timestamps and random values generated inside the run?
- Does the “already done” record live somewhere that survives a restart?
- Is there a defined recovery path for a run that dies between the effect and its record?
- Do data writes replace by natural key, or append blindly?
- Has anyone ever actually run the job twice on purpose, to look?
A system built for the run that goes exactly to plan is a demo. The version that survives contact with schedulers, sleeping laptops and human re-runs is the one that was designed assuming every job would, sooner or later, run twice.