A Risk Check That Passes When Its Input Is Missing Isn't a Check

The most dangerous line in a risk control is often the default branch: what it returns when the number it needs isn't there. Why risk checks should fail closed, how 'no data' quietly turns into 'all clear', and the tests that catch it.

The check that never fired

Imagine a guard in a trading system: before any order is submitted, compare current exposure to a configured cap, and refuse the order if the cap would be exceeded. It has a unit test. It has been “working” for months. Then a review of the logs shows that, for one whole afternoon, the guard approved everything — including orders that should have been blocked.

Nothing was wrong with the comparison. The exposure figure it depends on came from a data call that had timed out, returned nothing, and been converted to None. The comparison then ran as None > cap, which in some languages raises an error and in others quietly evaluates to false. “Not over the cap” became “approved.”

The check didn’t fail. It just never had anything to check, and its default answer was yes.

Two ways to be wrong

Every risk check can be wrong in two directions:

• False block: it refuses something it should have allowed. The cost is a missed opportunity or an annoying alert.
• False pass: it allows something it should have refused. The cost is whatever the control existed to prevent.

These are not symmetric, and the asymmetry decides what the default should be. When a check cannot compute its answer, because an input is missing, stale, malformed, or contradictory, it has to pick one of those two errors. A check that picks “pass” has silently converted the absence of information into permission.

How “no data” becomes “all clear”

The fail-open pattern rarely looks like a decision someone made. It arrives through ordinary, well-meant code:

Defaults that look harmless. exposure = data.get("exposure", 0) reads naturally. But a missing exposure is now zero exposure, which is the most permissive possible value. The default was chosen for convenience, and it encodes the least safe interpretation.

Swallowed exceptions. A try/except around a data fetch, with a bare pass or a continue, keeps the loop alive. It also means the step that was supposed to gate the next one is simply skipped, and the next one proceeds as though the gate had been passed.

Falsy values treated as absent, or the reverse. An empty list of open positions can mean “none are open” or “the lookup failed.” If both produce the same empty list, the downstream code cannot tell which world it is in.

Stale data that still has a value. The worst missing input is one that isn’t missing at all: a number that was correct an hour ago and is cached. It passes every null check. A control that doesn’t also check when its input was produced will happily compare today’s order to yesterday’s state.

Optional config. A cap that is only enforced if cap is not None means that deleting one line from a config file disables the control, with no error anywhere.

Designing for the unknown

A few structural habits make the conservative default the easy path rather than the disciplined one.

Return a three-valued result, not a boolean. Instead of pass/fail, a check returns approve, reject, or cannot-evaluate, and the caller has to handle the third case explicitly. If the type system or a simple enum forces that branch to exist, nobody can forget it.

Make required inputs required. Missing risk configuration should stop the program at startup, not be defaulted. A control with no configured limit is not an unlimited control; it is a misconfigured one, and the process should refuse to start rather than run without it.

Attach a timestamp to every input and enforce an age limit. A figure without an “as of” time is only half a fact. The check should reject inputs older than a stated tolerance, treating them the same as missing.

Distinguish the reason in the output. A rejection log line should say whether the order was refused because a limit was breached or because the limit could not be evaluated. These need different responses: one is the system working, the other is the system asking for help.

Alarm on the unknown path. If “cannot evaluate” happens at all, it should be visible immediately. A control that fails closed silently can look, from the outside, exactly like a strategy that has gone quiet for no reason.

Testing the branch nobody exercises

Normal tests feed a check good data and confirm it behaves. The fail-open bug lives in the test nobody writes: feed it no data and see what it does.

A useful pattern is a small table of hostile inputs run against every control:

• the input is missing entirely
• the input is present but None, empty, or zero
• the input is the wrong type (a string where a number belongs)
• the input is valid but older than the allowed age
• the source raises an exception partway through

For each one, the assertion is the same: the outcome is not approval, and the reason recorded is distinguishable from an ordinary limit breach. Five short test cases per control is a small price compared to discovering the behaviour in a live feed.

It is also worth running this deliberately in a paper environment. Cutting the data source mid-session and watching what the controls do is a more honest test of the safety design than any amount of reading the code.

What this generalises to

Worth auditing your own controls for:

• For each risk check, what does it return if its input is None? Have you actually run it to find out?
• Is any limit enforced only if it is configured, so that removing it disables it?
• Does every input carry an “as of” time, and does the check act on it?
• Can a log reader tell “blocked by the limit” from “blocked because the limit was unreadable”?
• Is there a test for each control where the data source simply fails?
• Does a swallowed exception anywhere sit between a data fetch and a gate that depends on it?

A control is only as trustworthy as its worst-case behaviour, and the worst case is almost never “the number was bad.” It is “there was no number.”