Bayes Can’t Save You From Ignorance
People sometimes talk about Bayesian modeling like it’s a kind of epistemic shield: start with a prior, update rationally, and you’ll avoid nasty surprises. But that picture is upside down.
Bayes is not a safety system. Bayes is a bookkeeping rule.
It tells you how to redistribute belief among the possibilities you already wrote down. It does not protect you from the possibilities you failed to imagine. And even when you did imagine them, it doesn’t magically make them preventable.
What you get, at best, is a kind of vacuous safety: either the catastrophe is already handled, or too expensive to handle, or unknown.
This post is an attempt to make that sharper.
1) Bayes is a reweighting rule, not a generator of possibilities
Bayesian updating answers one question:
Given a set of hypotheses I’m willing to consider, and evidence I observed, how should I reweight those hypotheses?
That’s it.
It’s not a machine for producing new hypotheses. It doesn’t create missing mechanisms. It doesn’t auto-invent the latent interaction term you forgot. It doesn’t detect that your model class is too small unless you already included an “I might be wrong” escape hatch.
A prior can be incredibly sophisticated and still be wrong in the only way that matters: it can put no mass on the thing that happens.
When that’s true, Bayes can’t warn you. It can only be confidently calm until reality arrives and humiliates the model.
2) The “sunrise” vs “well” mistake
A clean way to see the issue is the classic “sunrise” thought experiment.
If you model sunrise as a stable global law, you’ll say: “The sun will rise tomorrow.” Your uncertainty becomes tiny.
But put someone in a well, where the sky is intermittently occluded, and their experience says: “The sun only rises sometimes.” They’re not denying celestial mechanics; they’re living inside a different scope.
The core mistake isn’t “wrong probability.” It’s misplaced uncertainty.
- In the sunrise model, uncertainty is about events (“maybe it won’t rise tomorrow”).
- In the well model, uncertainty is about missing variables (“am I even in a context where my ‘sunrise’ concept applies?”).
This generalizes: many catastrophes are less like “rare random outcomes” and more like unmodeled interactions—hidden couplings, thresholds, regime shifts, occlusions, dependencies.
3) Tail events are evidence of missing interactions
Here’s a thing you can feel in your bones once you notice it:
When something “extreme” happens, the probability that you missed an interaction tends to go up.
Because “tail event under my current model” is often astronomically unlikely, while “tail event under a different mechanism” is merely unlikely—or even expected.
So a catastrophe is frequently a Bayesian update not toward “fatter tails,” but toward:
- new state variables exist,
- the system has regimes,
- dependencies matter,
- a threshold got crossed,
- a feedback loop engaged.
In short: the event is telling you “you’re in a well.”
4) The vacuous safety trilemma
Now for the sharper claim.
Take any Bayesian model (M), with some prior, some likelihood, and some action set (\mathcal A) (things you can actually do). Consider a “catastrophe” event (E).
Relative to that model and those feasible actions, one of three things must be true:
(1) It’s handled (modeled and cheaply controllable)
The catastrophe mechanism is in your model class, and there exists a feasible, affordable intervention that reliably reduces risk or caps loss.
This is the “add a fuse / add a watchdog / add redundancy / add a guardrail” case.
If you’re in this bucket, Bayes doesn’t save you. It merely tells you what you already know: the control is worth it.
(2) It’s priced out (modeled but too expensive or infeasible)
The catastrophe mechanism is in your model class, but every feasible intervention is too costly, too weak, or requires coordination you can’t get.
Here Bayes can become grimly accurate: “Yes, this can happen; no, it’s not worth preventing” (or “we can’t prevent it”).
That’s not safety. That’s accounting.
(3) It’s unknown (outside the model’s support/scope)
The catastrophe mechanism is not represented—because the model class cannot express it, or your prior effectively excludes it, or the scope assumptions silently rule it out.
In this case, the model can’t warn you in advance. It assigns vanishing probability to the thing that happens because it didn’t know how to talk about it.
This isn’t “Bayes failed.” This is what Bayes is: conditioning inside a chosen space.
Put differently:
Bayesian safety is either engineered, priced out, or imagined away.
That’s the trilemma.
5) Why this makes “Bayesian disaster prevention” feel trivial after the fact
After a disaster, you can always retrofit the story:
- “We should’ve put mass on that failure mode.”
- “We should’ve modeled that coupling.”
- “We should’ve added the circuit breaker.”
And often the mechanism is mechanically simple once it’s named.
That hindsight simplicity tempts you to conclude: “All known Bayesian disasters are trivial.”
But that conflates three different senses of “trivial”:
- Trivial to describe once observed (often true)
- Trivial to represent once you add the missing variable (sometimes true)
- Trivial to prevent given costs, incentives, and coordination (often false)
Many disasters are “trivial” only in the first sense.
6) The uncomfortable moral: Bayes doesn’t remove uncertainty, it hides it
When a Bayesian model feels comforting, it’s often because it has turned uncertainty into a number inside a scope. The real uncertainty moved outward:
- “Is my model class correct?”
- “Is my scope stable?”
- “Do the invariances hold tomorrow?”
- “What happens when the system leaves the regime my data came from?”
- “What interactions did I exclude because they were inconvenient?”
Bayes can give you extremely sharp beliefs while being extremely wrong about exactly what matters (tails, regime changes, rare couplings). That’s not a paradox; it’s a reminder that concentration is not correctness.
7) So what’s the alternative?
If you want non-vacuous safety, you need to represent and pay for one or more of:
- Model uncertainty explicitly (not one prior, but a set of plausible models)
- Robustness / worst-case reasoning over neighborhoods of models
- Regime detection and monitoring (tripwires, not just predictions)
- Graceful degradation (limit blast radius instead of “predict perfectly”)
- Incentive-compatible controls (safety that doesn’t require angels)
In plain language: if you want protection against the unknown, you don’t bet on perfect beliefs. You build systems that fail less catastrophically when beliefs are wrong.