# Problem of old evidence

I've come across the so-called "problem of old evidence" in Bayesian statistics/epistemology.

First, let me summarize the problem as I see it so we're on the same page.

Suppose I have a theory H, and I suddenly realize that my theory predicts with certainty an experimental effect (E) that other theories have struggled to explain, that is p(E | H) = 1

I turn the Bayesian handle...

p(H | E) = p(E | H) * p(H) / p(E) = p(H) / p(E)

The reasoning now goes that since the experimental effect is old and long-known,

p(E) = 1, thus p(H | E) = p(H).

This seems surprising. To me, the mistake is that p(E) = 1, which is suggested because the data is old and well-established, we have faith in it. This begs the question; yes, p(E | E) = 1 trivially, but p(E) is our prior belief in seeing the phenomena E, not our belief that we saw the phenomena E given that we saw the phenomena E. We could have seen many different phenomena E', say.

P(E) should be expanded as normal,

p(E) = p(E | model 1) * p(model 1) + ... != 1

Solving the problem of old evidence. This is all rather trivial, so it strikes me that I'm missing something... What's wrong with my trivial resolution?

I'm happy with Bayes' theorem, Bayesian inference etc. I want to understand why people think that p(E) = 1, resulting in the problem of old evidence, or perhaps any mistakes I've made about the problem of old evidence. I honestly think the "problem" stems from a basic misunderstanding about p(E) - this is not our belief that we have obtained the evidence in E, it is our belief that we would obtain evidence E, a priori.

• I think you're exactly right. The "problem of old evidence" is actually a "problem of remembering what notation is supposed to mean". – Rex Kerr Jan 28 '15 at 20:34
• @rex but this is actually something people have spent time - years - discussing, thinking about, writing about. it's mentioned here plato.stanford.edu/entries/epistemology-bayesian – innisfree Jan 29 '15 at 8:59
• Despite the time and effort, it may still be the case that they're essentially just confused. (I.e. the objection works out to be a misunderstanding, or equivalent to the problem of induction.) – Rex Kerr Jan 29 '15 at 21:22
• I'm a big believer in Bayesian reasoning, and this is the first time I've ever heard of the problem you describe. My knee-jerk reaction is the same as yours and Rex Kerr's, that it's based on a simple misunderstanding of what P(E) really means. – Adam Sharpe Nov 18 '19 at 23:00

I'm no statistician, but this makes sense to me if you translate it into ordinary language. The probability of a theory H being true in light of a single piece of evidence E is the probability of E being true in light of H times the probability of H, divided by the probability of E.

Essentially, if you're looking at a single piece of evidence, you could make up any number of theories to explain it. If the evidence is a gun in the study, one theory is that Professor Plum left it there, another is that it was Miss Scarlet. Given that, at most, the probability of your theory being true in light of that single piece of evidence is the same as the probability of your theory being true independent of that evidence.

If that evidence was less than sure, it would reduce the probability of your theory being true by the same amount as your theory was committed to E.

As far as old evidence --once again this makes intuitive sense to me: If you have a piece of evidence prior to creating a theory then you're not really demonstrating any added value by predicting that fact. You would need to predict something new.

• whilst most of this is (trivially) true, i can't see how it relates to the problem of old evidence? – innisfree Jan 28 '15 at 16:33
• thanks all the same, let me make sure the question is clear. – innisfree Jan 28 '15 at 16:40
• I was trying to address that in the middle paragraph, will edit to be more clear. – Chris Sunami supports Monica Jan 28 '15 at 17:07
• Can you write that in math? I can't see how you'd reach that conclusion without p(E)=1, which is the point I'm challenging. – innisfree Jan 28 '15 at 17:47
• Just trying to provide an intuitive justification for the mathematical rule --I'm guessing that's not what you're looking for. – Chris Sunami supports Monica Jan 28 '15 at 18:16

It appears that my criticism of this problem is well-known. I happened across this discussion

http://andrewgelman.com/2008/09/02/more_bayes_rant/

In the comments, Bill Jeffreys, a well-known Bayesian statistician makes similar arguments and notes that they were first made by Rosenkratz. Remarkably the criticism appeared in the same volume as the original argument and was never rebutted.

Rosenkratz's argument, which is identical to that in the question, may be found in e.g., p85 of

https://conservancy.umn.edu/handle/11299/185351

which was published in 1983.

Your calculation is absolutely right, but doesn't mean much.

For a clarification, assume the following experiment: I throw a coin that has head and tails. If it comes up heads, I leave it. If it comes up tails, I turn it around. Finally I check whether it ends with heads up: Yes, it does. Each and every time (because I turned the coin around every time it was tails). So I have a coin experiment E, and p (E) = 1.

Now you have a theory H, which is true with probability p (H).

The probability that your theory is true if my coin experiment succeeds is obviously exactly the same as the probability that your theory is true if I don't perform the coin experiment. p (H | E) = p (H).

• actually, your calculation here doesn't meant much. the problem of old evidence concerns whether p(E)=1 for old evidence, where E is an experimental effect. you've just constructed a silly special case scenario in which arguably p(E)=1. – innisfree Jul 5 '15 at 9:04
• Just trying to make things clear for you using a simple example. Feel free to overcomplicate things. – gnasher729 Jul 5 '15 at 21:43
• with respect, i don't think you've understand this problem at all. i'm not overcomplicating things. – innisfree Jul 6 '15 at 13:23