Problem of old evidence

Question

I've come across the so-called "problem of old evidence" in Bayesian statistics/epistemology.

First, let me summarize the problem as I see it so we're on the same page.

Suppose I have a theory H, and I suddenly realize that my theory predicts with certainty an experimental effect (E) that other theories have struggled to explain, that is p(E | H) = 1

I turn the Bayesian handle...

p(H | E) = p(E | H) * p(H) / p(E) = p(H) / p(E)

The reasoning now goes that since the experimental effect is old and long-known,

p(E) = 1, thus p(H | E) = p(H).

This seems surprising. To me, the mistake is that p(E) = 1, which is suggested because the data is old and well-established, we have faith in it. This begs the question; yes, p(E | E) = 1 trivially, but p(E) is our prior belief in seeing the phenomena E, not our belief that we saw the phenomena E given that we saw the phenomena E. We could have seen many different phenomena E', say.

P(E) should be expanded as normal,

p(E) = p(E | model 1) * p(model 1) + ... != 1

Solving the problem of old evidence. This is all rather trivial, so it strikes me that I'm missing something... What's wrong with my trivial resolution?

I'm happy with Bayes' theorem, Bayesian inference etc. I want to understand why people think that p(E) = 1, resulting in the problem of old evidence, or perhaps any mistakes I've made about the problem of old evidence. I honestly think the "problem" stems from a basic misunderstanding about p(E) - this is not our belief that we have obtained the evidence in E, it is our belief that we would obtain evidence E, a priori.

Answer 1

It appears that my criticism of this problem is well-known. I happened across this discussion

http://andrewgelman.com/2008/09/02/more_bayes_rant/

In the comments, Bill Jeffreys, a well-known Bayesian statistician makes similar arguments and notes that they were first made by Rosenkratz. Remarkably the criticism appeared in the same volume as the original argument and was never rebutted.

Rosenkratz's argument, which is identical to that in the question, may be found in e.g., p85 of

https://conservancy.umn.edu/handle/11299/185351

which was published in 1983.

Answer 2

I'm no statistician, but this makes sense to me if you translate it into ordinary language. The probability of a theory H being true in light of a single piece of evidence E is the probability of E being true in light of H times the probability of H, divided by the probability of E.

Essentially, if you're looking at a single piece of evidence, you could make up any number of theories to explain it. If the evidence is a gun in the study, one theory is that Professor Plum left it there, another is that it was Miss Scarlet. Given that, at most, the probability of your theory being true in light of that single piece of evidence is the same as the probability of your theory being true independent of that evidence.

If that evidence was less than sure, it would reduce the probability of your theory being true by the same amount as your theory was committed to E.

---

As far as old evidence --once again this makes intuitive sense to me: If you have a piece of evidence prior to creating a theory then you're not really demonstrating any added value by predicting that fact. You would need to predict something new.

Answer 3

Your calculation is absolutely right, but doesn't mean much.

For a clarification, assume the following experiment: I throw a coin that has head and tails. If it comes up heads, I leave it. If it comes up tails, I turn it around. Finally I check whether it ends with heads up: Yes, it does. Each and every time (because I turned the coin around every time it was tails). So I have a coin experiment E, and p (E) = 1.

Now you have a theory H, which is true with probability p (H).

The probability that your theory is true if my coin experiment succeeds is obviously exactly the same as the probability that your theory is true if I don't perform the coin experiment. p (H | E) = p (H).