6

Bayes' Theorem
P(H) = probability of a hypothesis
P(E) = probability of evidence
P(E|H) = probability of evidence given the hypothesis
P(H|E) = probability of hypothesis given the evidence

P(H|E) = P(H) × P(E|H) ÷ P(E)

Notice the division by P(E). So, if P(E) = 0, we have division by 0 and P(H|E) = ∞. I asked this on MathSE and the reply was simply that P(E) = 0 is not allowed. The standard way to exclude weird stuff in math is to say things like, using this particular theorem, P(H|E) = P(H) × P(E|H) ÷ P(E), P(E) ≠ 0.

What, if anything, do philosophers have to say about this?

EDIT 1 START

Asante sana to all the respondents. I just realized that if P(A) = 0, then P(¬A) = 1, because P(A) + P(¬A) = 1.

Consider the hypothesis God exists given miracles as evidence, which is in the context of Bayes' theorem this: P(G|M) = [P(G) × P(M|G)]/P(M). M = Miracles.

If the probability of miracles is 0, i.e. P(M) = 0, we're stuck with division by 0.

However, as suggested above, P(M) = 0 means P(¬M) = 1. Can't we then compute P(G|¬M)? What is the probability of God given that miracles don't occur?

P(G|¬M) = [P(G) × P(G|¬M)]/P(¬M)
P(G|¬M) = [P(G) × P(G|¬M)]/1
P(G|¬M) = P(G) × P(G|¬M)

If P(G) = 0.5 and P(G|¬M) = 0 (If miracles are evidence for God, the absence of miracles implies God doesn't exist), we can see that the posterior probability of god given no miracles is 0 i.e. P(G|¬M) = 0

EDIT 1 END

19
  • 3
    See Bayes' theorem. Jan 30 at 7:06
  • 7
    What mathematicians have to say about this is that when you want to condition on sets of probability 0 simple division of unconditional probabilities does not work as the definition and you need something more elaborate. This is why p(E)=0 is "not allowed". Philosophers concur. In some cases, you can still define it but then you need something more elaborate than naive Bayes's theorem as well.
    – Conifold
    Jan 30 at 9:11
  • 3
    Having said that, the theorem is base on the conditional probability, that measures “the chance of an event H happening given that we have already observed some other event E”. In both formulas (the conditional prob and B's Th) the magnitide P(E) is under the division sign, and thus the restriction: but what does it mean to "measure the chance of H provided that we have observed E" if P(E)=0, i.e. we have not observed it? Jan 30 at 10:16
  • 17
    If P(E) = 0 then P(E|H) = 0 as well, so you actually have a 0/0 situation, which is undefined. It does not mean that P(H|E) tends to infinity. It just means that trying to use Bayes' theorem in this situation is rather pointless.
    – Bumble
    Jan 30 at 11:39
  • 5
    From a mathematical point of view, "not allowed" is incorrect, there's a well developed theory of what conditioning on null events means (see, e.g., discussions on math.SE ). In fact there's nothing exotic about this: the probability of observing a particular value for a continuous random variable is 0, but this is a bog standard event to condition on a few weeks into an undergrad probabiilty class. However, Bayes' theorem in the form written in the post only holds if P(E) \neq 0, and more sophistication is needed to handle null events. Jan 31 at 17:07

10 Answers 10

15

Bayes' Theoreom has P(E) != 0 as one of its pre-conditions. As with any other theorem, if any of the premises is not true you cannot validily deduce the conclusion.

So the math thing to say is not "using this theorem, blah blah blah"; the math thing to say is "you can't use this theorem".

Of course, you can cut things off even earlier, and observe that the definition of P( · | E) doesn't make sense if P(E) = 0. (Though you can do something in the case of E corresponding to a continuous random variable assuming a single value, thought it's not Bayes' theorem as presented in the original question. h/t @Conifold)

But if we're going to sidle past that problem, and if you're to be a bit more flexible about just exactly what constitutes Bayes's theorem, you could consider the version

P(E) × P(H|E) = P(H) × P(E|H)

and say that this usually allows us to find the value of P(H|E), but in our degenerate case we get 0 × P(H|E) = something, and say that in this case the equation doesn't let us compute P(H|E), which could be hand-wavily taken to mean that a zero probability event happening doesn't help us update our probabilities.

Getting extra hand-wavey we could analogize this to gimbal lock, i.e. we usually use Bayes' theorem to deal with our probabilities, but in this one odd situation our usual methods exert no control.

8
  • 1
    Note that, since P(E|H) must be less than or equal to P(E) and P(H) is less than or equal to 1, the limit as P(H) goes to 0 of P(H|E) P(E) / P(H) equals 0.
    – g s
    Jan 30 at 6:09
  • 7
    @gs What? Why should P(E|H) be less than or equal to P(E)? That statement is not true in general. In particular, in the extreme case where H=E="Winning the lottery", P(E|H) = P(E|E) = 1 is certainly not less than or equal to P(E) = 0,00000000715.
    – Stef
    Jan 30 at 12:00
  • 3
    We statisticians have been using Bayes Theorem with continuous random variables for a very long time now :)
    – Cliff AB
    Jan 31 at 5:27
  • 1
    The problem is more fundamental than Bayes' Theorem. Zero and infinity are poorly behaved numbers in our algebra. You don't really know what you can do with X if it can be either one. These "numbers" destroy information. Jan 31 at 10:51
  • 1
    I guess Bayes didn't design his theorem correctly.
    – JonathanZ
    Feb 14 at 1:01
10

I expect that philosophy cannot say more about the case from the OP’s question than stochastics can say:

If the event E has probability zero to happen, then stochastics cannot derive a probability that H happens in case that E happens before.

Therefore it is reasonable to make the assumption P(E) > 0 when applying Bayes’ theorem.

Otherwise: If the Mount Everest were in the Netherlands, what is the probability that its height would be less than 100 meters? :-)

1
  • Most perceptive. Thank you.
    – Hudjefa
    Jan 30 at 11:03
6

tl;dr Dividing-zero-by-zero (0/0) is indeterminate. This means that it doesn't imply a unique-solution. Still, we can force a unique-solution through context.


Math review: Indeterminism.

In basic math, a lot of expressions can be reduced to simpler expressions. For example, 1+1 can be reduced to 2.

Reductions work when there's sufficient information to perform them. For example, if we have a value of 1, and then add another 1, we have enough information to know that the result would be 2 in all cases.

Now let's consider this scenario:

  1. We know that x = 10.

  2. We choose to multiply both sides by 0, finding 0 * x = 0 * 10.

  3. We reduce the right-hand side to find that 0 * x = 0.

  4. We divide (anti-multiply) both sides by 0 to find that x = 0 / 0.

  5. So now we've found that x = 0 / 0, and while it's still true that x = 10. What's going on?

Basically, 0 / 0 equals 10 – in this specific case, rather than a determinate form like 1 + 1 being 2 in all cases.

A lot of simpler explanations of math might say that

  • 1+1 is 2.

rather than

  • 1+1 implies 2.

. That distinction may seem pedantic, but it's more useful here because

  • 0 / 0 is 10.

might be pretty misleading, whereas

  • 0 / 0 implies 10 [in this context].

might be less misleading.

To skip a long explanation, it's just easier to say "Don't divide-by-zero.". Just like it's easier to say "Don't play with fire." and "Don't roll your own crypto.".

(If anyone has a good reference to explain the topic, please feel free to comment it!)


Let's divide-by-zero!

Okay, so the big problem's that P(A|B) doesn't make sense if P(B)=0, right? Because P(A|B) would seem indeterminate.

This image from Wikipedia looks good!:

So, in the above, if the Region-B were zero-sized, then how could we assess what portion of Region-A would fall within it vs. not within it? And if there's no way of determining a consistent value for that portion, then, ya know – indeterminate.

BUT! — there's an exception!

What if we defined Region-B as existing within the Universe-U, while the Region-A was exo-universal, encompassing Universe-U entirely? Then, there might be an argument for determining P(A|B)=1, even if P(B)=0.

Example:

  • A is the event where TRUE is TRUE.

  • B is the event where pigs fly in the Universe-U.

  • P(A) is the probability that TRUE is TRUE.

  • P(B) is the probability that pigs fly in the Universe-U.

  • P(A|B) is the probability that TRUE is TRUE given that pigs fly in the Universe-U.

  • P(B|A) is the probability that pigs fly in the Universe-U given that TRUE is TRUE.

And:

  1. We assert that, in the Universe-U, pigs can't fly.

  2. Because (1), P(B)=0.

  3. Is P(A|B) indeterminate? This is, can we not assign a probability that TRUE is TRUE?

We might argue that TRUE is TRUE is such a fundamental assertion that it'd hold throughout some exo-universe in which

  • P(A|[anything])=1

as a general principle. Then, to be consistent, within Universe-U, which is a mere subset of the exo-universe, we'd have to acknowledge that

  • P(A|B)=1.

This is, TRUE would still be TRUE no matter what, regardless of the impossibility of pigs flying.

Then, we can give the more conceptual form of Bayes's theorem as

  • P(A) × P(B|A) = P(B) × P(A|B),

and, BAM! – we (sorta) made it work!

3
  • 1
    Had to get a bad taste out of my mouth, so writing a funny answer seemed like a reasonable distraction.
    – Nat
    Jan 30 at 7:42
  • 1
    BTW -- I'd probably have preferred that A was just TRUE, rather than TRUE being equal to TRUE. Sorta like, how in programming, it's better to do if (TRUE) than contriving it to if (TRUE == TRUE). But, since this is SE.Philosophy and a lot of folks seem to find a primitive like TRUE being an expression non-intuitive, went with the equation to have a dummy-expression to sink the evaluation.
    – Nat
    Jan 30 at 7:56
  • 1
    Thanks for the effort. I like the term indeterminate. For P(B) = 0, P(A ∧ B) = P(A) × P(B|A) = P(A) × 0 = 0.
    – Hudjefa
    Jan 30 at 11:11
6

Bayes Theorem is essentially a translation of intuition about how evidence affects probabilities translates into math. But it only applies in situations that actually make sense in the real world.

P(H | E) means the probability that H will happen given that E has happened. But if E is impossible (which is what P(E) == 0 means), it makes no sense to ask about the probability of something dependent on it. Consider the analogistic:

Given that someone flies to the moon without a ship, what's the probability that they'll do X?

No matter what you substitute for X, the question makes no sense because the premise is impossible. Whatever X is, its probability is unaffected by the absurd premise; if it's "win the lottery" it's still very low, if it's "will wake up every day until they die" it's high probability.

So Bayes Theorem simply disallows P(E)==0, because it's useless for making any predictions. If you allowed it you would get division by 0, which is meaningless, which corresponds to the meaningless of the question.

1
  • Sir, correctamundo! I edited my answer. It seems to know X is impossible means we know ¬X is certain. So if the probability that someone could fly to the moon without a ship is 0 means the probability that no one can fly to the moon without a ship is 100%.
    – Hudjefa
    Feb 13 at 0:14
4

An intuitive way of looking at this is If there is no chance of E happening, Bayes theorem isn't going to be useful in understanding that situation. This is a case where the math has a common sense, real-world interpretation.

1
  • I edited my answer. Please leave a reply if you wish.
    – Hudjefa
    Feb 13 at 0:15
3

Let's take a concrete example. I used to live on Guadacanal 40 years ago. If you has asked me what was the probability of finding a ship in the middle of the road to the airport, I would most likely have said zero (or questioned your sanity). One day we had a cyclone, and, for the rest of my contract there was indeed a ship in the middle of the road: we had to stop, and drive around it carefully.

In retrospect, my prior of zero was wrong. But every prior is the result of a calculation. If we get a silly result ("what just happened is impossible"), we need to revise the calculation, including revising our priors. Meta-Bayesian?

1
  • Good point, but I feel you're conflating 0 with 0.000000001
    – Hudjefa
    Feb 13 at 0:15
1

This is a convenient notation for the theorem that allows us to calculate an unknown from a number of knowns, but it's derived from an underlying truth about set theory.

Suppose I have two boxes. One has nine red marbles in it. The other has three red marbles and six blue marbles. I pull a marble from one of the two boxes, and it is red. We can model this as a set for each box that shows us the color of each marble:

Box 1: R R R R R R R R R
Box 2: R R R B B B B B B

This is great if we want to determine the probability of getting a particular color marble from a particular box, but it's not so great if we want to figure out the opposite. Bayesean probability is the process of adjusting our sets to isolate the datapoint we're looking for. So let's make a set for each color, that shows us which box each marble is in:

Red:  1 1 1 1 1 1 1 1 1 2 2 2
Blue: 2 2 2 2 2 2

Now we can see that there are twelve total red marbles, and nine of them are in box 1. So the probability that a red marble came from box 1 is 9/12, or 3/4.

The statement that Bayes' Theorem is making is that the intersection of the set of red marbles and the set of marbles in box 1 is a subset of both the set of red marbles and the set of marbles in box 1, and that this causes a relationship between the various subsets involved:

P(red and box 1) = P(red | box 1) * P(box 1)
P(red and box 1) = P(box 1 | red) * P(red)
P(red | box 1) * P(box 1) = P(box 1 | red) * P(red)

Or more generally,

P(B|A) * P(A) = P(A|B) * P(B)

The division in the given form is just an algebraic convenience. Asking what happens when P(B) is 0 is the same as asking what happens if I were to pull a yellow marble out of one of the two above boxes. Well, that wasn't even a thing that could happen. It's excluded from the domain of the problem.

Suppose we modeled it anyway:

Box 1:  R R R R R R R R R
Box 2:  R R R B B B B B B

Red:    1 1 1 1 1 1 1 1 1 2 2 2
Blue:   2 2 2 2 2 2
Yellow:

Well, you still couldn't have drawn a yellow marble because there are no elements in that set.

2
  • 1
    I think you made a mistake there. P(box 1 | ¬yellow) = P(box 1) * P(¬yellow | box1) / P(¬yellow) = 0.5 * 1 / 1 = 0.5
    – Devsman
    Feb 15 at 18:32
  • Indedd P(box 1| ~yellow) = [P(box 1) × P(~yellpw|box 1)]/P(~yellow) = (0.5 × 1)/1 = 0.5
    – Hudjefa
    Feb 16 at 1:08
1

I was able to stabilize on P(E) = 0 using calculus. Unwrap the expressions and apply L'Hôpital's rule.

If P(E) is 0 and P(E|H) is also 0 you're just dealing with an infinite evidence selector. If you add error bars to your evidence gathering you ought to get P(E) is nonzero again. It's also possible that you have P(E) = 0 if you use the same kind of forged evidence filters I do; in which case the best conclusion is the evidence is forged, which is where you started.

If P(H) is 0 and P(E) is 0 then what we have here is the evidence as observed is telling you that P(H) is 0 because it's an inadmissible hypothesis and the evidence means at least one inadmissible hypothesis (not necessarily the one you are looking at) is correct. It's time to revisit all inadmissible hypotheses in the area.

If P(E) is 0 and P(E|H) is nonzero then P(H) had better be 0; in which case we have something interesting going on philosophically. I can't fathom an infinite number of hypotheses (even countably infinite) because the power of the mind is limited; however that's what the math is saying here.

But there is one remaining case. I tried to write a diagnostic reasoning software engine in college. It didn't work at all because Bayes theorem has a problem. It does not account for the unknown unknowns; that is the things you don't even know are possible yet. My course writeup for why it didn't work at the end of the quarter gave several examples of newly emerging viruses that broke the rules.

2
  • Couldn't parse the answer, except that it implies a new theorem is called for.
    – Hudjefa
    Feb 13 at 0:18
  • 1
    @AgentSmith: Most of this answer is just applying second semester calculus to the Bayes Theorem formula. Only the last paragraph suggests a new theorem.
    – Joshua
    Feb 13 at 0:57
1

It is important to bear in mind that Bayes rule is about assessing the evidence under a hypothesis, but it is equally true for the negation of that hypotheses.

What does it mean if P(E) = 0?

We can write P(E) = P(E|H)P(H) + P(E|~H)P(~H) as H is either true or it isn't. So P(E) = 0 means that the observations are impossible under H AND under ~H. I suspect this is why it is a requirement of Bayes rule.

So if we have a case where the evidence we are talking about is continuous, e.g. a value from the set of real numbers, we could say that P(E) is zero because there is zero probability of selecting particular value from an infinite set. However, we can also write P(E) by marginalising over H and it's negation. Say H is the hypothesis that the value is exactly equal to PI=3.14... , then ~H is that the value is something other than PI. In that case P(E|H) is 1 if we observe PI and zero otherwise, and P(E|~H) = 0 if we observe PI and 1 if it is something else, and we end up with P(E) having a well defined non-zero value determined by the prior probability of H.

P(E) is the probability of the evidence within the model set up by the choice of hypothesis.

Alternatively, if P(E) = 0 means the evidence is impossible, then by definition you have not observed it and the assessment of the hypotheses under E can never actually happen either.

2
  • 1
    P(E) = P(E|H)P(H) + P(E|~H)P(~H) seems important. I made an edit to my question, basically stating P(E) = 0 implies P(~E) = 1
    – Hudjefa
    Feb 13 at 0:21
  • 1
    @AgentSmith ... within the model set up by the choice of hypothesis Feb 13 at 13:58
0

Suppose someone is told, “There is something called Bayes’s theorem, it says ‘P(A|B) = P(A) * P(B|A) / P(B)’.” We need to know what this string of symbols is supposed to mean. Imagine someone defines P(X) as a ratio of magnitudes, as in, “There are three red marbles in a bag of 7 marbles; the probability of drawing a red one is 3/7”. Or, maybe we could extend this to “possible worlds” theory: “There are 100 distinct possible worlds; in one of them you win the lottery; assuming each possible world is equally likely to happen, perhaps chosen at random, then the chance of you winning is 1/100”.

If “probability” is nothing more than trivial arithmetic ratios, at its foundation, then define P(X, S) as |X| / |S| - the probability of X in relation to S (or “within some enclosing space of total possibilities, choices, or elements S”) is the cardinality of X divided by the cardinality of S.

The problem so far appears to be that division is an operation needing definition. I am pretty sure cardinality is defined in terms of ordinal numbers and the concept of a function. Thus, when we say “a set S has cardinality 3”, we are actually saying “there exists a bijection between S and ordinal 3”. I have to go way, way more into mathematics to understand my options here, but at least commonly, multiplication is defined in terms of ordinal numbers as well, and division in terms of multiplication.

Assuming we find a way to define ratios of at least finite cardinalities (for our purposes - thus, rational numbers, not even real ones, yet), then we have a bit more grounded of this idea of “relative cardinality”.

At that point, we also need to think about what “conditional probability” is. At least the way we’ve been working so far, it doesn’t seem like a very important concept. It just means the enclosing space has changed in some way. For example, “the probability that I draw a red marble, given that I just drew a green one”, is a way of saying, “When I draw a green marble from a bag of seven marbles, I now have six marbles left, three of which are red.” Thus, the essential nature of the question of “probability” did not change - it is just the contextual set or reference set which updated. And perhaps there are many ways one can “update the reference set” - eliminate every other element, add in 3 times as many elements, or whatever. The ratio, a function of 2 parameters, only changes with regards to the “focal set” (i.e. X), and the “reference set” (i.e. S).

In this framework, if the conditional probability of something is 0, perhaps this is a way of saying, “function f transforms set B1 into B2, in some way, and A1, a subset of B1, into A2. Determine how f transformed A1 and B1; then recompute their ratios.”

I am going to admit this answer sucks but I am leaving it up to think aloud and motivate myself to delve deeper into these questions. This was an original train of thought which I have zero clue if it is correct. It requires a lot more thinking to go into many of its aspects.

(Related: fractional cardinalities)

1
  • Arigato for your answer. Any updates?
    – Hudjefa
    Feb 13 at 0:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .