Should X, if there’s no evidence for X, be given a non zero probability?

Question

There may be no evidence that a fairy is sitting on a table. Many argue that one cannot prove a fairy doesn’t exist. Thus, many decide to attach an (infinitesimal) probability to it existing, as many Bayesians regularly do.

However, attaching a non zero probability to fairies implies that it is possible for fairies to exist. But there is no evidence that they exist. So on what basis can one justify (even an infinitesimal) nonzero probability?

Is the better solution to refuse to attach a probability to this altogether?

Answer 1

You mention Bayesians, so I'll reply in that context.

The Bayesian framework is (roughly) a way to take a "prior probability" of a statement, multiply it by a factor accounting for new evidence (the "support", or the ratio of the probability of the new evidence if the statement was true to the probability of the new evidence independent of the statement), to get the probability of the statement with the evidence taken into account (the "posterior"). (Symbolically, P(H|E) = P(H) * P(E|H)/P(E).)

The general approach to "Bayesian" reasoning is using this with continuous updating. You can use the posterior probability after one observation/experiment as the prior for the next. With enough observations, the influence of the original prior you started with becomes negligible, and your statement of probability is (almost) entirely based on the evidence.

Well, mostly. You run into issues at the extremes of the probability distributions. If your prior is 0.0 (exactly), then it doesn't matter what happens with the support term. Anything times zero is zero, and using Bayes rule you'll never get a non-zero posterior for any statement with an exactly zero prior. (If P(H) = 0.0, then P(H|E) = 0.0)

Which means that if you have an exactly zero prior for a fairy sitting on a table, it doesn't matter if later on you get new evidence where you see a fairy sitting on the table, with the President and the Prime Minister awarding her medals, being covered by hundreds of reputable international news organizations. Anything times zero is zero, and all the Bayesian updates in the world won't budge "there's a fairy sitting on this table" above zero, regardless of how incontrovertible the evidence is.

Now certainly you could say "well, a fairy having a news conference with the President is a completely fluke event, and if that ever was to happen I'd just simply discard my prior and make a new one.". But that just shifts things. What do you think the probability of needing to remake your prior is? You've just admitted it's possibly non-zero. Why don't you then just include the probability of remaking your prior into the prior itself? (The Bayesian approach allows you to incorporate probabilistic evidence.)

This is why people who are following a Bayesian approach avoid assigning exactly zero probabilities to things. From a practical perspective there's very little difference between a probability of exactly 0.0 versus a one-in-a-centillion probability. (They both in practice mean "neither I nor anyone I'll ever hear about will see it happen".) But when used with Bayes rule a one-in-a-centillion probability has the possibility of being updated with enough evidence in a way an exactly zero prior does not.

---

So why not just refuse to assign a probability at all? Well, to incorporate evidence in a Bayesian fashion, you need to have a prior to update, so at some point you'll need to create a pre-evidence probability. Whether you do that now versus right before you incorporate the evidence is immaterial.

You likely have some evidence from prior experience to use for the prior. You have knowledge of tables and of fairies, so you're not completely at a loss for assigning probabilities. You have "evidence" in the form of prior experience seeing (or not seeing) other fairies on other tables, so you can form a prior on that basis. (Hopefully the specific evidence regarding this fairy on this table will be such that it doesn't really matter much what the prior was.)

And you can't really "refuse" to assign a probability to something if your choices and actions were to change depending on whether it was true or not. Every time you set a book down on a table, you're implicitly making a call on the probability that there was a fairy there that you would squish. The fact that I ignore the possibility doesn't mean that I refused to assign a probability to it, rather, it means I implicitly thought the probability was negligible. Contrast a situation where I have no evidence there is a child with a ball on the sidewalk in front of my driveway. I might "refuse to assign a probability", but that doesn't mean I shouldn't look as I back my car into the street. I may have absolutely no evidence for the specific statement, but there's a non-negligible probability that it is true, so it behooves me to check.

---

It should also be mentioned that often the Bayesian approach isn't used with precise probabilities, but rather probability distributions. That is, you can say things like "for my prior, I think the probability is somewhere between 1% and 2%, leaning toward the lower end of that range". It can get a bit complicated because you start talking about probabilities of probabilities, and often the form of the priors are chosen due to mathematical tractability rather than other considerations, but there are ways of dealing with uncertainty in the specification of the prior, and examining the way those predictions become more certain as evidence is incorporated. So just because you assign a probability doesn't mean you assign an exact probability.

Answer 2

In training ML models (as an example), it is important to account for previously unseen data, otherwise you will get nonsensical results when you do see something rare for the first time. In this case (and if using a probabilistic model), it makes sense to attach a small probability to events that were never observed in training.

It's actually (probably) better not to include something in a model than to include it but say it is impossible.

Out of the options:

1. include in your model as impossible
2. include in your model as possible with a very small, arbitrary probability
3. don't include in the model

Option 1. is the worst, it is basically guaranteed to come back and bite you. Rare events will happen eventually, and your choice is between your model being inaccurate at that point (options 2., 3.), or the model simply not giving sensible answers at all (option 1.).

Answer 3

This is known as Cromwell's rule, which states that

the use of prior probabilities of 1 or 0 should be avoided, except when applied to statements that are logically true or false, such as 2+2 equaling 4.

If you and I start with different priors that satisfy Cromwell's rule, our posterior probabilities will eventually converge as we accumulate evidence. If one or the other of us starts with exactly 0 or 1, we will never agree, regardless of the evidence.

BTW, to a Bayesian, probability is belief that is justified by evidence. If I get more evidence that either supports or rebuts my belief, the Bayesian probability changes.

Answer 4

Probabilities are empirical predictions about future measurements given past measurements: how many times should I expect to win if I play the game a certain large number of times?

I think you mean extremely small finite, not infinitesimal.

Literally infinitesimal probability and zero probability are identical over any finite number of trials because that's what infinitesimal means. 1-(1-dx)^n = 0 for any finite n, infinitesimal dx: you can play the game any finite number of times and still expect to win zero times.

Extremely small finite probability and zero probability are identical over any sufficiently small number of trials. Sufficiently small can still be impossibly huge. For example, there's about a hand grenade worth of available Potassium-40 alpha decay energy in your body right now. The probability of you exploding because a significant fraction of the Potassium-40 in your body spontaneously alpha decays all at once is finite. (I calculate order of 10^(-10^19) for a significant fraction decaying in one second. How's that for an exponent?) But that finite number is so low that if you populated every star in the observable universe with a trillion humans and ran the experiment for a trillion times longer than the time between now and the Big Bang, you'd expect to get exactly zero Potassium-40 explosions.

There is no difference between winning the game zero times if you play the game as often as it is possible to play the game (e.g. spontaneous explosion) and winning the game zero times if you play the game a bazillion times more often than it is possible to play the game (e.g. a macroscopic object quantum teleporting across the room), because you can't play the game more times than it is possible, and zero wins is zero wins.

Answer 5

Okay, I am going to try to plumb this Bayesian inference thing from the POV of an electrical engineer. In communications systems we use Bayesian inference regularly to postulate what symbol was originally transmitted when some particular signal was received. I think, when Bayesian inference is used in science, it is to consider the likelihood that some hypothesis is true when some other evidence is observed. The communications problem of the electrical engineer is simply a specific example of the more general inference of hypothesis, given evidence, used in science.

Now, if we call a specific hypothesis, H, and some set of evidence (that may or may not support the hypothesis), E, then Bayes rule (along with some other quantitative axioms of probability) says this:

Now, I would like you to focus on the third line of the conditional probability, P(H|E). The quantitative value depends completely on the second term of the denominator:

Now, if that term is equal to 1, then the entire dependent probability is P(H|E) = ½ and the odds (given evidence E) are equal and it's just as likely that fairies exist as not. If that term gets "astronomically large" in value, then it's astronomically unlikely that the hypothesis, H is true. But if that term gets extremely small, very close to zero, then the likelihood that H is true, given this evidence E.

So this is how a Bayesian reasoner would frame the debate. Now, of course, it depends on what the evidence E is.

The OP posits that there is no evidence that fairies exist. Well, I would like to, just for the sake of argument, say that perhaps that there are books that say something about fairies, that this be treated as "evidence". Now, it seems to me that the fact that these books that say that fairies exist, that this does not help the case for the existence of fairies.

Now, with the fairy hypothesis, let's say that the only evidence to support fairies is that there are books that talk about fairies. Now we have books that talk about George Washington. I have never seen George Washington nor have seen direct evidence of the existence of George Washington other than what I see in history books and in modern political discussion of American history. It is my judgement that P(E|¬H) is much much smaller than P(E|H) so, even if I start out with the initial belief that P(H)>0, but is not ≈0 (that the a priori likelihood that George Washington existed is not astronomically small) then all of that contributes to the value of that term above being very close to zero, so I judge that, given the evidence of George Washington that I read in the literature (and in all other references to the first president), the hypothesis that George Washington is an actual historical person is quite likely true. P(H|E)≈1 . It's because if Washington did not exist, then the likelihood of seeing this evidence is much smaller than the likelihood of seeing it if Washington did actually exist.

But with fairies, the evidence referring to and describing fairies in books would exist almost as likely whether fairies actually existed or not. In that case P(E|¬H) is about the same as P(E|H), so that ratio is about 1. So for me to conclude that fairies exist given the evidence in the literature, I would have to start with a reasonably strong a priori likelihood that fairies exist P(H)≈1. But if I start out with P(H)≈0, then the "evidence" in the literature does not help.

Now, however, suppose there existed some magical pixie dust, that had supernatural properties, as evidence E? That would change things. That is because P(E|¬H) << P(E|H) and that ratio becomes very close to zero. So then, for me to conclude that fairies don't exist, I would have to start with the a priori likelihood of fairies existing P(H) being astronomically small to begin with. So, in other words, if you are staring at evidence of pixie dust with supernatural properties, for you to conclude that fairies don't exist, in the face of this evidence, would require a strong a priori belief that fairies don't exist to start with. I would consider that to be a prejudice.

Now, in my opinion, a better hypothetical scenario to consider is more like this: Suppose you are seated at a poker table for the very first time in your life and you are dealt, for your very first hand, a Royal Flush in hearts. What are you gonna think? That you're a superb poker player? That you're extremely lucky? Or, simply based on the probabilities, that maybe someone is stacking the deck (and maybe whoever did that really likes you)?

The likelihood of a Royal Flush of a specific suit is 1 outa 2598980. That is P(E|¬H) where E is the fact that you're dealt this Royal Flush and H is the hypothesis that someone is stacking the deck. If those probabilities are not sufficiently astronomically low, consider the hypothetical that one person wins the Lotto six or eight times in a row. It's not impossible, but if it were to happen, based solely on the probabilities, all of us would reasonably suspect that someone is nefariously fixing the game.

I believe that the original intent about the original post is not so much about fairies, but is more about the existence of God. If that is the case, might we discuss this Bayesian reasoning in light of the intended question beneath the surface?

Answer 6

None of the other answers seem to entertain your final question, "is it better to refuse to assign a probability at all?". So here is an argument that probability (or, at least, exact probability) is just not appropriate for questions where there is no evidence.

Bayesian interpretation of probability is extremely flexible. As Simon points out, Cromwell's Rule makes discussion of absolutely certain events very trivial in a Bayesian framework - it is reserved for beliefs that cannot be persuaded by any evidence. That includes evidence that your senses are deceiving you, that the universe is a simulation, that you are a brain in a vat, and so on.

Conversely, if you take a radically skeptic position here and allow for doubt in anything besides your own existence, almost anything can be modelled as a probability between 0 and 1. You have some gut feeling about how likely the event is relative to other events, and that gives you a sort of ordering. The likelihood of fairies existing seems clearly lower than the likelihood of it raining tomorrow (after all, I've seen it rain before, but I've never seen a fairy exist). You keep going with your other beliefs until you've assigned a probability that's lower than any other probability you've ever observed. If you do more trials of seeing fairies not exist, your probability should be going down all the time, but there is no magic way to find out how fast.

If you are happy to take this position, then I cannot talk you out of it. If you are prepared to assign a confidence number, say, 1 in 100 quadrillion, to the event "everything I have ever perceived is a lie", then it is impossible for you to distinguish between your confidence in events that are 1 in a googol vs 1 in a googolplex. Your notion of confidence is just meaningless for probabilities that low.

However, I still claim you should not be happy to take this position. First of all, most people do not behave like radical skeptics. So you might allow some events about which you are sure. For those events, a Bayesian approach does not make sense (again, by Cromwell's Rule).

If you are prepared to behave like a radical skeptic, then you have to explain your confidences. What's the ratio between your confidence that fairies exist and your confidence that the sun exploded in the last 8 minutes? If you think that's a stupid question, I agree with you. But you can only say it's a stupid question if one or both of the confidences I asked about is unquantifiable.

You can avoid most of these problems by assigning confidence intervals rather than precise confidences. So, I know that my confidence of fairies existing is less than 1 in 100 million, but I can't make any claims beyond that. It could in fact be 0, or it could be 1 in a billion, or some other number, I can't tell. You can even apply Bayes' rule using interval arithmetic, but it won't tell you anything interesting.

Answer 7

You cannot assign a probability to an event that has already happened. Fairies either exist (probability 1) or they don't (probability 0).

You could, in theory, calculate a probability for a fairy popping up in existence randomly in the future, or for a current species to evolve to a species like fairy. But this was not in the question, which is about fairies existing now.