How does one show that a theory that explains only one or a few data points is less likely to be true?

Question

Let’s take the example of a coincidence. Suppose I pray to God to help me win two straight lotteries and I do. It would seem remarkable. From a personal standpoint, it might even seem so remarkable that I start believing in it.

But one cannot ignore all other historical evidence. Once someone takes the rest of the world’s history into account, the complete lack of divine intervention in it, how others have not had their prayers answered, etc, it seems to make the god theory more implausible.

However, one can always retort that god may have intervened only once: for me. How special! On the face of it, this itself seems implausible. But why?

If god is simply defined as a Being who intervenes only once to help me specifically, the rest of the background evidence of the world seems to matter less. How does one actually show that this kind of god is less likely to exist? In Bayesian terms, what is the rationale behind this having a lower prior probability? It is not as if god’s existence is the kind to result from a game of chance, so I don’t see how one can figure out the different likelihoods of different gods existing. If this cannot be shown, could it be that this retort is not actually as baseless as it seems?

Answer 1

It seems that you should distinguish between the possibility of the theory being correct and the possibility of it being incorrect, as well as its significance. If the theory has at least one positive example, it gives strength to the likelihood of the theory being correct; however, its significance might be considerably low, with a p-value close to 1. The difference lies there. On the other hand, if the theory has at least one counterexample, it will lean towards being incorrect, and the more counterexamples there are, the greater the significance. The p-value will approach 0.

Answer 2

If god is simply defined as a Being who intervenes only once to help me specifically, the rest of the background evidence of the world seems to matter less.

If god is such a being, then statistics and probability is a tool inapt to deal with him/her/it.

The thing is, the reason we try to make theories is to understand the past and be prepared for the future. To that end we compile data and build models about how the world works. And if we have an understanding how the world works (those models), we can make predictions how the future will look like.

So now that you have an expectation of how the world works (whether it's true or not; it isn't) you're able to quantify a deviation from that expectation. You can assign a level of credence to your measurements, idk you can measure the same thing multiple times and take the mean as value and the standard deviation as margin of error and there are mathematical tools that let you compute the combined margin of error from the individual components or you could argue that you're fine if the values are within an arbitrary margin of error, idk 5% might be a popular one.

However all of that rests on the core assumption that reality has some level of consistency and follows some sort of patterns. "Ideally" (with respect to predictability) you'd have a deterministic reality where you can map a cause and effect relation and are truly able to predict what's going to happen next or in a more realistic sense you'd at least have some regularity in your randomness like a dice throw or something like that. Where you can't predict what side the dice will show but for large numbers you can estimate that each side will come up with equal probability and so you can still predict for example that 100,000 dice rolls will yield around 16,666 +-833 (~5%) accounts of a "6" for example. So despite not being able to tell when they will come up you can nonetheless predict the outcome (within a margin of error).

However with a god that strikes once and once only without a precursor and without leaving a trace of their action or existence, you're pretty much out of luck when it comes to explaining and predicting anything.

Don't get me wrong the problem is not that you cannot come up with an explanation of what happened or a prediction of what will happen, the problem is that you can come up with infinitely many of those and each of them would be equally valid as long as it is within the vicinity of that singular data point.

Like you came up with an explanation of what happened: divine intervention. And it's quite easy to look at you your particular situation, the intervention and whatnot and make expectations of what will happen next. And if you do that you will likely gather more data with negative results making your particular theory drop from high to low credence. So even in the case of true divine intervention the scientific method would still work. If "god" would interfere frequently with the world we'd probably incorporate it as a "force of nature" and make models explaining and predicting it's actions. Even if it would be a person/agent interfering with the world you could apply some theology/psychology of a god.

However what you propose would go down the road of pseudoscience where you presuppose the conclusion and make it invulnerable. So if you presuppose that a god exists, that it only ever interacts with reality in that one instance and refuses to be tested, understood or revealed by any other means. Then it's very hard to refute that, "failure to replicate?" - "God doesn't want to be tested!", "alternative explanation?" - "God's way of doing it!".

The entire method of expectation - validation doesn't work if there is only one event and no expectation of another. However while that might seem like a strong and good theory to the naive, in reality it's completely useless.

The thing is it fails at all accounts of what a theory is supposed to do. It doesn't actually "explain" much of what happened in the past... "Sure a god did it, ... but HOW and WHY?" And it fails at making any useful predictions for the future. So in other words: it's completely useless and self-serving. And for that reason alone you'd probably reject it, even though it might be true.

So if you make it a testable hypothesis the likelihood of that would probably shrink rapidly given the discrepancy between expectation and reality. And if you shield your hypothesis against that by decoupling it from reality, you're making it pretty much useless as you'd have a case of pure unpredictable randomness, so your ability to make use of that is pure chance and with unlimited options pretty close to 0.

Answer 3

In Bayesian terms, what is the rationale behind this having a lower prior probability?

This is incorrect. The prior probability represents your state of knowledge before seeing the evidence, not afterwards. In Bayesian terms, the reason "a theory [hypothesis] that explains only one or a few data points is less likely to be true" is because of the likelihood, which measures how well the hypothesis explains the data points, not the prior.

I'm going to avoid examples based on deities or psychic abilities as they just play on cognitive biases and get in the way of thinking clearly about the reasoning. So lets choose a less contentious example instead. Assume we have only two mutually exclusive and exhaustive hypotheses: H1 the coin has two heads, and H0, the coin is fair and has a head on one face and a tail on the other.

We start by stating our prior belief. Being objectivist Bayesian for this sort of problem, I am going to assume that I have no prior knowledge which hypothesis is more likely to be true, so I will assign a uniform prior distribution p(H1) = p(H2) = 1/2.

We then need to determine the likelihood for each function, which is the conditional probability of observing the data under a specific hypothesis. Let X represent the number of heads, then for a single coin toss under H1 we have p(X=1|H1) = 1 and p(X=0|H1) = 0, and under H0 we have p(X=1|H0) = p(X=0|H0) = 1/2.

So lets assume that we have a witnessed a single head, then we can work out the posterior probability of each hypothesis using Bayes rule:

P(H1|X=1) = P(X=1|H1)P(H1)/P(X=1) 
          = P(X=1|H1)P(H1)/[P(X=1|H1)P(H1) + P(X=1|H0)P(H0)]  
          = 1*0.5/[1*0.5 + 0.5*0.5]
          = 2/3

and similarly:

P(H0|X=1) = P(X=1|H0)P(H0)/P(X=1) 
          = P(X=1|H0)P(H0)/[P(X=1|H1)P(H1) + P(X=1|H0)P(H0)]  
          = 0.5*0.5/[1*0.5 + 0.5*0.5]
          = 1/3

So H1 (two-headed coin) has a higher posterior probability because it "explains" the observations with a higher likelihood.

Now if coin tosses are independent (the probability of one coin flip doesn't depend on any other flip of the coin) then we can Use our posterior probability after the first coin flip as our prior belief for the second coin flip and apply Bayes rule again to update our state of knowledge after the second flip (which is also a head). So now P(H1) = 2/3 and P(H0) = 1/3,

P(H1|X=1) = P(X=1|H1)P(H1)/P(X=1) 
          = P(X=1|H1)P(H1)/[P(X=1|H1)P(H1) + P(X=1|H0)P(H0)]  
          = (1*2/3)/[1*2/3 + 0.5*1/3]
          = 4/5

and similarly:

P(H0|X=1) = P(X=1|H0)P(H0)/P(X=1) 
          = P(X=1|H0)P(H0)/[P(X=1|H1)P(H1) + P(X=1|H0)P(H0)]  
          = (1/2*1/3)/[1*2/3 + 1/2*1/3]
          = 1/5

So you can see that as we add more datapoints, our posterior belief in H1 grows (as long as they are all heads), but it is nothing to do with the priors, it is because of the likelihood. The more data we have, the more the likelihood can change our state of knowledge.

So returning to God, what an objectivist Bayesian would do is to assume that a-priori the existence of God (H1) is equally likely as the non-existence of God (H1) and say P(H1) = P(H2) = 0, or adopt a hyper-prior, which we have gone over on another (but not very different) question.

The difficulty is then how to specify the likelihood of the evidence (e.g. testimony of prayers being answered or not answered) under both hypotheses. I suspect that will be the sticking point as your view on how the likelihood should be constructed will be different for different people, and you will have to make do with a subjectivist Bayesian analysis. In that case different people may come to different conclusions, but at least both will have explicitly stated their assumptions in a quantifiable manner which makes them more open to testing than if they just stated their opinion in natural language. That is the benefit of subjectivist Bayesian reasoning - it is demonstrably rational (c.f. Cox axioms) and it stops people from hiding behind the ambiguity of natural language (for some that is probably a bug rather than a feature).

I really would recommend reading an introductory book or tutorial on Bayesian analysis before asking more questions about it, it would be a much more efficient approach.