How improbable does an event have to be before we can say it didn't happen by chance?

Question

What is the probability threshold below which we can confidently say that a blind process did not create the supposed event?

For example, how many heads in a row would we need to say that it did not happen by chance. Keep in mind when I say "did not happen by chance", I don't mean that there's a probability of 0 that chance created that event. What I mean is how many tosses would I need before I can say that something else (i.e. ~H if H is chance) is more likely than chance to explain it? (i.e. P(~H) > P(H))

In the case of coins, one might compare the priors of rigged coins etc but what if there is no alternative prior? What if the supposed alternative theory is an unknown one?

For example, say one saw a bunch of clouds spontaneously spell out "I am God" in front of our eyes. Apart from blind natural processes creating those clouds, we wouldn't have any other priors for alternative theories. And yet, no sane person would conclude that it happened by chance. This means that one doesn't need direct evidence or a direct explanation of some other alternative theory before concluding that something didn't happen by chance. So it begs the question: how improbable must something be before concluding it didn't happen by a blind process?

Put another way: what's more likely? An event being unlikely under a known process (say chance) or an event being likely under an unknown/possibly nonexistent process (say God)?

Answer 1

It depends on context. At one end we have:

"Mr Bond, they have a saying in Chicago: 'Once is happenstance. Twice is coincidence. The third time it's enemy action' " -Ian Fleming

This might apply if what you suspect is happening might put your life on the line, and you have reasonable grounds to think someone might want to do that ("Never go against a Sicilian when death is on the line!"). In casinos people who seem 'too lucky' get kicked out and banned, even without proof of cheating.

In science they use confidence intervals:

The 'gold standard' for new physics is six sigma certainty. That's only a quarter of one percent more than three sigma, but for results that could potentially overturn lifetimes of work by others, or form the basis of the next generation's work, the difference can be crucial. Here's a nice article by Sean Carroll about an anomaly in coin tosses before sports games: A 3.8-Sigma Anomaly. Personally I'd still look at whether some sneaky way of biasing the result is involved.

Specifically in physics, it's not uncommon to reference 'unlikely in the age of the universe'. 100 heads in a row has a likelihood of order 1 in 10^30. There are 10^17 seconds or so in the current age of the universe, so even at 100 flips per seconds this result would be extremely unlikely to occur in fourteen billion years. So, it would be a lot more likely to indicate a biased coin.

The results where we can't rigorously model things is the Philosophical topic of miracles, and whether a given one violates Naturalism. We could actually do a pretty good model of water vapour interactions; you'd have to give parameters for how fuzzy the letters are, but you could get a fairly robust likelihood. It's going to be unlikely in the current age of the universe. But what exactly the consequences you take are, requires more thought. You'd probably look to Bayesian reasoning, and there are a lot of priors for Naturalism. You might think: it is faked somehow (other examples? Technological possibilities?); I am mentally ill (did others see? Photograph?); we are in a computer simulation (Are we living in a simulation? The evidence).

It's a lot more likely we see faces in clouds: Pareidolia: Seeing Faces in Unusual Places. Babies respond to human faces before they can focus their eyes, so we know there is a brain structure as well as learning involved. People really do feel things they see in clouds have significance, and this has been linked to religious behaviour: Faces in the Clouds: A New Theory of Religion. Statistically, it's a lot more likely a human brain is 'a biased coin', than that we witnessed something unlikely in the age of the universe. At best, we might expect to infer deism, consciousness at some larger than human scale involved in the universe's setup, but with no evidence of interest in our daily happenings (eg, impacts of prayer - this is a whole topic).

Answer 2

Probability and chance are unrelated concepts.

Probability is a mathematical concept. The probability of a specific outcome is 1 divided by the number of possible outcomes in standard uniform distribution. In a normal probability distribution the probability is higher around the average value.

Chance is a philosophical concept meaning that the outcome is not selected on purpose by anyone. The opposite of chance is choice.

Each lottery result has a very low probability (1/several millions) and despite this one result occurs every week. We are shown the ball mixing machine and we can trust that each ball is selected by chance and not by choice.

Answer 3

There are several ways to answer that question, based on the mathematical science of statistics. Statistics furnishes us with equations which can be used to assess the statistical significance of a result- that is, how likely the result is to have been caused by some hidden but nonrandom cause as opposed to purely chance, as in the case of coin-flipping. Those equations yield numbers which cover a range of possible values; numbers on one extreme of the range indicate the results are indistinguishable from random chance while numbers on the opposite end of the range indicate the results are distinctly different from what one would expect on the basis of random chance.

Those equations also explicitly take into account the effect of population size (i.e., the number of events you are working with, called "n") on the robustness of the calculations. Large n means you can be more confident in the validity of the calculation, small n means less confidence. This gives rise to the saying, "anything can be proven with a small enough sample size", or the dismissive comment "yeah, but that's an 'n equals one' experiment".

These statistical tools are based on the data distributions that CriglCragl references in his answer. The mathematical field containing these general concepts is called descriptive statistics.

Answer 4

What about one in 100,000? There’s a 19th century adventure book about a group of people who leave Britain for almost three years. They left on my birthday and returned on my wife’s birthday. I’m sure our parents didn’t read the book. And I’m sure they didn’t know each other. I’m sure it’s a one in 100,000 chance.

No, you would have to examine exactly what happened. And instead of one event happening, there is probably a large class of events that could happen. My phone has a little code generator that produces a different six digit code every 30 seconds. The chances for a specific code are one in a million. The chance of some code are very very close to 1 (not quite one because my phone could decide to put itself on fire). And there are many events happening every day.

8 billion people on earth. Something that has an 8 billion chance happens to someone today.

Answer 5

A most interesting question. I believe CriglCragl is right on the money - his/her exposition on the mathematics &science of chance is on target. Kudos to him/her.

For what it's worth the OP's query can be summed up as a simple question: After how many consecutive heads/tails, do I come to the conclusion that the coin is loaded?