Can an unlikely event be evidence of the existence of a new process?

Question

Note the word “existence” in the question where I’m trying to be careful with my wording here. This can be better illustrated with an example.

Take the example of the cheating process. Suppose one observes that John has won four straight lotteries, each of which only has a 1 in a 10 million chance of winning. This obviously seems to make it likely that John cheated. If H = John cheated and F = John won by chance, it seems obvious that H is more likely.

But there is additional information here that makes it more obvious that John cheated apart from the fact that his winnings were improbable. For starters, we already know that people cheat and have an incentive to cheat. Secondly, we also know that cheating is possible as a mechanism. But what if we didn’t know this?

What if H instead was = cheating in lotteries is possible. Does these series of observations make it more likely that H is true? Or must this H be independent of any observations or predictions?

If we didn’t know beforehand that cheating was possible and that people have cheated before, should we now believe that cheating as a process is more likely to be true after observing John win many lotteries?

Answer 1

It depends on your prior. If you are absolutely certain that John doesn't cheat (maybe John is a computer program incapable of cheating), then no amount of wins by John should convince you that he's a cheater, nor suggest that it's even possible to cheat.

On the other hand, it becomes very, very unlikely for John to win fairly multiple times. If there is any possibility that John may cheat, eventually this likelihood of cheating outweighs the likelihood of winning an arbitrary number of times. We may attribute the non-randomness of John's wins to many possible causes, of which cheating may be one.

Answer 2

If you’re looking for a more practical (less mathematical) answer, we need to define “evidence”. For this you might look to the legal system, which has standards like “beyond a reasonable doubt”.

Answer 3

This is exactly how new discoveries are made in the scientific world, as follows.

Imagine we understand the root cause behind some physical process- for example, the rules of a card game, which yield a well-defined probability of winning versus losing the game.

Now comes a new player who consistently beats the game and wins at a rate not accounted for by the rules of the game. He has introduced a second mechanism (a "cheat") by which the game can be won, in addition to the first one as established by the rules.

So, the presence of anomalous data which the known mechanism of some physical process cannot account for hints at the existence of an additional mechanism by which the process can proceed, that our existing model of the process does not include. That newly-discovered process then becomes the subject of research.

Answer 4

Evidence helps construct a story. And the number of different stories that can be created with evidence is inversely proportional to the quality and quantity of evidence. With only one unlikely event, a plethora of stories can be created that can be supported by that one piece of evidence..

For example, John claims that a psychic provided him with these numbers and the psychic received this information from an alien named Klaatu. The psychic saved Klaatu's life after his ship crashed in remote Washington State and in gratitude began giving the psychic tips about future events. John had rescued the psychic from a brutal assault. In return for his kindness and bravery, the psychic gave John the winning numbers without his knowledge. John didn't cheat.

If it can't be proven that John cheated, then his fantastic tale must be true (how else can his winning be explained) and is evidence of psychics, aliens, and knowledge of future events.

This is as likely as winning four lotteries in a row. Isn't it?

As, more evidence is accumulated, the number of possible stories that can be created from this evidence decreases until, ideally, only one reasonable story can be told.

Unlikely events are meaningless by themselves. More evidence is required to create a likely story: John cheated the lottery.

Answer 5

Strictly, the probability of possibility can only take 0 or 1. Things can be possible and unlikely, possible and likely, or possible and necessary.

The confusion here comes from importing the common language equivocation of vanishingly small probability and impossible.

If we rephrase your question I think you already know the answer:

There is additional information here that makes it more obvious that John cheated apart from the fact that his winnings were improbable. For starters, we already know that the probability of cheating is nonnegligible. But what if we had assessed that the probability of cheating was negligible?

What if z was the probability of cheating and H instead was z > x for some small but nonnegligible x. Does this series of observations make it more likely that H is true?