Why do people perceive the randomness of events so poorly?

Question

People who are not trained in statistics and randomness (and even sometimes those who are) tend to draw horrible conclusions about whether an event is random or caused. Fundamentally my question is - why is this true?

---

Note that I am talking about random from a statistical perspective, with the example of a series of coin tosses will show up heads on average 50% of the time.

Edit: Clarifying the definition of "random" that is used - it is a macro-scopic, probablistic definition, as indicated in the above sentence. I ignore whether it is causal on a microscopic level - for example, calculating every force and influence on the coin to determine that it is going to be heads this time. As humans can't calculate these factors, to us the coin toss is a 50-50 proposition. You could extend the question to situations with other probabilities, such as 70-20-10 percent for 3 different outcomes - that just makes the analysis more complicated. Note that it may still be useful to question this definition...

---

This point was driven home by a professor of finance who's class I attended, who said he could identify a true sequence of random numbers compared to one that was made up. He handed out 3 pennies and then left the room - 3 students flipped the pennies 50 times each and recorded the results - meanwhile, 3 other students made up a series of 50 heads/tails sequences they believed would represent random tosses.

I guessed at his method and suggested one of the people making up a sequence put a long string of heads or tails together (say 6 or 7). They did not, and he ended up being able to correctly determine which 3 of the sequences were "real" from the pennies and which 3 were the "fabricated" sequences. The real sequences of coin tosses all had at least one string of between 4 and 6 in a row.

Humans seem to feel that a sequence of coin tosses is not random when the same result shows up more than about 2-3 times in a row. This is clearly not a mathematically sound basis for evaluating whether something is based on a random or probablistic frequency curve, but it seems to be inherent to most people on an instinctual level.

What is the cause of this misperception? Does it have a name?

Answer 1

Causality is one pattern. You could say that the tiger example of @SwamiVishwananda in your comments states that humans more often are abductive (thinking of a rule intuitively out if a single event that usually is more often right than wrong or where the wrongness is catastrophical, testing it afterwards by the experience of regularities) rather than inductive (having regularities over at least some events and tying these into a rule). Just because it is evolutionary stable to do so.

Causation simply is a rule that is not positively provable, so that the abductive > inductive principle holds as long as there is not enough data (regularities) in form of knowledge underlying. The problem where fallacies occur regarding probability at least in this interpretation lies between abduction and induction, because we can err there and do it more often than in inductive claims that have at least some evidence already. Still, abduction is a very powerful tool and perhaps the very source of Kuhn's revolutionary science.

For more on this topic and the reason why pragmatism evolved out of these thoughts can be read in the works of Charles Sanders Peirce.

Another layer of problems and in my understanding the main source of fallacies is that because inductive rules tend to be fulfilled, but do not have to in cases of probabilities, because this is the very nature of probability, even logic including inductive claims fails, say "This coin was heads for 1000 times now, I know from induction that coins tend to fall on both sides equally often. So there has to be a tails soon.". In normal, deterministic circumstances it would be correct to infer an upcoming event from regularities experienced. Here, it is not.

Conclusion: For understanding probabilities neither abduction nor induction in a normal way works, but only a meta-inductive knowledge that includes the total and utter failure of induction in the one case of probability, which is the essence of this concept. So you have to understand this concept and in addition that the events are falling under this concept. That is quite much you have to know, more than in any other empirical experience. I do not know of a name for this, though.

Answer 2

Looking at a bunch of numbers or other data and trying to judge whether it is 'random' (whatever your definition of random) is not easy. As you say, people are not good at inventing data that is supposed to be random. The effect you mention is called the clustering illusion: the tendency to overestimate the significance of runs or clusters. Another common error is to ignore Benford's law: financial auditors use this to identify cooked books.

It is true quite generally that people are remarkably bad at reasoning with uncertain information. And it is not just the lay person: scientists, researchers, economists, business people, lawyers, etc., no matter how well educated constantly tend to overestimate their ability to calculate with probabilities. Some of the common mistakes are confusion of the inverse, the base rate fallacy, assumption of independence, gambler's fallacy, confusing conditional probablity with marginal probability, Simpson's paradox, Berkson's bias, etc.

If you don't mind getting upset about a real life tragedy, look up the case of Sally Clark for an example of how an expert witness in court screwed up a probabilistic inference and it resulted in a serious miscarriage of justice.

Answer 3

I think the problem is that randomness is a folk concept. Everyone believes that they understand what is meant by randomness and that their understanding applies to all cases of perceived randomness, including statistical randomness.

For someone versed in finance, randomness is equated to unpredictability. The price movements of financial markets are perceived to be random since they are unpredictable. In a liquid market, such as treasury bonds, we expect successive trades to be initiated by a even distribution of buyers and sellers. If too many buyers flooded the market with orders, we would assume that there was some well define cause such as a change in interest rates. A financier would not see this as random, even if it still fell within the definition of statistical randomness from a purely transactional point of view.

While it is true that certain areas of mathematics give precise definition of randomness, even here different areas of mathematics give different definitions. For example, statistical randomness is not necessarily the same as information-theoretic randomness even if information-theoretic randomness would necessarily contain some degree of statistical randomness.

EDIT

As per your request, I shall try to make my answer more clear.

Everyone has a naive / folk idea of what they think constitutes randomness.

In the case of the coin toss example, unless one is familiar with the concept of statistical randomness it is this folk concept that will be applied, or possibly some alternative definition of randomness such as that used by the information theorist.

A typical folk notion would be "lacking pattern". When someone sees a "pattern" of what they consider too many heads or too many tales, either in consecutive sequence or in aggregate, they will automatically make the judgement that the sequence of coin tosses is not random.

The coin toss example is an interesting example. The statistical distribution of the digits of the expansion of π is another. To make this analogous to the coin toss, let's consider the binary expansion of π. This is statistically random. However, it can be shown that sequences of consecutive zeros or consecutive ones of length n occur for any value n. Thus, for example, there exists a sequence of a trillion, trillion, trillion consecutive zeros in the binary expansion of π even though there is a statistically random distribution of 0's and 1's. If such a sequence occured near the beginning, the most non-mathematicans would assume non-randomness.

Answer 4

From a genetic point of view focussed on group psychology, humans are not evolved to deal directly with nature. They are evolved to live in groups of people that selectively isolate them from nature. http://apa.org/science/about/psa/2011/11/human-evolution.aspx

Although there are strong survival forces toward correctly interpreting your environment, and those can favor a lack of objectivity on their own, social benefits and anthropomorphism of nature can more readily explain irrational biases of many kinds. http://www.sscnet.ucla.edu/comm/haselton/webdocs/HaseltonNettle.pdf?q=paranoid

(So if you want a name, this is a component of what some economists and sociologists call 'Gesellschaft' - the focus on the goals of one's groups and society over individual goals as a determinant of who we are.)

This perspective predicts several psychological biases that we can easily confirm experimentally. Three that I think interfere directly with our intuitions of probability include these:

1. a bias toward rapid rule formation,
2. a bias toward optimism, and
3. a bias toward seeing the customary as either objective or planned.

The first of these increases our ability to be educated. People give a few examples and we are meant to intuit a rule. (This is discussed as 'over-imitation' in the first reference.) So child rearing and work coordination are better. But it is not reasonable to look at nature-in-the-raw in this way, we need to back away very far from the idea that nature is actively teaching us the truth, in order to really learn from it.

The second of these allows us to take risks that might benefit our group, but may be unwise for us individually. We overestimate our chances of success, which allows us to make fatal or otherwise costly errors, but still succeed genetically, because our family produces a number of copies of our genes in individuals that will all collectively share the results of success. If we were objective about normal situations, most of us would be too passive or risk-averse for the group as a whole to use us to its benefit. (We see this in that fact that the severely depressed are the best at estimating odds. But do not integrate well into a functional family or society.)

The third allows us to take part wholeheartedly in groups' politics. We follow leadership that is not really there, taking direction from our peers that leads to collaboration and the formation of values. At the same time, we are motivated to look behind uniformity for (others') biases, or we would converge too quickly into rigid forms that serve only the leadership or the status-quo, and undercut the benefits of competition. We want to split our experience of rule-following into preferring agreement (the 'Social Exchange Heuristic') and avoiding being controlled (e.g. 'Commitment Skepticism'), so we lose sight of what ordinary, objective uniformity looks like.

Answer 5

Why do people perceive the randomness of events so poorly? @LightCC.

because randomness is a pure concept/speculation/abstraction/fantasy and the rationalist has still no idea how to connect his speculations back to phenomenon.

in your post, you suppose that the rationalist is right beforehand, but not a single rationalist can tell you why the rationality is relevant.

Answer 6

I am still groping towards answers here, so forgive my amateur attempt. But I suspect that "randomness" is a complex, inherently paradoxical term akin to "singularity" or "nothing" or "infinity."

You refer to the distinction between "random" and "caused." But this is a rather odd dichotomy. It assumes "uncaused events" or, in a Humean vein, events with no conjunctions or perhaps "no correlations."

Moreover, in Kolmogorov's definition, the random sequence cannot be compressed, reduced, or divisible by... anything not equal to "itself." Or at least, I believe this is a crude rendering of his meaning. So this too contains a hint of absolute "singularity."

This is a bit like the problem of parallel lines in Euclid's fifth postulate. Infinity surreptitiously enters the assumptions. From what viewpoint can we know that two lines never, ever meet? And if some observer is making such a claim about two lines, isn't that consciousness itself mediating them? Aren't the two "identified" as two "lines" and thus already "meeting" in the mind...or perhaps in the "visual plane," which is then abstracted out.

Something similar happens with randomness or having "no correlation" with anything else. From what viewpoint can we ascertain that there are no correlations, no hidden variables? Hasn't the observation itself already "correlated" the particular events, whatever they are? We argue that what we mean is simply that it is unpredictable, yet obviously nothing can be known to be "perfectly unpredictable." Unpredictability in the single event can only get down to 1/2 never less and never 0.

We simply cannot reason about, re-flect upon, or re-cognize singular events, they are like the Kantian ding-an-sich. Yet the idea of randomness introduces this assumption of a kind of noumenal singularity. It surely exists, but we cannot know it exists. Like zero, it is a kind of negative construct that we introduce into ordering processes for certain abstract manipulations, but nothing the eye or even the mind's eye can see.

So people may either use the term colloquially, as we do with "infinity," or they must actually practice and learn all the complex mathematics within which some assumption of randomness is inscribed and reified. So I see nothing simple or intuitive about it.

Answer 7

We think that when we look at truly random events, nothing unlikely will happen. For example, throwing a coin ten times and it landing heads up each time is unlikely to happen, so we think it isn't going to happen.

But in reality, there are two things against us: One, unlikely events will happen, just not very often. Ten coins in a row landing head up will happen about one in thousand times. Things that are more likely and that we still consider unlikely will happen more often. If something is so unlikely that it has only a one in fifty chance to happen, then during the 50 coin throws mentioned it is quite reasonable to expect that it happens once.

Two, there are many things that are very unlikely. Like 10 heads in a row, but also ten tails in a row, 5 heads followed by 5 tails or vice versa, alternating between heads and tails, and so on. That greatly increases the chances that something very unlikely will happen. If we can think of 20 things that each have a one in thousand chance to happen, then we can expect one of these twenty things to happen during just 50 coin throws.

So when we are asked to simulate coin throws, we will tend to avoid patterns that seem unlikely to happen, not aware that unlikely looking patterns are not really that unlikely.

Answer 8

I don't know that it has a name but I think the phenomenon that you refer to is related to the misconception that if you continue to flip a coin over and over again, it is most likely that the number of times it comes up heads will equal the number of times it comes up tails. In fact the opposite is true. If required I can show examples of this but anyone can work this out by simply creating the permutations and counting how many times the number of heads equals tails.

Based on my own prior dillusions based on this misunderstanding of "regression to the mean" I think there is an implication that there is some sort of force driving the later flips back when they have gone too far our of the norm like a probabilistic gravity.

Cause? Most likely because so many people, including teachers don't really have a firm grasp on the concept.