8

People who are not trained in statistics and randomness (and even sometimes those who are) tend to draw horrible conclusions about whether an event is random or caused. Fundamentally my question is - why is this true?


Note that I am talking about random from a statistical perspective, with the example of a series of coin tosses will show up heads on average 50% of the time.

Edit: Clarifying the definition of "random" that is used - it is a macro-scopic, probablistic definition, as indicated in the above sentence. I ignore whether it is causal on a microscopic level - for example, calculating every force and influence on the coin to determine that it is going to be heads this time. As humans can't calculate these factors, to us the coin toss is a 50-50 proposition. You could extend the question to situations with other probabilities, such as 70-20-10 percent for 3 different outcomes - that just makes the analysis more complicated. Note that it may still be useful to question this definition...


This point was driven home by a professor of finance who's class I attended, who said he could identify a true sequence of random numbers compared to one that was made up. He handed out 3 pennies and then left the room - 3 students flipped the pennies 50 times each and recorded the results - meanwhile, 3 other students made up a series of 50 heads/tails sequences they believed would represent random tosses.

I guessed at his method and suggested one of the people making up a sequence put a long string of heads or tails together (say 6 or 7). They did not, and he ended up being able to correctly determine which 3 of the sequences were "real" from the pennies and which 3 were the "fabricated" sequences. The real sequences of coin tosses all had at least one string of between 4 and 6 in a row.

Humans seem to feel that a sequence of coin tosses is not random when the same result shows up more than about 2-3 times in a row. This is clearly not a mathematically sound basis for evaluating whether something is based on a random or probablistic frequency curve, but it seems to be inherent to most people on an instinctual level.

What is the cause of this misperception? Does it have a name?

  • 1
    Our brains are wired to find patterns. Our ancestors found out quickly that it was better survival technique to think you saw a tiger in the bushes and be wrong, than to not see a tiger and die. Our brains look for patterns. – Swami Vishwananda Dec 16 '15 at 4:28
  • @SwamiVishwananda Is finding a pattern the same thing as drawing conclusions about causality? – LightCC Dec 16 '15 at 4:38
  • @SwamiVishwananda Then why don't we converge over time toward correct and objective patterns? As you point out, over-reaction to fear would increase individual survival, in theory, but humans are actually overly optimistic on average. – jobermark Dec 16 '15 at 23:09
  • @LightCC Yes, if drawing the conclusion is based on a 'perceived' (but not necessarily valid, or studied, or rational) causality. Conspiracy theorists draw conclusions on causality based on their perceived patterns observed. – Swami Vishwananda Dec 17 '15 at 4:30
  • @jobermark Why would we converge over time toward correct and observable patterns? We can't discern in a new and unobserved beforehand event what patterns will be observed. If you live in a modern city, your brain no longer looks for tigers in the bushes, but maybe your brain is looking for cheating patterns in the seller when buying something expensive and new? Most people may be optimists, but most err on the side of caution when buying something expensive... – Swami Vishwananda Dec 17 '15 at 4:39
3

Causality is one pattern. You could say that the tiger example of @SwamiVishwananda in your comments states that humans more often are abductive (thinking of a rule intuitively out if a single event that usually is more often right than wrong or where the wrongness is catastrophical, testing it afterwards by the experience of regularities) rather than inductive (having regularities over at least some events and tying these into a rule). Just because it is evolutionary stable to do so.

Causation simply is a rule that is not positively provable, so that the abductive > inductive principle holds as long as there is not enough data (regularities) in form of knowledge underlying. The problem where fallacies occur regarding probability at least in this interpretation lies between abduction and induction, because we can err there and do it more often than in inductive claims that have at least some evidence already. Still, abduction is a very powerful tool and perhaps the very source of Kuhn's revolutionary science.

For more on this topic and the reason why pragmatism evolved out of these thoughts can be read in the works of Charles Sanders Peirce.

Another layer of problems and in my understanding the main source of fallacies is that because inductive rules tend to be fulfilled, but do not have to in cases of probabilities, because this is the very nature of probability, even logic including inductive claims fails, say "This coin was heads for 1000 times now, I know from induction that coins tend to fall on both sides equally often. So there has to be a tails soon.". In normal, deterministic circumstances it would be correct to infer an upcoming event from regularities experienced. Here, it is not.

Conclusion: For understanding probabilities neither abduction nor induction in a normal way works, but only a meta-inductive knowledge that includes the total and utter failure of induction in the one case of probability, which is the essence of this concept. So you have to understand this concept and in addition that the events are falling under this concept. That is quite much you have to know, more than in any other empirical experience. I do not know of a name for this, though.

  • Thank you - very useful things in here to go research more about. I especially like the concept of a meta-inductive knowledge... – LightCC Dec 17 '15 at 1:35
2

Looking at a bunch of numbers or other data and trying to judge whether it is 'random' (whatever your definition of random) is not easy. As you say, people are not good at inventing data that is supposed to be random. The effect you mention is called the clustering illusion: the tendency to overestimate the significance of runs or clusters. Another common error is to ignore Benford's law: financial auditors use this to identify cooked books.

It is true quite generally that people are remarkably bad at reasoning with uncertain information. And it is not just the lay person: scientists, researchers, economists, business people, lawyers, etc., no matter how well educated constantly tend to overestimate their ability to calculate with probabilities. Some of the common mistakes are confusion of the inverse, the base rate fallacy, assumption of independence, gambler's fallacy, confusing conditional probablity with marginal probability, Simpson's paradox, Berkson's bias, etc.

If you don't mind getting upset about a real life tragedy, look up the case of Sally Clark for an example of how an expert witness in court screwed up a probabilistic inference and it resulted in a serious miscarriage of justice.

  • Thank you, very helpful. That's a lot of terms to work through though! – LightCC Dec 17 '15 at 0:20
  • But the question was 'why?' not 'in how many different ways?' There is no attempt to trace causation or frame a theory here. – jobermark Dec 18 '15 at 0:46
2

I think the problem is that randomness is a folk concept. Everyone believes that they understand what is meant by randomness and that their understanding applies to all cases of perceived randomness, including statistical randomness.

For someone versed in finance, randomness is equated to unpredictability. The price movements of financial markets are perceived to be random since they are unpredictable. In a liquid market, such as treasury bonds, we expect successive trades to be initiated by a even distribution of buyers and sellers. If too many buyers flooded the market with orders, we would assume that there was some well define cause such as a change in interest rates. A financier would not see this as random, even if it still fell within the definition of statistical randomness from a purely transactional point of view.

While it is true that certain areas of mathematics give precise definition of randomness, even here different areas of mathematics give different definitions. For example, statistical randomness is not necessarily the same as information-theoretic randomness even if information-theoretic randomness would necessarily contain some degree of statistical randomness.

EDIT

As per your request, I shall try to make my answer more clear.

Everyone has a naive / folk idea of what they think constitutes randomness.

In the case of the coin toss example, unless one is familiar with the concept of statistical randomness it is this folk concept that will be applied, or possibly some alternative definition of randomness such as that used by the information theorist.

A typical folk notion would be "lacking pattern". When someone sees a "pattern" of what they consider too many heads or too many tales, either in consecutive sequence or in aggregate, they will automatically make the judgement that the sequence of coin tosses is not random.

The coin toss example is an interesting example. The statistical distribution of the digits of the expansion of π is another. To make this analogous to the coin toss, let's consider the binary expansion of π. This is statistically random. However, it can be shown that sequences of consecutive zeros or consecutive ones of length n occur for any value n. Thus, for example, there exists a sequence of a trillion, trillion, trillion consecutive zeros in the binary expansion of π even though there is a statistically random distribution of 0's and 1's. If such a sequence occured near the beginning, the most non-mathematicans would assume non-randomness.

  • I am a little confused by what your getting at, especially with the 3rd paragraph talking about other types of randomness - the question gives a specific, well-defined type - could you be more clear on how randomness being a folk concept explains the misperception? – LightCC Dec 17 '15 at 0:18
  • @LightCC You're right. I haven't expressed my self clearly. Everyone has a naive / folk idea of what they think constitutes randomness. In the coin toss example, unless one is familiar with the concept of statistical randomness it is this folk concept that will be applied, or perhaps some alternative definition of randomness such as the information theorist. A typical folk notion would be "lacking pattern", so when someone sees a "pattern" of what the consider too many heads or too many tales they will automatically make the judgement that the sequence is not random. – Nick R Dec 17 '15 at 0:58
  • I suggest editing your answer to reflect the improvement. :) – LightCC Dec 17 '15 at 1:28
  • @LightCC I've added an edit to the end of my answer. I also included an example using the number pi. – Nick R Dec 17 '15 at 1:58
2

From a genetic point of view focussed on group psychology, humans are not evolved to deal directly with nature. They are evolved to live in groups of people that selectively isolate them from nature. http://apa.org/science/about/psa/2011/11/human-evolution.aspx

Although there are strong survival forces toward correctly interpreting your environment, and those can favor a lack of objectivity on their own, social benefits and anthropomorphism of nature can more readily explain irrational biases of many kinds. http://www.sscnet.ucla.edu/comm/haselton/webdocs/HaseltonNettle.pdf?q=paranoid

(So if you want a name, this is a component of what some economists and sociologists call 'Gesellschaft' - the focus on the goals of one's groups and society over individual goals as a determinant of who we are.)

This perspective predicts several psychological biases that we can easily confirm experimentally. Three that I think interfere directly with our intuitions of probability include these:

  1. a bias toward rapid rule formation,
  2. a bias toward optimism, and
  3. a bias toward seeing the customary as either objective or planned.

The first of these increases our ability to be educated. People give a few examples and we are meant to intuit a rule. (This is discussed as 'over-imitation' in the first reference.) So child rearing and work coordination are better. But it is not reasonable to look at nature-in-the-raw in this way, we need to back away very far from the idea that nature is actively teaching us the truth, in order to really learn from it.

The second of these allows us to take risks that might benefit our group, but may be unwise for us individually. We overestimate our chances of success, which allows us to make fatal or otherwise costly errors, but still succeed genetically, because our family produces a number of copies of our genes in individuals that will all collectively share the results of success. If we were objective about normal situations, most of us would be too passive or risk-averse for the group as a whole to use us to its benefit. (We see this in that fact that the severely depressed are the best at estimating odds. But do not integrate well into a functional family or society.)

The third allows us to take part wholeheartedly in groups' politics. We follow leadership that is not really there, taking direction from our peers that leads to collaboration and the formation of values. At the same time, we are motivated to look behind uniformity for (others') biases, or we would converge too quickly into rigid forms that serve only the leadership or the status-quo, and undercut the benefits of competition. We want to split our experience of rule-following into preferring agreement (the 'Social Exchange Heuristic') and avoiding being controlled (e.g. 'Commitment Skepticism'), so we lose sight of what ordinary, objective uniformity looks like.

  • I understand the logic, and did find some info on Wikipedia here: en.wikipedia.org/wiki/Gemeinschaft_and_Gesellschaft, but do you have any other sources or references to follow up with? I understand that you can test for these experimentally, and that is helpful - I find the reasoning that attempts to explain them a little less so without more data. – LightCC Dec 17 '15 at 0:14
  • Please add these references to your answer directly, they would be helpful and make it more complete. – LightCC Dec 17 '15 at 18:05
  • OK, killed the comment, put refs in the post. Not sure they are the best, but they probably ultimately point toward something more reasonable. – jobermark Dec 17 '15 at 18:58
  • Really good and interesting answer. The only point I am tempted to doubt is that of bias towards optimism. It seems to me that such bias requires an very large group and considerable time frame in order to manifest in aggregate. Though manifest it clearly does. – Nick R Dec 17 '15 at 20:51
  • @NickR I am not sure what you are doubting. You seem to agree (with the data) that this bias is real. Are you doubting it has any genetic basis? We find it in small, isolated cultures, as much as in big ones, and across groups with staggeringly different histories, as well as in other animals related to us who have complex societies: buffalolib.org/vufind/Record/1840501/Reviews. So most theories maintain an explanation that proceeds from some inborn social nature over one that is strictly cultural. – jobermark Dec 18 '15 at 1:02
2

Why do people perceive the randomness of events so poorly? @LightCC.

because randomness is a pure concept/speculation/abstraction/fantasy and the rationalist has still no idea how to connect his speculations back to phenomenon.

in your post, you suppose that the rationalist is right beforehand, but not a single rationalist can tell you why the rationality is relevant.

  • 1
    I am not sure from this curt post whether I agree with you or not, but I would like to hear you elaborate on this idea. I may agree with your concept of randomness as "speculative," but I am not sure how you are using the term "rationalist" here. – Nelson Alexander Dec 19 '15 at 4:58
1

I am still groping towards answers here, so forgive my amateur attempt. But I suspect that "randomness" is a complex, inherently paradoxical term akin to "singularity" or "nothing" or "infinity."

You refer to the distinction between "random" and "caused." But this is a rather odd dichotomy. It assumes "uncaused events" or, in a Humean vein, events with no conjunctions or perhaps "no correlations."

Moreover, in Kolmogorov's definition, the random sequence cannot be compressed, reduced, or divisible by... anything not equal to "itself." Or at least, I believe this is a crude rendering of his meaning. So this too contains a hint of absolute "singularity."

This is a bit like the problem of parallel lines in Euclid's fifth postulate. Infinity surreptitiously enters the assumptions. From what viewpoint can we know that two lines never, ever meet? And if some observer is making such a claim about two lines, isn't that consciousness itself mediating them? Aren't the two "identified" as two "lines" and thus already "meeting" in the mind...or perhaps in the "visual plane," which is then abstracted out.

Something similar happens with randomness or having "no correlation" with anything else. From what viewpoint can we ascertain that there are no correlations, no hidden variables? Hasn't the observation itself already "correlated" the particular events, whatever they are? We argue that what we mean is simply that it is unpredictable, yet obviously nothing can be known to be "perfectly unpredictable." Unpredictability in the single event can only get down to 1/2 never less and never 0.

We simply cannot reason about, re-flect upon, or re-cognize singular events, they are like the Kantian ding-an-sich. Yet the idea of randomness introduces this assumption of a kind of noumenal singularity. It surely exists, but we cannot know it exists. Like zero, it is a kind of negative construct that we introduce into ordering processes for certain abstract manipulations, but nothing the eye or even the mind's eye can see.

So people may either use the term colloquially, as we do with "infinity," or they must actually practice and learn all the complex mathematics within which some assumption of randomness is inscribed and reified. So I see nothing simple or intuitive about it.

  • 1
    It is better to talk about a probability-based outcome rather than a random one? Or would that entail the same problems? I did mention an event having a probability distribution as an alternative for "random event", as a way of discussing the type of randomness meant. Because regardless of whether or not it is possible to determine exactly whether the next coin flip is heads or tails, there is no practical way for a human to determine it, and the odds are roughly 50/50. Are you trying to redefine what I mean by randomness in the question? – LightCC Dec 18 '15 at 22:07
  • Not really. I admit that I'm digressing a bit, or more than a bit, but I see real contradiction with your use of "random" as the 50:50 outcome of an event. If I am "redefining" the term, I'm doing so mainly by abstraction, trying to take in more than a mathematical definition and more than the "colloquial" use....and by analogy to other terms that cause so much metaphysical mischief, such as "infinity" or "nothing." This is not an argument, but an ongoing effort on my own part to get at the meaning. And a response to "why the misperceptions." – Nelson Alexander Dec 18 '15 at 22:35
  • Randomness is a strange concept without a clear definition. One's understanding of random goes hand in hand with one's interpretation of probability. With physical probabilities, randomness relates to indeterminacy and is conventionally described using the term stochastic process and modelled using random variables. With epistemic probabilities, randomness relates to lack of information, so to say that something happens at random is not to say it is uncaused or indeterminate, merely that we don't know what caused it... – Bumble Dec 19 '15 at 6:02
  • Your question about knowing there are no correlations relates. We can think of scientific enquiry as the attempt to find predictability in what we observe. Determinacy and randomness act like the prongs of a fork: we like to find determinate relationships because they are easy to work with, so we impale as many problems as possible on the determinate prong and use the random one to hold everything that has (so far) resisted the attempt to find determinacy. Determinacy and randomness can thus be understood as methodological devices that we use to clarify our knowledge of the universe. – Bumble Dec 19 '15 at 6:03
1

We think that when we look at truly random events, nothing unlikely will happen. For example, throwing a coin ten times and it landing heads up each time is unlikely to happen, so we think it isn't going to happen.

But in reality, there are two things against us: One, unlikely events will happen, just not very often. Ten coins in a row landing head up will happen about one in thousand times. Things that are more likely and that we still consider unlikely will happen more often. If something is so unlikely that it has only a one in fifty chance to happen, then during the 50 coin throws mentioned it is quite reasonable to expect that it happens once.

Two, there are many things that are very unlikely. Like 10 heads in a row, but also ten tails in a row, 5 heads followed by 5 tails or vice versa, alternating between heads and tails, and so on. That greatly increases the chances that something very unlikely will happen. If we can think of 20 things that each have a one in thousand chance to happen, then we can expect one of these twenty things to happen during just 50 coin throws.

So when we are asked to simulate coin throws, we will tend to avoid patterns that seem unlikely to happen, not aware that unlikely looking patterns are not really that unlikely.

0

I don't know that it has a name but I think the phenomenon that you refer to is related to the misconception that if you continue to flip a coin over and over again, it is most likely that the number of times it comes up heads will equal the number of times it comes up tails. In fact the opposite is true. If required I can show examples of this but anyone can work this out by simply creating the permutations and counting how many times the number of heads equals tails.

Based on my own prior dillusions based on this misunderstanding of "regression to the mean" I think there is an implication that there is some sort of force driving the later flips back when they have gone too far our of the norm like a probabilistic gravity.

Cause? Most likely because so many people, including teachers don't really have a firm grasp on the concept.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.