3

I asked this question on Skeptics SE, and they referred me here.

I've heard some skeptics say that any anecdotal evidence should be dismissed immediately, as anecdote is notoriously unreliable.

While I do believe that anecdote should be given it's proper weight when evaluating a claim, I don't believe that it should be dismissed completely.

The question then becomes: how much weight, if any, should we give anecdote?

3
  • Evidence if not conclusive proof. Dec 2 '14 at 6:25
  • anecdotal evidence is not strong evidence, but adds a little bit of support. If it is the only thing you have, you better have a lot of it. Dec 3 '14 at 18:36
  • Statistics are comprised of anecdotes.
    – user18800
    Mar 25 '17 at 23:06
3

You probably really ought to ask somewhere that deals with statistics.

But the brief answer is as follows: yes, anecdotes can be informative. They can't be tested since there's only one of them, but they can still be highly suggestive. However, this is only true in Sherlock Holmes style: when you have eliminated the impossible, whatever remains, however improbable, must be the truth. This also works if "impossible" just means "way, way less probable than 'however improbable'".

The intuition is pretty clear. Swallowing a battery-sized model rocket engine ought not make someone explode. It might make them sick (it's probably poisonous) or hurt them, but not explode. But people really really don't go around exploding much.

Suppose you, one time, see someone gulp down a model rocket engine and then explode. Heck--you don't even need to see it, as long as you know it happened by some reliable source (i.e. it's much more likely that it happened, weird as it is, than that your source is wrong).

Now, you think: well, people don't go around exploding under normal circumstances; this would be incredibly incredibly weird that just in those few seconds after swallowing a model rocket engine he'd explode for some other bizarre reason. So, maybe model rocket engines actually can make someone explode.

You can quantify this with Bayesian statistics (and find that it is valid, at least with reasonable distributions of priors).

The problem with anecdotes is that people habitually underestimate the chance that the account is wrong, or fail to realize that the world has seven billion people in it and all sorts of stuff happens by chance and we select out the weird-seeming things to pay attention to. So if you want a rule of thumb: ignore anecdotes. But if you want to be statistically accurate: yes, they contain information (and can possibly justify you changing your model of what is likely by quite a bit*).

*(Addendum - the clearest case where an anecdote has huge power is a single instance of something happening that is said to be completely impossible. For example, if "all ravens are black", and you see a white raven, that's pretty good evidence against "all ravens are black". Of course it might not actually be a raven so without a lot of investigation it's not conclusive, but single examples can go a long way towards falsifying statements.)

1

The goal of gathering information is to gain knowledge. The aim of statistics is to measure the level of certainty in the knowledge we gain from information. That level of certainty is proportional to the number of observed events. Fewer events, less certainty. One event, very little certainty.

You also have to consider how closely each anecdote agrees with the others. In general, the variation in anecdotes is large due to the differences in the setup and also in the results. One of the biggest problems with anecdotes is that they are uncontrolled and you really don't know all the inputs and the number of inputs that could affect the result is often huge. Without more information it is usually difficult to turn this information into useful knowledge.

Take for example that someone eats a certain plant and is cured of a certain disease. Then another person also eats that plant and gets cured of that same disease. If that is all you know then that is close to useless because those people also ate many other things and did may other things which could all cause an effect, individually or in concert.

I'll also reiterate the point about Bayesian statistics. It is essential that you consider prior knowledge in any statistics. If something is thought to be likely you need less proof to "prove" it. Likewise, "extraordinary claims require extraordinary evidence". The failure to do this well is endemic in the social and medical sciences largely because there are so many input factors and it is so hard to control studies (think about controlling a study about murder - you can't ethically create this study, so you have to observe the uncontrolled world). But this applies to all anecdotal evidence as well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.