1

Consider a combat tournament with a large number (sufficiently large that small sample size is not a problem) of combatants, in which each match is zero-sum and has a winner (there are no ties). The rules specify that each combatant may choose to bring into the tournament exactly 2 of the following 3 "weapons": a pillow, a thimble, a sword. The sword obviously being superior to the other "weapons", ALL combatants choose a sword as one of their 2 weapons. For the other weapon though, the combatants are split: exactly half choose to bring a pillow, and exactly half choose to bring a thimble.

After the conclusion of the tournament, a statistician analyzes the results of the tournament. The statistician finds that:

  • combatants with a sword had a combined winning percentage of exactly 50% (remember that EVERY combatant brought a sword, so every win by any combatant with a sword meant a loss by another combatant with a sword, resulting in a winning percentage of exactly 50%)

  • combatants with a thimble had a combined winning percentage of less than 50%

  • combatants with a pillow had a combined winning percentage of more than 50%

Simple example with round-robin tournament of only 4 combatants to demonstrate the mathematics:

Alice and Bob both wield sword and pillow. Carol and Dave both wield sword and thimble. The 6 matches:

  • Alice defeats Bob.
  • Alice defeats Carol.
  • Alice defeats Dave.
  • Bob defeats Carol.
  • Bob defeats Dave.
  • Carol defeats Dave.

Then, standings:

  • Alice (sword and pillow) 3-0
  • Bob (sword and pillow) 2-1
  • Carol (sword and thimble) 1-2
  • Dave (sword and thimble) 0-3

Winning percentages by weapon:

  • sword 6-6 (50%)
  • thimble 1-5 (16.7%)
  • pillow 5-1 (83.3%)

Since pillow has the highest winning percentage, even higher than sword, the statistician concludes that a pillow is a better, more effective weapon than a sword.

What logical or statistical fallacy is this statistician making? I checked the Wikipedia list of fallacies and misuse of statistics articles, but those articles list so many different fallacies that I am uncertain.

6
  • Something is off with the setup. There could be combatants with a sword that won or lost all of their matches, depending on how the tournament is arranged. E.g. the winner and the first round losers, respectively, if it is an elimination tournament. It is hard to see how all combatants could win exactly half of their matches. – Conifold Feb 1 at 11:14
  • Re: "It is hard to see how all combatants could win exactly half of their matches": The question is not saying that each individual combatant wins exactly half his or her matches. It is saying that the COMBINED winning percentage of all combatants wielding swords is exactly 50%, which is a mathematical certainty given that ALL combatants are wielding swords and in each match, one combatant wins and the other loses. – SeekingAnswers Feb 1 at 11:17
  • Example: simple tourney of only 4 combatants: Alice and Bob both wield sword and pillow. Carol and Dave both wield sword and thimble. The 6 matches: (1) Alice defeats Bob. (2) Alice defeats Carol. (3) Alice defeats Dave. (4) Bob defeats Carol. (5) Bob defeats Dave. (6) Carol defeats Dave. Then, standings: Alice 3-0, Bob 2-1, Carol 1-2, Dave 0-3. The sword wielders are a combined 6-6 (50%). The pillow wielders are a combined 5-1 (83.3%). The thimble wielders are a combined 1-5 (16.7%). – SeekingAnswers Feb 1 at 11:57
  • 2
    The obvious mistake here is to consider each weapon separately when the available data has all the fighter using them in pairs (with probable synergy from their combined properties, i.e. historically "sword alone" > "shield alone" does not mean that "2 sword" > "shield+sword", as is made clear by the fact that very few fighters used 2 swords). This creates an artefact, as all one can interpret from this data is that the style "sword+pillow" is better than the style "thimble+pillow". – armand Feb 1 at 14:37
  • 2
    Whatever category of bad math that falls into, can we agree that the punishment for it should be to send the statistician to fight with two pillows? – ptyx Feb 1 at 23:41
1

IMO it is an example of "misunderstanding" the probability formulas...

Consider Bayes' theorem and let:

P(A) the probabilities of winning: assume 0.5 (everyone has the same chance),

P(B) the probability of having a sword: 1 (everyone has a sword).

Let P(B | A) the conditional probability measuring the likelihood of event "having a sword" occurring given that event "winning" is true. Also this probability must be 1, because every winner has a sword.

Let P(A | B) the conditional probability related to the likelihood of event "winning" occurring given that event "having a sword" is true.

We compute it using Bayes' theorem: P(A | B) = [P(B | A) P(A)]/P(B).

We have that: P(A | B) = 0.5, and this makes sense, because the probability that someone having a sword will win is half-and-half: everyone has a sword and half of them will win, while half of them will lose.

1st conclusion: it is not the same to use the information that someone has a sword in order to infer that he won versus use the information that someone win in order to infer that he has a sword (certainty).

If we apply the same formula to the pillow and thimble cases, assuming P(Pillow)=0.6 and P(Thimble)=0.4 what we get is, due to the fact that in both cases P(B)=0.5, that P(A | B) = P(B | A).

And also this makes sense: if we know that 60% of winners have a pillow, the probability that someone having a pillow win is 0.6.

2nd conclusion: we have no three "coexistent" possibilities here, because if we sum the probabilities, what we get is 1.5, that is impossible.

1

There are two flaws with this reasoning.

The first is that, because the weapons are paired, you cannot compare them individually, as that would constitute a fallacy of division. This is because you do not actually have data on the weapons (sword) or (pillow); instead, you only have data on the weapons (sword and pillow) and (sword and thimble).

Now, you might think that your data would at least allow you to conclude that (sword and pillow) is better than (sword and thimble), but depending on your sample size and how close the win ratios are to 50%, the difference might not actually be statistically significant, in which case you would not conclude that one is better than the other. In addition, since the combatants choose their own weapons (rather than having them randomly assigned), we would not be able to distinguish differences in the abilities of the weapons from those of the combatants.

10
  • As a side note, in my non-expert opinion, I do think that a sword paired with a pillow would be more effective than paired with a thimble, as the pillow could potentially be used for blocking in a manner similar to a cloak in the "cloak and dagger" style of combat. – Sandejo Feb 1 at 21:56
  • Actually this is the opposite of the fallacy of composition. The experiment here gives a joint probability distribution over the (weapon, defense) pairs, but the mistake is to make a comparison across dimensions (i.e. one specific weapon vs one specific defense). So one might call it a fallacy of decomposition. – Fizz Mar 5 at 13:28
  • @Fizz You're right; I got it backwards. Edited. – Sandejo Mar 5 at 21:15
  • It's not a fallacy to compare them individually. This is in fact done in many multi-factor statistical analyses; you have many possible influences on a result, and you try to quantify how significant each influence is. The only reason you can't statistically say the sword is better in this case is that you don't have information on how people do without the sword. If you did have that information, then you could make a statistical case for the sword being better, even if weapons are always paired. – causative Mar 5 at 21:51
  • @causative Even if some of the combatants used the (pillow and thimble), you still would not be able to draw conclusions about the (sword) by itself. For all you know, it could be the case that the sword merely enhances the capabilities of the pillow and thimble, while not being effective by itself. – Sandejo Mar 5 at 22:24
0

Ultimately this is a fallacy of relevance, although there's some flavor of a mereological fallacy in there as well.

The experiment design only allows calculating a winning frequency (thus estimating the distribution) of a joint probability of (win, offense, defense) where offence ∈ {sword} and defense ∈ {pillow, thimble}. From this one can deduce marginal frequencies but these are for the two dimensions of this design, i.e. sword vs... ahem sword and pillow vs. thimble.

The fallacy of relevance comes in because these offense and defense sets are disjoint and comparing their marginal probabilities with each other is no experiment at all. More concretely, as a [degenerate] three-way contingency table, your data looks like this:

                    fight
offense  defense  win  lose
sword    pillow     5     1
         thimble    1     5

The sword (6-6) numbers you gave are marginal frequencies (obtained by summing across defenses), as well as conditional frequencies--since there's only one level for that offense variable. There is no contrast in this design between sword and well... anything else, so of course if you do a Chi-square test for sword it is totally and perfectly independent of the marginal win-lose results, which are also 6-6 across all offense-defense pairs.

I had a professor who used to say "you can't compare one thing", which applies here too because the experiment design is basically doing that with sword... comparing it only with itself... and then drawing a conclusion about a comparison that wasn't actually tested.

As a kind of inference, the argument (from the real design & data) really looks like

Sword is no worse than sword (s ≤ s)
Thimble is worse than pillow (t < p)
---
[Therefore] Sword is worse than pillow (s < p)

Which of course is an invalid judgment, but is a mathematical argument rather than a purely logical one, meaning you need some definition of order, unless you reformulate order in terms of set-membership (which you can do, but it's unwieldy and will not illuminate the flaw in the argument better). In this judgement form, the "fallacy of relevance" is basically reflected in the fact that the two premises have no term (s, p, t) in common.

When probabilities/frequencies are added in a judgement like the above, the sleight of hand (which hides the lack of relevance to the complete neophyte) is that because the probably or frequency of... well... anything is a number, it looks like you can arbitrarily infer conclusions about any putative experiments by comparing those numbers. For instance,

The chance of me winning the 2024 (or any) election if I run as sole candidate is 1.
The chance of Trump winning the 2024 election vs a Democratic candidate is less than 1.
---
[Therefore,] I have a better chance of winning the 2024 election than Trump.

Basically, the notion of probability is based on a [countable] partition of some set of events (see σ-algebra ). Comparing probabilities from disjoint sets of events (which happens when they are conditioned on disjoint scenarios: here one is that I run as sole candidate, the other is that Trump runs vs a Democratic candidate) is meaningless as to the relative chances (probabilities) of those events that can only occur together in some larger experiment (here, an election in which myself, Trump, and Democratic candidate would run in the same election).

There mereology fallacy angle in the experiment you gave is that because sword, pillow, and thimble are "tested together", one might conclude they (all pairs) are tested against each other, but that's not the case for all pairs (and in fact only one pair/contrast is actually tested.)

0

The essential part is this:

Since pillow has the highest winning percentage, even higher than sword, the statistician concludes that a pillow is a better, more effective weapon than a sword.

The first fallacy is the incorrect assumption that "winning percentage" on its own is a meaningful statistic. Anything that is used by everybody must have a 50% winning percentage. Someone with a clue about statistics would remove matches where either both or neither of the combatants have a particular weapon - which would leave us with zero matches sword against no sword. And then we would easily see that we have no information at all about the correlation between using a sword and winning, and with the given numbers, the correlation between having a thimble and winning can be explained with coincidence because of the low number of samples.

The second fallacy is the failure to see the difference between correlation and causation. For example, in a judo tournament I would expect a strong correlation between owning an olympic medal for judo, and winning the next tournament. It's a "real" correlation, not just due to coincidence, but it's not causation.

And the third fallacy is the failure to see the difference between causation and "being a good weapon". Let's say some competitors have carefully studied the other competitors, analysed their strengths and weaknesses, and written it all down on paper. Having such a paper will actually cause you to be more successful than others, but that paper is totally useless as a weapon.

There are probably other problems. One is ignoring the fact that a combination of weapons might be successful. A gun is not very useful in a fight, and a bullet is quite useless. But the combination let’s you kill one opponent from a distance and will beat any sword.

-1

The basic fallacy is poor experiment design; no control group is used for the sword (a control group for the sword would be a group of combatants without the sword). I don't think there's a specific fallacy name for failure to include a control group. We might say it's an example of the "post hoc ergo propter hoc" fallacy, but that's only speaking very broadly.

The first thing to realize is that on purely statistical grounds there is no reason to believe the sword is better. Everyone has a sword because they chose it because it's better, but nothing in the winning statistics indicates that. The same statistics would also be consistent with the scenario where everyone is required by the rules to have a sword even though the sword is no good.

It would in fact be reasonable, based purely on the statistics, to conclude the pillow is a better weapon than the sword. (That could be true if the goal is to knock the opponent down, and to injure him disqualifies you).

The next thing to understand is that the statistical power of any conclusion about the sword, pillow, or thimble is low. If you conduct a significance test, 95% of the time there will be insufficient evidence to reject the null hypothesis that all weapons are equal. This doesn't mean the null hypothesis is true, it means that this tournament is not a source of evidence for disproving it. To attain greater statistical power, you would need to require some of the contestants to not use swords, so that you can compare performance with swords to performance without.

3
  • Re: "The basic fallacy is poor experiment design": The scenario is that the tournament wasn't intended as an experiment in the first place, and the statistician has no power over the tournament's design or rules. The statistician is just trying to draw conclusions from data for a concluded event that wasn't intended as an experiment to test any specific variable. – SeekingAnswers Feb 2 at 0:23
  • Well, regardless of why the tournament took place, the reason conclusions about it have low statistical power is that there is no control group. We might say "low statistical power" is the real problem, and "no control group" is the cause. But it's not wrong to say there's no evidence the sword is stronger; just from the numbers, there isn't any. – causative Feb 2 at 0:26
  • Perhaps the true fallacy is applying statistics instead of common sense when there isn't much statistics has to say (due to the low power). – causative Feb 2 at 0:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.