What is the Texas sharpshooter fallacy?

Question

Please note in the comments if the question is too long and should be rephrased more concise. I am happy to do so if so wished.

---

The story:

The name comes from a joke about a Texan who fires some gunshots at the side of a barn, then paints a target centered on the tightest cluster of hits and claims to be a sharpshooter. (0)

I. Sources of description of the fallacy

1. wikipedia.org

stated that

The Texas sharpshooter fallacy is an informal fallacy which is committed when differences in data are ignored, but similarities are overemphasized. From this reasoning, a false conclusion is inferred. (1)

and also states that

It is related to the clustering illusion, which is the tendency in human cognition to interpret patterns where none actually exist. (2)

and

The Texas sharpshooter fallacy often arises when a person has a large amount of data at their disposal, but only focuses on a small subset of that data. (3)

2. rationalwiki.org

describes it as

The Texas sharpshooter fallacy (or clustering fallacy) occurs when the same data is used both to construct and test a hypothesis (4)

and goes on to explain

The fallacy's name comes from a parable in which a Texan fires his gun at the side of a barn, paints a bullseye around the bullet hole, and claims to be a sharpshooter. Though the shot may have been totally random, he makes it appear as though he has performed a highly non-random act. In normal target practice, the bullseye defines a region of significance, and there's a low probability of hitting it by firing in a random direction. However, when the region of significance is determined after the event has occurred, any outcome at all can be made to appear spectacularly improbable. (5)

The Texas sharpshooter fallacy uses the same data to both construct and test a hypothesis. A hypothesis must be constructed before data is collected based on that hypothesis. If one data set is used to construct a hypothesis, then a new data set must be generated (ideally, in a different way, based on predictions made by the hypothesis) to test it. (6)

3. philosophyterms.com

says

A Texas sharpshooter fallacy occurs when someone draws conclusions based on only the consistent data – the data points that are similar to each other — ignoring data that may not support the conclusion. This does not allow the data to paint the full picture of what is really going on. (7)

This fallacy gets its name from a story in which a Texas shooter fired many bullet holes into the side of a barn. He then drew a target around a tight cluster of bullet holes and called himself a “sharpshooter.” He wasn’t necessarily aiming anywhere in particular, but several of his bullets seemed to find a similar position in the barn wall. Drawing a target around the area made it look like he aimed, and succeeded in hitting, that particular area. (8)

II. Analysis

It seems there are multiple fallacies mixed into one. Let's break this down.

However, first we need to state that multiple things are at play here:

1. data - which is the input into our reasoning process)
2. method - which is our reasoning process
3. hypothesis - a model of the world we have, which can be the output of a method (constructing the hypothesis) or the input (testing the hypothesis). If we are testing a hypothesis. The hypothesis can be either be described as a model or as expected output. If it is described as a model and we want to test the hypothesis we need to see how well the data fits the "into" the model. If it is described as expected output, we either to see how similar the expected with the new, real data is.
4. result - which is the output of our reasoning process

For example we might put data into the method and get a set of clusters as a result. Another example is that we want to see if two things are

1. Fallacy: Similarity Illusion

This fallacy is about seeing a connection between the data and the hypothesis, because we emphasize the similarities between the data (structure) and the hypothesis (structure) and disregard the discrepancies. In practice this means leaving out variables/features that do not support similarities.

The task for the method is to check if two things are similar.

Here our method is not at fault, but the way we are filtering our parts of the data and the hypothesis.

In https://www.clocktimizer.com/a-quick-guide-to-data-fallacies-and-how-to-avoid-them/ this is contrasted to "cherry picking" (selecting results that fit your claim and excluding those results which do not fit your claim), but I do not see the difference.

• This fallacy is described by (1) and (3).
• The (3) is missing that the ignoring is selectively done, but fair enough.
• I think that (5) is aiming to explain this.
• I think that (7) is aiming to explain this.
• (8) mentions this when saying "Drawing a target around the area made it look like he aimed, and succeeded in hitting, that particular area."

Making the story work

The story (0) would work if we were to tell it differently:

The shooter shoots at a barn, all over the place (not at a single spot as suggested by some accounts). Then the shooter draws a circle around the place where there are many hits at the same place. Finally, the shooter patches up the other wholes that one cannot see them (the ignoring part).

2. Fallacy: Pattern / Clustering illusion

This fallacy is about seeing patterns in randomness. If one wants to be less general, one could say that a pattern is a set of clusters. Clustering is the task of grouping samples such that samples in the same group are similar to each other and samples in different groups are dissimilar to each other.

Here, we do not have hypothesis, we put the data into the method and get a clustering result.

The task for the method is to check if there are patterns (and which) in data.

The fallacy is with our method: The method returns clusters that actually do not exist. It does so by emphasizing similarities and dissimilarities, but disregard that these similarities and dissimilarities are actually not that great.

• This fallacy is described by (2).
• The (4) is mentioning "clustering fallacy" in parenthesis, but is talking about something else otherwise.
• (8) mentions this when saying that the bullets "seemed to find a similar position"

Making the story work

Not quite sure how...

3. Fallacy: Using the data to construct and test a hypothesis

Here are actually two reasoning processes at play. First we use data to construct the hypothesis than we test the hypothesis with data. If the same data is used in both processes, we commit the fallacy.

• The (4) and (6) are talking about this fallacy.

Making the story (semi) work

The story (0) would work if we were to tell it differently:

The shooter fixes the gun to a post and shoots the bullets at the same spot. Then he draws a circle around the holes. Finally, he shoots again (into the circle) and claims the gun always shoots into the circle.

4. Fallacy: Using the result as input

This is basically the same as begging the question, which, according to wikipedia, is

a type of circular reasoning: an argument that requires that the desired conclusion be true.

So, we are using the result as part of the premise, we could also say, as part of the reasoning method.

I am not sure what is the case here.

Somehow I first thought of this fallacy when I first heard the story (0), because the original story suggests to me that the point is that the shooter is claiming he aimed THERE and then "showed" that he did by drawing the circle.

I am not sure if Fallacy 1 and 2 are the same as this (I think not quite), but in all three cases it is a type of circular reasoning, it seems to me. However, according to the table at wikipedia, the Texas sharpshooter fallacy does not fall under "Question-begging fallacies" or is a "Circular reasoning" fallacy, but falls under "Questionable cause".

III. Questions

Basically my questions are

1. what did I get right,
2. what did I get wrong,
3. what the Texas sharpshooting fallacy is,
4. what it is not and
5. what those fallacies are called that are not the Texas sharpshooting fallacy?

Answer 1

I think all four of your models seem to miss the mark for me. (haha) A consideration which I'd like to introduce here is that there isn't anything in the fallacy that hinges on whether or not the conclusions we draw are true or confirmed by the data; merely that those conclusions are methodologically suspect

Let's start by digging into the "Sharpshooter", who I'll call Ted for the sake of brevity.

Let's suppose that Ted wants to be recognized as an expert shot, so Ted makes a decision to aim at one specific area of the barn. And, in fact, Ted is a sharpshooter, and so Ted manages spectacularly well to intentionally bundle all of the shots within the same small plank of wood on the barn. After having done so, Ted walks up to the cluster of bullet holes and draws the target around it.

All of Ted's shots do indeed form a cluster that would confirm the hypothesis if it were put in place prior to Ted shooting. The clustering and similarity are not illusory (contra 1 and 2), and Ted shooting intentionally at a particular point does not invalidate the data gathering itself (contra 4) (that was the point of the exercise, after all). Ted has still done something shady by drawing the boundaries after the shots were fired - namely, had some shots fallen differently, the target would also have been drawn differently.

The closest thing to this is your Fallacy 3 - Ted's drawn target (the "model") around the bullet holes (the data) is being presented as the shots being on target (the data confirms the model), when it remains an open question as to whether Ted could now continue to hit the target (will subsequent data back up the model).

But even this doesn't quite get the heart of the matter, because your sketch suggests that Ted is feeding the same data into the model, finding that it fits, and arguing that the conclusions are confirmed. That's already a step further than the Texas Sharpshooter takes it; Ted's shady act is drawing the target.

Our statistical counterpart is the idea that we've picked our model in order to post-hoc capture as much of the positive similarities of our data as possible - that is, we have overfit our model to the specifics of our data. We're arguing that our data as we've captured it supports a particular conclusion because everything we've tested fits that model, but we haven't been actually testing that conclusion as a hypothesis - it's just our methodology to by design produce the picture that most closely fits the data we do have.

Had we set out to specifically test the hypothesis that we eventually came up with, we would probably find that our data isn't really even good confirmation for that hypothesis. But the fallacy is in presuming the validity of a conclusion chosen by design to match the data, rather than the assumption of the quality of the data in confirming the hypothesis.

So I can see why the general treatment of the fallacy is as a "Questionable Cause" inference. It's not so much circular reasoning so much as we're just arguing from insufficient data, but holding it to be sufficient because it maximally satisfies our data set.

Answer 2

This question is all over the place, so I'm going to cut to the chase and explain the Texas Sharpshooter fallacy in simple terms, and then work from there.

---

The (not quite) Fallacy.

The Texas Sharpshooter fallacy is more of a caution about logic than a proper fallacy; that's typical of most informal fallacies. It rests on the understanding that there are two qualities involved in being a sharpshooter:

• Consistency, or the ability to reproduce (replicate) a result multiple times. This is tantamount to the statistical concept of 'variability': low variability means high consistency.
• Accuracy, or that ability to produce the result one aims at. This is roughly equivalent to the normative statistical concept of 'central tendency'.

Ideally, one expects a marksman to be both accurate and consistent: i.e., to aim at a given point (the central tendency) and produce strikes that are tightly clustered around that point. But it's perfectly possible for someone to be accurate and not consistent (to give a wide spread of strikes centered on the target point), or to be consistent and not accurate (to give a tight cluster of strikes centered on a point far away from the target point. Imagine a man who can consistently produce a tight grouping of strikes, but whose strikes are always clustered two feet to the left of the point he was aiming at. Imagine he then strolls over, moves his target point two feet to the left, and tries to claim the prize for marksmanship. That's the Texas Sharpshooter fallacy in a nutshell.

Now, this is not always the wrong thing to do. In fact, most of statistics and a lot of perfectly valid science uses this process to estimate means of populations by taking the means of sample distributions. The assumption in such cases is that the mean and variance of a sample — when the sample is selected by a properly random method — should serve as good approximations of the mean and variance of the population the sample is drawn from.

However, in the Texas Sharpshooter fallacy this assumption of properly random selection is thrown into question. The Sharpshooter is not using a sample of strikes to estimate an unknown population mean; instead, the joke is that he is changing a 'known' population mean — the actual target point he was aiming at — to match the sample he produced. As such, it's a kind of data fudging: like the (unfortunately common) practice of taking a sample of opinions from one political constituency and claiming that it represents the opinions of the population as a whole. And the difficult part of it, of course, is that unless the Sharpshooter specifies what he is aiming at in advance (or the political pollster specifies what group he is polling in advance) it's almost impossible to determine whether he is being accurate; all we know is that he's being consistent.

---

Statistical aside: I say 'almost' impossible because — technically speaking — strikes that are skewed significantly away from the mean will show an uneven distribution around this centroid. Variance will be elliptical or ovoid, not circular: greater in the direction of the mean than away from it. That can be detected with a large enough sample size. But does any one really want to know that?

---

The 'fallacy' here lies in an assumption about intention: is the person justifiably estimating the central tendency from the sample, or is the central tendency a pre-given norm that the person is skewing by using the sample CT instead? It's a sociological misdirection, not a logical (symbolic) error. I mean, there's a natural presumption that someone who can shoot consistently can also shoot accurately, but that natural presumption is not a logical necessity, and we leave ourselves open to misinformation if we don't take the possibility into account.