Imagine there is a quiz question that some researchers give to students at various universities. They find that a higher percentage of students at “selective” schools answer the question correctly than at “less selective” schools. The researchers infer that there is something on average about the cognitive traits of students at the upper-tier schools interpretable as something valuable, like “intelligence”, “critical thinking”, etc.

Someone proposes an alternative explanation. There is a distant race of psychic aliens who are interfering with the experiment. They want to create the false impression that the quiz question produces patterned data, seemingly hinting at some correlation. They are manipulating the experiment so more students at the high-status schools give the right answer when prompted to.

“Common sense” says the former sounds more likely than the latter.

Let’s reject “common sense” and try to provide a rigorous argument either for that the first hypothesis is

  • a strong candidate for an explanation of the observed data
  • at least preferable over the second hypothesis

or, argue that

  • there does not appear to be a good reason to assume the former over the latter, without further work done on the theory.

My intuition is telling me that a large body of people would prescribe the first interpretation of the data over the second, but the question is, why?

Obvious answers: Occam’s razor, Popperian falsifiability, or some sort of “probabilistic” argument for why “it’s more likely”. My issue with all of these is each is a large theory in its own right, resting on its own body of contingent assumptions. I am more interested to know what a priori, bulletproof knowledge a person could make use of in such a scenario to decide if they should accept that this is evidence that the average top-tier student is “smarter” in some way, or to affirmatively reject that interpretation as incorrect.

  • The study sounds "rigged" ... But it comes down to probability when doing it rigorously Commented Mar 23 at 16:47
  • 3
    "Bulletproof" and "knowledge" don't go well together.
    – NotThatGuy
    Commented Mar 23 at 17:12
  • From a probability perspective, it would come down to prior probabilities of the existence of the aliens (and their strange motivations). The Bayes factor (which tells you how much the evidence affects your conclusions (posterior belief) is likely to be close to one, so this is yet another question on this SE that boils down to strength of your prior beliefs. So scientific methodology can't reject anything because the evidence available is equivocal. Commented Mar 23 at 17:25

3 Answers 3


You are talking about phenomena in the world, the domain of science, and arguments based on evidence. Proof belongs in the realm of logic and mathematics, and involves truths found to be necessary given specific assumptions or axioms, but to have bearing on the world only in proportion to how sure we can be of those assumptions. See Cartwright's How The Laws of Physics Lie (links to book info).

Pursuing a Foundationalist approach to knowing about the world, of axiomatising physics, was part of the Hilbert Programme, and is considered to be shown to be impossible by Godel's Incompleteness theorems.

A Foundationalist approach to any kind of knowledge founders on Munchausen's Trilemma - look at the 'See Also' of this Wikipedia link for some alternative approaches to epistemology. Or see a general introduction like The Structure of Knowledge and Justification (Stanford Encyclopedia of Philosophy article).

  • 1
    Sadly, a big prepackaged set of perfect Philosophical answers appears permanently out of reach. Perhaps it should be the motto of this site?
    – Scott Rowe
    Commented Mar 24 at 23:34

Both explanations suffer from the hasty generalization fallacy:

A hasty generalization fallacy is a claim made on the basis of insufficient evidence. Instead of looking into examples and evidence that are much more in line with the typical or average situation, you draw a conclusion about a large population using a small, unrepresentative sample.


One school scores higher than another in a standard test. Why? There are a variety of possible reasons that require investigation.

Any reasons that can't be investigated, are the lowest priority. Psychic Aliens are not available for interrogation so let's move on.

The records of each school are available, interviews with parents, students, and teachers are possible. Financial records, IQ tests, statistics about the neighborhood, etc.

There is plenty of information/evidence available to prove or disprove the first explanation. If the first explantion is disproven, move to the next highest priority on the explanation list.

In research and development its called the process of elimination.


Why? Because by construction you've declared that the schools are "selective" with respect to social and mental faculties (grades, test scores, activities etc.). This phenomenon of selective universities has been a well observed phenomenon in human society for 100s of years. Thus within the framing of this hypothetical you've inserted a plausible explanatory mechanism for the outcome of the experiment that does not involve previously unobserved phenomenon (i.e. space aliens with mysterious powers).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .