Does significance testing contain a logical flaw or not?

Question

This question was sparked from a comment Conifold had made. Link to comment here: Is probabilistic modus tollens a fallacy?

He says, and I quote, “The valid form used in significance testing is: If P then Q (if the null hypothesis then no significant observations); probably not Q (observations are probably significant), therefore probably not P (the null hypothesis is probably false).”

I am failing to see how this form of inference characterizes significance testing. Is it not the case that if the null hypothesis is correct, then it is improbable for significant observations to occur, not that no significant observations would occur? So if P = null hypothesis, and Q = no significant observations, it is not “if P then Q”. Rather, it is “if P, then Q is very probable.” Is this correct?

If so, is it then a fallacy to think that the null hypothesis is false or even probably false just because significant observations occur? Elliott Sober criticizes this reasoning and says that Fisher’s significance test contains a fallacy here: http://philonantes.free.fr/ElliottSober_IntelligentDesignAndProbabilityReasoning.PDF

If we take P = null hypothesis, and Q = significant observations occur, then the fallacy would be of the form

1. If P, then Q is very improbable
2. Q occurs
3. P is improbable

Sober argues that just because an observation is very unlikely under a theory, it does not imply that the theory is very unlikely. In fact, it may even count as evidence for that theory.

The reasoning is that evidence is comparative. If an observation is very unlikely under a theory, it tells us nothing about the theory’s likelihood. One must show what an alternative theory predicts before making that conclusion.

The example he uses is of an urn. Suppose there are two urns. One contains 2% white balls and the other 0.0001% white balls. Suppose now that one of those urns is in front of you. The hypothesis I am testing is whether or not this is the urn that contains 2% white balls. If I get a white ball, this would be evidence for my hypothesis, even though drawing the white ball in both cases is very unlikely.

So, to summarize, if one attains significant observations from the significance test, is it a fallacy to then make the inference just from this that the null hypothesis is probably false?

Answer 1

It is not so much that the hypothesis is improbable if we see significant observations, but that the frequentist considers that the hypothesis merits rejection. If the distinction seems subtle, see if this explanation helps.

Bear in mind that significance testing is a frequentist method, not a Bayesian one. The difference is important.

The frequentist is not trying to attach a probability to a hypothesis. For the frequentist, a hypothesis is a statement about how the world is: it is either true or false and that's it. We don't know which, but that's our problem. Having chosen a given hypothesis, the frequentist designs an experiment and constructs a probability distribution for the data that one might expect to observe on the basis of that hypothesis. In the case of a null hypothesis, the hypothesis takes the form of a claim of independence. For example in a drug trial it might be something like: this drug has no effect, so we expect to see no significant difference between subjects who take it and those who don't.

The frequentist then gathers a data set. The data set is 'random' in the sense that it is unbiased, i.e. the method of gathering the data is independent of the hypothesis. The frequentist then chooses to accept or reject the hypothesis based on how significant the result is. Significance levels are usually expressed as a p-value. This represents the probability that you would get this data set, or a similar or more extreme one, on the assumption that the hypothesis is correct. If the result is highly significant, i.e. the p-value is very small, then the frequentist rejects the hypothesis. How small the p-value needs to be depends on the circumstances.

A common mistake is to suppose that the p-value is the probability that the hypothesis is correct. This is completely wrong. It is the probability of getting the observed data set, or a similar or more extreme one, on the assumption of the hypothesis.

The Bayesian looks at things entirely differently. The hypothesis is uncertain and therefore we assign it a probability, which represents our rational degree of credence in it. The data is then used to update this probability. To the frequentist, the hypothesis has no probability, but the data has a probability because it is drawn at random from a large set of possible data observations. To the Bayesian, the hypothesis has a probability because we are uncertain about whether it is true, but the data does not have a probability because we have the data, so it is certain. The frequentist and the Bayesian have opposite perspectives.

The two approaches may give similar results. If highly improbable observations are made, the frequentist rejects the hypothesis while the Bayesian conditionalises and has a posterior probability for the hypothesis much lower than the prior. But the two can also differ. If the observations are not significant, the frequentist just says: I have no reason to reject the hypothesis. This does not mean there is a reason to accept it, just no good reason to reject it. The Bayesian happily updates on the data either way and comes up with a new posterior probability. There are other differences too: e.g. Bayesian methods treat nuisance parameters differently.

Bayesians usually prefer to use Bayes rule in its ratio form to compare rival hypotheses. Often we cannot say categorically that this data makes this hypothesis probable or improbable, only that it favours this hypothesis relative to some alternative. This seems to be what Sober is referring to with his example of the two urns.

To your final question, I would say that under some narrow conditions it may be OK to reject a null hypothesis based on significant data, but only when no alternative to rejecting it is available. If the drug trial shows that all 1000 of the patients who took the drug got better, while all 1000 who didn't died, that's pretty good evidence that the null hypothesis can be rejected. We might still want to check carefully that there are no confounding variables and the data set was completely unbiased and nobody was fiddling the results. But a highly significant result like that is pretty good. What alternative explanation for the result is plausible?

However, it is true to say that null hypothesis testing is overrated and overused. You might care to consult the book: The Cult of Statistical Significance by Ziliak and McCloskey. There is also a short paper by Amrhein, Greenland and McShane, published in Nature 567, 305-307 (2019), in which the authors, together with more than 800 other signatories, call for a restriction on the use of significance testing and confidence intervals. Speaking of confidence intervals, they are also frequently misunderstood and misused.

Answer 2

The quote is incorrect for the reasons you already outlined. However, significance testing in general would only contain a flaw if one uses significant observations by themselves to posit a likelihood on a hypothesis. The test itself merely computes the probability of an outcome under the null hypothesis.

In practice, before testing a hypothesis, we already have antecedent reason to think that it is plausible, either through a proposed mechanism, or some level of antecedent knowledge that allows us to attach a non zero probability to the hypothesis. For otherwise, there is not usually much incentive to test it in the first place.

Sober is correct in principle that evidence is fundamentally comparative, and thus obtaining significant observations by themselves tells us nothing about whether the hypothesis being tested is true.

However, in practice, when we do observe results that are very unlikely to occur under the null hypothesis, the prior plausibility of the hypothesis being tested, combined with the likelihood of the observations under the hypothesis, is usually higher, in Bayesian terms.

That is why for the most part, but not all the time, obtaining significant results is enough to make the inference that a hypothesis is probably (if this can be coherently defined) true.

Answer 3

In frequentism probability applies to the observed data given a probability model, but not to the model or its parameters itself. There is no well defined probability for a null or alternative hypothesis to be true in frequentism. Therefore a significance test cannot make statements about the probability that null hypothesis or alternative are true.

What a significance test formalises is the intuition that an event that is very unlikely assuming a model makes the model seem implausible. The p-value, or when reduced to a binary decision, "reject" or "not reject", regards how compatible the data are with the null hypothesis.

Further keep in mind that probability models are idealisations. A frequentist wouldn't typically believe that any precisely formulated probability model is true ("all models are wrong but some are useful", George Box). Therefore it doesn't make much sense to say that the null hypothesis or the alternative are "probably true". In fact none of them is. However, if the data do not "contradict" a hypothesis, i.e., they don't reject it significantly (which of course only "contradicts" the hypothesis with a certain error probability), the data at least do not provide evidence that the model isn't very good at modelling the process behind the data. On the other hand, a significant result, even though it doesn't provide evidence that any specific alternative model/hypothesis is true, can provide evidence that reality at least deviates from the null hypothesis in the direction formalised by the test, e.g., that for example a new medicine works better than a placebo on average in case the test statistic actually compares the average outcomes. This, so to say, is the comparative element in testing. (Specific alternatives such as two different normal distributions with equal variances and one mean larger than the other are thought constructs that allow to justify tests as optimal in specific situations, and "power calculations" giving orientations about required sample sizes, but this doesn't mean we should "believe" them. Frequentist probability models are not for believing them!)

The term "accept" is widely used in case the null hypothesis is not rejected by the test, but this is unfortunate and misleading, and I advise against it, because to some it seems to imply that we should believe that the null hypothesis is true, which really we shouldn't, see above. The term makes sense in industrial "acceptance sampling", where a batch of products maybe accepted as fit for selling/delivery if quality control on a sample doesn't give a significant indication against the required quality standard. This is quite different from most hypothesis testing in science.