Logic of hypothesis testing

Question

The logic of hypothesis testing is this:

• State the hypothesis (called the null hypothesis)
• Get some data
• If the data is very unlikely under the assumption that null hypothesis is true then conclude that it is very likely that null hypothesis is false (called rejecting the null hypothesis)

Can you make this logical reasoning in a formal way? I don't think this reasoning can be made formal with predicate logic, should we use some kind of probabilistic logic?

My second question is: it is generally agreed that failing to reject the null hypothesis will not lead to accepting the null hypothesis. Why is this so is not clear to me. I'm asking this second question in connection to the first one since I hope that a formal treatment of the logic of hypothesis testing might also help with the second question.

Thanks

Answer 1

I can't help you with the first part of your question, but this is how I explain the second to my students:

I stand at the front of the class, and I flip a coin. Each time I do so, I look at the coin and I tell them that it has come down heads. I do this 10 times. At first they pay little attention, but by the 10th flip they are all amused, and have obviously figured out that I am lying to them. How have they done this? Well, if I were telling the truth, it's very unlikely that a fair coin would come down heads 10 times in a row.

What they are doing is a hypothesis test. Their (implicit) null hypothesis is:

H0: there is a 50% chance of heads and 50% chance of tails, and I am telling them which it is each time.

The likelihood of getting 10 heads in a row is 1/1,024 (or the likelihood of getting 10 of the same in a row is 1/512). As this is a very small probability, they decide to reject H0. They all do this without having come across hypothesis testing before.

Now consider a different situation.

I toss the coin, look at it, and tell them either heads or tails. I do this and tell them 4 heads and 6 tails. I do this irrespective of which way the coin actually comes down.

4 heads and 6 tails is a perfectly unremarkable result from a fair coin, and so they have no evidence to accuse me of lying again. However, I have lied. It doesn't matter how the coin comes down, I always tell them 4 heads and 6 tails. So while they have no evidence to reject the null, it's also not really evidence to accept it either.

Answer 2

There's a misunderstanding here. The null hypothesis is a nonentity: a statement of what we expect to happen if nothing actually happens. The null is merely statistical, suggesting what we would see if there are no effects beyond random chance. A statistical test tries to show that some effect exists which isn't a matter of mere random variation. If we don't get a significant result, we haven't shown anything at all.

In silly terms, if we see a UFO we have reason to believe that something unusual is happening, but if we don't see a UFO that isn't reason to believe anything either way.

Statistical reasoning is inferential, and doesn't translate smoothly to (deductive) formal logic. I'm not sure how far you'd get with that line of approach.

Answer 3

I don't know if there is much more to say than you did.

Let us consider the following reasoning:

1. if A, then B
2. not B
3. not A by 1. and 2.

This is a pretty common reasoning.

Let us now consider the usual statistical tests, stated as algorithms.

1. compute B from A; the computation is such that if A follows some probabilistic law, then B follows some other law;
2. check if B is between some bounds, and if not...
3. say "we reject A";
4. if B is between the bounds, say nothing, or "we do not have enough evidence to reject A".

I think it is okay to say that a statistical test is just an algorithm that mimicks the reasoning above, and nothing more.

Answer 4

"Can you make this logical reasoning in a formal way?"

No. There is a good reason why statisticians only say "we reject the null hypothesis" or "we fail to reject the null hypothesis", which is that a frequentist statistician fundamentally cannot assign a probability to the truth of a proposition. This is because they define a probability exclusively in terms of a long run frequency, and the truth of a proposition has no (non-trivial) long run frequency - it is either true or it is false. To get around this, frequentists assign a probability to a (usually fictitious) population of experiments, rather than the experiment you actually performed, so they can calculate their p-value. Unfortunately generally we want an indication of the plausibility of the research hypothesis, so we tend to take the p-value as an indication of that, but unfortunately it isn't, and mistakes/misinterpretation of results abound. Frequentist tests cannot say anything directly about the truth of a particular hypothesis and you need to be aware of the subtle substitution that is performed.

Also, the probability that the research hypothesis is true also depends on prior knowledge, so you won't get a logically reasoned formulation without them. This is nicely lampooned in an XKCD cartoon:

See my answer to this here on the stats stack exchange.

Similar problems apply to confidence intervals, see my answer here on the stats stack exchange.

"My second question is: it is generally agreed that failing to reject the null hypothesis will not lead to accepting the null hypothesis. "

Never say you accept anything on the basis of a frequentist null hypothesis statistical test - rejection of a null hypothesis does not imply the alternate hypothesis is true. Part of this is because there is no logical connection between rejecting the null and the alternative hypothesis being correct, it is just a convention that we require rejection of the null before promulgating our research hypothesis as a measure to enforce self-skepticism. It is essentially often little more than a ritual performed without understanding of the principles, just the ability to perform the mechanics of the test.

Note both Bayesian and frequentist statistics have their uses, but you need to understand both frameworks and know which is the more appropriate for the task at hand.

Answer 5

Suppose that you are testing the fairness of the coin. To perform the test, an experiment must be devised, in our example flipping the coin a predetermined number of times, say 12, and then the result analysed in three steps.

First, specify the outcome space. In our example 2*12 possible sequences of 12 heads or tails. The result of the experiment should be summarised in some numerical form, e.g. the number of heads in the outcome. This summary is called test-statistics.

Second, calculate the probability of every possible value of the test-statistics, given the hypothesis you are testing (Fisher called it the null-hypotheses). This is the sampling distribution of the test-statistics. In our case it is, with X the number of heads:

pr (X) = (12 over X) (1/2)*X (1/2)*12-X

Pr (0) = 0.000144141*

Pr (1) = 0.002929688*

Pr (2) = 0.016113281*

Pr (3) = 0.053710938*

Pr (4) = 0.120849609

Pr (5) = 0.193359375

Pr (6) = 0.225585938

Pr (7) = 0.193359375

Pr (8) = 0.120849609

Pr (9) = 0.053710938*

Pr (10) = 0.016113281*

Pr (11) = 0.002929688*

Pr (12) = 0.000144141*

Third, look at all results which could have occurred (given the null-hypothesis) and which, as Fischer put it, are more extreme than the result that did occur. It means their probability is less than or equal to the probability of the actual outcome. Then calculate the probability pr* that the outcome will fall within this group.

For example, if our experiment produced 3 heads in 12 flips, the result with less or equal probabilities to this are X= 0,1,2,3,9,10,11,12; and the probability of at least one of them occurring (c.f. * values in the table above) is pr*= 0.15. Fisher’s accepted convention is to reject the null-hypothesis just in case pr*<0.05. Hence our null-hypothesis of the fairness of the coin is not rejected.

Some statisticians recommend 0.01 or even 0.001 as the critical pr*. The adopted critical probability is called the significance level of the test, and the null-hypothesis is said to be rejected at this significance level if pr* is less than or equal to it.

“The null-hypothesis is rejected at a significance level” is a technical expression, which means that the result of the experiment fall in a certain region (declared “the rejection region”). But what does it really say about the null-hypothesis? Today the standard view (introduced by Neyman) is that a rejection or non-rejection of a null-hypothesis is not an inductive inference, but just an instruction for inductive behaviour. If we behave according to the instruction, in the long run we shall reject a true hypothesis H, i.e. we shall make a type I error, no more than once in a hundred times, when significance level is 0.01.

We may also worry, as Neyman and Pearson did, about accepting a false hypothesis H, i.e. making a type II error. The probability of type II error is the probability of rejecting a true alternative hypothesis, let’s call it Ha, by accepting the false H. The complement of the significance level of rejecting Ha is called the power of a test and, in this context, the significance level of rejecting H is called its size. An ideal would be to maximize the power and to minimize the size of a test. But that ideal is inconsistent. A contraction in size brings with it an expansion in power, and vice versa.

Apart from the volatility of what is declared to be “the rejection region”, the incoherence of contracting the size and expanding the power of a test, and considering only one or two hypotheses, there are other problems with this approach.

For example, different test-statistics may by defined on an outcome space, not all of them leading to the same conclusion when used in a significance test. This is the notorious problem of “which test-statistics to choose?”

There is also the problem of “the stopping rule”. Consider again that a coin has been flipped 12 times, giving 3 heads and 9 tails. Is this the evidence that the coin is biased? With the data provided, we cannot even begin to answer this question. Namely, from these data it is not clear what the outcome space for the data is. If we are told that the experimenter’s plan was to flip the coin 12 times, then analysis can proceed as above. But this is not the only way for these data to be produced. The experimenter may have planned to flip the coin until he produced 3 heads, or until he becomes bored with the flipping. In this case, the outcome space will be different, even infinite or ambiguous, and the final result of the significance test may also be different.

It seems that this approach is very subjective, because the identifications of outcome spaces, the choices of test statistics, the declarations of rejection regions, the choices of null -hypothesis among alternatives, the contradictory choices between sizes and powers etc., depend on thoughts or even whims of the experimenter.

But the basic problem of this analysis is that, in search of a rejection region, it evaluates a single hypotheses by taking into account data that could have happened. But what this possible data have to do with our problem? We have made our experiment, we have got the real data and we want to estimate hypotheses given this real data.

The result of this analysis is a behavioural attitude towards a single hypotheses, prompted by data that could have occurred but did not.

Contrary to that the result of the bayesian analysis is the probability distribution of every possible hypothesis H, given one real data set D.

But this is another theme.