2

I understand the gist of all the definitions of internal and external validity I encountered to be the following:

  • Internal validity is the extent to which confounding factors can be ruled out to cause the measured effect.
  • External validity is the extent to which the results can be generalized to apply to other circumstances than those the study was conducted under.

However, if the measured effect varies with some circumstances, aren’t these circumstances confounding factors? If this were true, a study could not lack external validity without lacking internal validity.

I therefore find it hard to see why a threat to external validity wouldn't also be a threat to internal validity. Am I getting it wrong? I couldn't find any satisfying answer in the literature.

Improve this question
1
  • "If the measured effect varies with some circumstances, aren’t these circumstances confounding factors?" No. It is not enough for the measured effect to vary for something to be a confounding factor, the manipulated variable or something related to it also has to vary with them. And while that might have been ruled out for the circumstances of a study such ruling out may not be possible in more general circumstances.
    – Conifold
    Commented Sep 5, 2021 at 20:54

2 Answers 2

Reset to default
2

Suppose you do an experiment where you start with an initially room-temperature mercury thermometer, then stick it outdoors to see whether the mercury in the thermometer rises or falls as a result. You limit the scope of your experiment to a few days in August.

In your experiment you find that the mercury rises every time you put the thermometer outdoors. So you conclude that putting the thermometer outdoors causes the mercury to rise. This is the correct causal relationship (within the predefined scope of the experiment, i.e. a few hot days in August). So your study has internal validity.

But your study lacks external validity, because if you tried it in the month of January, or in a different part of the world, the mercury might fall when you put the thermometer outdoors.

Generally speaking, when you study something, you do so by first setting out a narrow scope. Then you try to work out cause and effect within this narrow scope. If you've done that correctly, your study has internal validity. But then someone might like to generalize your study's results to a broader context - out of the narrow scope in which you studied it. Causal relationships that hold in the narrow scope might not hold in the broader scope, and this would cause your study to lack external validity, without impacting its internal validity.

Improve this answer
6
  • I’ve seen many explanations like yours. What they do not answer is, why the circumstances that prevent the hypothesis to hold in the broader scope can not be considered confounding variables in the narrow scope. Say, the broader scope is applying the results of your experiment world-wide. But in Australia it is winter in August and the mercury falls. Why should the location not be considered to be a confounding factor of the original experiment, impacting its internal validity? After all, the outcome heavily depends on the location and it has not been controlled for.
    – 303
    Commented Sep 5, 2021 at 12:24
  • @303 because, within the narrow scope (of a few hot days in August in the place where you tried the experiment), the circumstances do not vary in a way that would change the result; they are held sufficiently constant. They cannot be a confounding variable when they are not variable.
    – causative ♦
    Commented Sep 5, 2021 at 17:19
  • @303 We may also say that it is possible to study the behavior of fire ants, without at the same time making claims about the behavior of army ants. Someone might later wish to broaden these claims to all ants. If the claims do not generalize to all ants, that is not a problem of internal validity, nor is the species of ants a confounding variable in the original study, which was only about fire ants.
    – causative ♦
    Commented Sep 5, 2021 at 17:24
  • The misunderstanding might lie in the meaning of "scope". As I understand you, it does not mean the circumstances under which the experiment has been conducted but rather the circumstances under which the hypothesis is claimed to hold. For example, if you claimed the results of your experiment to hold worldwide, the location would be a confounding factor and threaten the internal validity of your experiment. But since you merely claim the results to hold in the location you've conducted the experiment in, it is internally valid. Is that in line with what you're saying?
    – 303
    Commented Sep 10, 2021 at 8:25
  • @303 Yes, exactly.
    – causative ♦
    Commented Sep 10, 2021 at 8:26
1

The key is that scientist must define the scope of an experiment before collecting any data.

In the example causative gave, if the scientist limited their scope to the month of August, then the fact that temperatures change annually is not a confounding factor.

However, if the scientists used a much broader scope, such as an entire year, then the fact that they only gathered data in August made the annual changes in temperatures a confounding factor.

To put this another way, by defining the scope of an experiment the scientists are explicitly claiming that their results are accurate when certain factors are held constant. They are not claiming that these factors have no effect on the observed phenomena; only that the effect of these factors did not change between observations.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .