Metaphysics of statistical tests and the principles of identity of indiscernibles

Question

I am an imaging facility manager working with high throughput microscopy; we expect to compare different populations based on different statistical tests.

May I ask you for help to suggest how it is possible [if it is possible in general] to build a consistent system of the application statistical tests together with the principles of identity of indiscernibles https://plato.stanford.edu/entries/identity-indiscernible/ ? To be more specific, how can I accept that the two populations have the same variations if PII defines [to the best of my knowledge] that the the two populations can not be numerically identical, and the variation should be [in general] different? Would you be so kind as to suggest to me any publications which cover the robustness of statistical tests in combination with the PII?

Thank you in advance for your time and help.

Best regards,

Volodymyr

Answer 1

When you say, "how can I accept that the two populations have the same variations?" I suppose you are referring to a significance test for equality of variance. A significance test for equality of variance cannot reach the conclusion that the two populations have the same variance.

The most it can say is that the observations given to it do not disprove the hypothesis that they have the same variance.

Significance testing is always lacking in this way. In significance testing we try one way of disproving a hypothesis, and all we can report is whether that particular way of disproving it succeeded. If the hypothesis is true, we can never say so, because maybe some other way of disproving it would succeed.

In Bayesian analysis we would account for the prior distribution of the variance for the two populations, and we would say that, indeed, as identity of indiscernibles would suggest, the probability two populations have exactly the same variance is always extremely low, perhaps zero.

Still, a significance test for equality of variance is useful. If it rejects the null hypothesis, it means the variances are probably far apart and we have enough data to show it clearly. This is useful for deciding whether we should build a model with two separate variances or just one shared variance. Modeling with two variances comes at the cost of potential overfitting (even though, in fact, the two populations almost always do have different variances). More data, and more difference between the populations, lower the risk of overfitting.