Sometimes, people consider epistemology to be prescriptive rather than descriptive. (Worrisomely frequently, many fail to distinguish between the two at all.) I have never found my way through the Münchhausen trilemma, so I cannot speak about prescriptive epistemology. Some, however, include descriptions under the mantle of epistemology, and feel more comfortable speaking to that.
Jane asks her friend to think of a number between 1 and 100,000 and tries guessing it. She guesses it correctly. People are astonished.
If you ever witness this happening, I encourage you to note what happens next. My expectation is that this event is immediately followed by the demand that Jane do it again. The probability of this happening two or more times consecutively is vanishingly small, and that might explain our human reaction of disbelief, which is often appropriate: sometimes it is chance or coincidence. Repetition-- replication-- is how we understand how many trials something required. When Jane gets the answer right, we do not know many trials led up to that. When we demand that she repeat her performance, we are immediately starting a new trial.
However, there are very important, real-life examples of this principle-- not a paradox, but perhaps a cognitive deficit- that we can examine to talk further about it.
Let's say you design trials for a pharmaceutical company and you're told to get research showing that some drug works, when that drug is actually ineffective. Is there any way you can do that? Sure. You could run twenty different trials, perhaps distinguished in slight ways like outcome measurement, and then abandon any trials that are unpromising, or simply fail to publish any trials with a negative result. Even if you did try to publish all trials, journals are not interested in negative findings for a new drug, so those wouldn't get published anyways-- no blood on your hands. You're left, probably, with a single published trial, demonstrating only that people get better on Placebo A than they do on Placebo B about half the time.
Does this happen in reality? We can't look into anyone's mind to see if anyone is intentionally trying to push drugs that they believe are ineffective. However, yes, pharmaceutical companies run many more trials than are ever published. In combination with publishing bias, journal editors' biases toward positive findings, this leaves us with a misleading picture of research. This fact is perhaps part of what's responsible for the replication crisis in psychology and the claim that, probably, more than 50% of published medical research is incorrect. I don't expect that these issues are limited to either field, although I do expect them to be worse in some fields than in others.
Note that we don't even have to run multiple trials to do this; we can choose, after we have gathered data, how to interpret that data. We can easily imagine twenty different outcomes to measure for a drug-- perhaps all-cause mortality, perhaps days called in sick, perhaps hospital time. These are referred to as "researcher degrees of freedom." The more degrees of freedom the researcher has, essentially, the more tries they have at a positive result; those degrees of freedom are not restricted to any particular domain. They might even decide, after the fact, that some of the people enrolled in the trial weren't really eligible for the trial; maybe those eligibility rules were decided after the researchers already knew who got better and who got worse. Doing this intentionally is often referred to as "p-hacking," but there's no reason to assume this only ever happens intentionally-- my introduction to this problem was a doctor who, after categorizing the diets of pregnant women into a few hundred categories, proudly reported that breakfast cereal increases the likelihood of a male child. I don't believe that doctor was intentionally trying to mislead anyone.
Some of the most fun, headliner findings of science over the last few decades have been related to these degrees of freedom, from positive ESP findings to the brain activity of dead salmon. We might say in these cases that Jane tried to guess the number of 100,000 people, but none of those people were aware of the other guesses, and that Jane managed to stun one of them.
One of researchers' first acts to address this problem was correction for multiple comparisons. We can calculate the probability of getting one hypothesis correct by chance; we can also calculate the probability of getting one of any number of hypotheses correct by chance. So we can run a trial and consider multiple outcomes or multiple subsamples and still accurately report how likely the results would be if our explanation was false. However, this is of limited use with regards to other degrees of freedom, which are potentially uncountable, and especially, with regards to intentionally misleading anyone.
Perhaps the better solution is to demand that researchers rigorously document and share their plan prior to collecting any data. In the USA, this is now legally required by the FDA for many pharmaceutical trials (but keep in mind, there is a big difference between making a law and enforcing a law; I don't know the state of current compliance, but it has been spotty in the past.)
If we were to consider Jane's case, were she required to register her trial, she might document that she would make 100,000 attempts to guess a number, and then count how many times she got it right. Could she instead say that she would guess until she got it right? She could, but she would be criticized for doing so. For one thing, such a trial could potentially never end, and so it would call her methodology into question: if she never guessed correctly, she would never finish her trial, biasing the trial toward a positive result. So high quality research is never designed this way, and even when there are reasons for ending a trial early-- ethical reasons, perhaps, in the case of a drug that seems to save lives-- the decision to do so is still a reason to suspect the measured outcomes.
But note that researchers do not all necessarily agree that this is a problem, or that even if it is, that it is possible or worthwhile to do anything about it. And Society or Science aren't entities that can believe in anything; it is people and scientists who believe things, and they don't have to believe in those things for the same reasons. There remains significant resistance to trial registration and correction for multiple comparisons. And there are fields, say paleontology, where the concept of registration before data collection doesn't make a lot of sense. If there is some flowchart we can follow to true statements, then I don't know where it's kept, but perhaps we can make do with getting a lot of smart, educated people in one place and listening to them argue.
