2

Suppose Adam has a theory that goblins exist in his attic. He hears noises coming from the attic even though he knows noone is in there. He realizes that even though it is possible that goblins in the attic may be causing noise, he has no reason to think so. Other more natural accounts are more likely.

Suppose John, on the other hand, hears noises in the attic. From these noises, he postulates whether goblins are in the attic making noise. He again, after wondering this, makes the conclusion that natural accounts are more likely.

Although on the face of it the theory seems ridiculous either way, something about the second account is different. It was created to accommodate the data. In the former case, the theory was created before the data, even though technically, the theory doesn't actually predict the data (although in this case, the prediction may not make a difference to the conclusion).

Bayesians often argue that a theory's construction before data often is evidence of higher prior probability. But by this logic, does that imply that technically, the P (H|D) with H = goblins, and D = noise, is higher in the former case, even though both seem equally ridiculous?

Improve this question
5
  • Both do seem ridiculous, but where do you get that they are equally so?
    – Conifold
    Dec 28, 2022 at 14:21
  • Going from a noise in the attic to goblins in one leap? With such a small amount of information anything can be postulated and whether data leads or lags the theory becomes insignificant.
    – user59124
    Dec 28, 2022 at 16:16
  • I believe you should Google "Design of Experiments" as applied to statistics. IAnswers to many of your questions can be found here.
    – user59124
    Dec 28, 2022 at 16:29
  • "a theory's construction before data often is evidence of higher prior probability" is funny - after all, as in this example, we can construct any fanciful theory a priori. Adam could have imagined aliens, unicorns, etc, etc... and so just because he could think of any of these, it would be "evidence" of something? On the other hand, maybe this is similar to abductive reasoning, where if Adam could think about it, it's because he's seen similar things in the past, so his prior experience is summarized in his coming up with that idea, and those past experiences are "evidence".
    – Frank
    Dec 28, 2022 at 18:15
  • Maybe a theory is never constructed purely before data. It's probably always a case where a theory is figured out based either on conscious or unconscious prior data. So Bayesians could make an argument that no theory is fully ex nihilo, which lends some weight to a theory. Conversely, a true ex nihilo or random theory would have no claim whatsoever to being likely.
    – Frank
    Dec 28, 2022 at 18:18

3 Answers 3

Reset to default
1

Please keep "truth" away from "your knowledge of truth".

It don't matter you know or not know about a thing. A thing can exist without you ever knowing about it.

Theory itelf may be correct or wrong. It may happen that a pre-conceived theory (conceived before acquiring data) is correct. However, this approach is wrong. It will result in you having more wrong theories than correct theories.

"Suppose Adam has a theory that goblins exist in his attic. He hears noises coming from the attic even though he knows noone is in there". This is contradiction in terms. If he knows noone is there (in his attic) then he cannot have theory that goblins exist in his attic, not even a wrong theory.

"Does that imply that technically, the P (H|D) with H = goblins, and D = noise, is higher in the former case , even though both seem equally ridiculous?"

"Seems equally ridiculous" is putting horse before the cart. Why are you concluding ridiculousness BEFORE doing the analysis? Its like a judge already deciding a case before listening to either side. That is ridiculous.

Probability of an event is an objective reality. It don't matter what your pre-conceived theory / conclusion is, you are never going to get more than 12 as sum of output of 2 dice.

You conclusion do get affected ofcourse if you already have a theory. Its human weakness - biasness. What will happen is that you will look only at favourable-to-theory data if it exist, imagine it if it dont, while ignoring all counter data.

Improve this answer
1

Firstly, you are correct to note that the goblin-hypothesis in the first case already has very slender evidence and so is not a plausible conclusion on the facts. However, you are also correct that the second case is actually a bit worse, since it involves formulation of a post hoc hypothesis without any test against later data. The basis problem of using data to assess a post hoc hypothesis is that it is formed with the data under consideration, and so that data has no chance of counting against the hypothesis. When we form a hypothesis from data, it is formed as an explanation of the data that was observed, so it is inherently consistent with that data. This is why statisticians advise that if you form a post hoc hypothesis from data, you then need to get new data to test that hypothesis.

If you are examining this situation from a Bayesian perspective, you need to incorporate your knowledge that the post hoc hypothesis is formed from the data and so would not have occurred (and may have been a different hypothesis) if that data had not been observed. I do not agree with your characterisation of the Bayesian reasoning at issue here. The mathematics of Bayesian analysis incorporating hypothesis-selection is done by incorporating a variable specifically for the hypothesis-selection and is analogous to analysis involving missing data --- the hypothesis-selection variable essentially functions as the "missingness indictor" for the hypothesis of interest. If you examine Bayesian analysis involving missingness indicators you can establish that post hoc formulation of the hypothesis leads to a situation where the the posterior probability of the hypothesis being true conditional on being chosen is low even when the data is confirmatory to the hypothesis. This occurs because the hypothesis is only chosen if the data are confirmatory to the hypothesis.

Improve this answer
5
  • According to the famous Bayes rule, I suppose your "post hoc formulation of the hypothesis leads to a situation where the the posterior probability of the hypothesis being true conditional on being chosen is low even when the data is confirmatory to the hypothesis" means the maximum likelihood P(H|D) which is in proportion to P(D|H)P(H) would always be very slim here since the ad hoc P(H) is supposed to be very small even P(D|H) could be confirmatorily high?
    – Double Knot
    Dec 29, 2022 at 3:28
  • To frame this in Bayesian terms you need three variables, not two: you need the hypothesis, the data, and the hypothesis-selection indicator. Try expanding your analysis to look at those three together.
    – Ben
    Dec 29, 2022 at 3:44
  • If formulating in your parameterized Bayesian inference framework then we need an additional assumption that all possible hypothesis could be parameterized in a (continuous) parameter space θ with hyperparameters α, then my above simple 2 variables formulation just becomes P(D|θ,α)P(θ|α), which essentially has similar interpretation that OP's hypothesis likelihood is slim since ad hoc P(θ|α) is extremely small though P(D|θ,α) is reasonably high. Is such additional 3 variables formulation reasonable and worth it here for OP's attic Goblin conjecture? And how can OP get new data to test?
    – Double Knot
    Dec 29, 2022 at 4:27
  • No, that's not how you would do it. I recommend you have a look at the use of missingness indicators in missing data problems. The hypothesis-selection variable is similar to a missingness indicator (though it operates with respect to the hypothesis, not the data). There is no need to use continuity here and you can formulate things much more simply with a binary hypothesis and a binary hypothesis-selection.
    – Ben
    Dec 29, 2022 at 5:18
  • 1
    Missingness indicator can be used to identify which observations in a dataset have missing values, so that only the complete observations are included in the analysis. Similarly your hypothesis-selection binary variable proposed above can be used to identify which hypothesis is more relevant to the evidence. However, for OP's simple case of one hypothesis, this may sound overkill and my above 2-variable traditional Bayesian rule seems to be able to explain same effect/ Or even a more traditional frequentist p-value test could do? –
    – Double Knot
    Dec 29, 2022 at 6:09
0

It doesn't really matter if a theory is formulated first, then verified by data, or if data is found first then a theory is formulated. In the end, the data and the theory have to match anyway. Whether the theory and the data agree or not cannot be controverted.

Maybe if you start from the data, you may get a different theory than if you start with a theory then verify it (while avoiding confirmation bias, which is bread and butter of scientists). Maybe one theory has more explanatory power than the other. Maybe several theories can explain the same data. That need not be a problem though, if one realizes that a theory does not claim more than some explanation of the data.

There are subtler issues to consider too. For example, in physics, it's not uncommon to have an explanation of the data "to first order" only, or to have theories that do not contradict each other over some range of parameters but diverge in different regimes of the data. It's not always "either theory A or theory B", but in all cases, you have to match the data and make accurate predictions with the theory.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .