It is usually said that the Bayesian probability is a subjective concept, quantifying one's degree of belief in something, while the frequentist probability is the the fraction of certain outcomes when observation is conducted many times (either in space or in time).

My simple questions are:

  1. If we consider statistics as a science, instead of a philosophy or even a psychology, isn't it necessary that a statistical prediction (or the outcome of a statistical calculation) be tested in the real world (at least in principle), as physical sciences usually do?

  2. If statistical calculations need to be tested (at least in principle), is the method of test the same as physical sciences, namely, through experiments and observations?

My thinking is that if the answer to both questions are yes, then the Bayesian probability can be interpreted as frequentist, and ultimately the two are not so different.

You may stop reading here and give your comments.

For example, suppose a geologist says that "there is a 0.9 probability that dinosaur was extinguished by a meteorite impact some 70 million years ago", a Bayesian may interpret the number 0.9 as the "degree of confidence" of this geologist.

To me, this "degree of confidence" is too "void" to understand. I would prefer to understand it as something like the following:

Given the current knowledge of the world, especially of the paleontological and geological discoveries, if there is a God who can create the universe for many times, then among the Earths (within these universes) that have been created and share the same features as our present Earth, dinosaurs are killed by meteorite impact in around 90% of them.

Well, of course, apparently there is not a God who is willing to do such kind of "experiment" for us (we may observe many extraterrestrial worlds instead, though that's also extremely difficult if not impossible in the imaginable future). But without such kind of "thought experiment", I can hardly understand what does the number 0.9 mean, and, how to prove that it is correct.

If the number 0.9 is not testable, even in principle or in imagination, or it is not meant to be tested, then I think it should not be spoken out in the first place.

But if you agree that "degree of confidence" also needs to be tested, then isn't it equivalent to the frequentist understanding of probability?

Note that in this example, the prior knowledge of paleontological and geological discoveries simply helps selecting out a subsample of the (hypothetical) universes in the "frequentist" approach to probability. I mean, if I were a frequentist (I am not sure whether I am, since I don't quite understand yet), I would not mind making use of prior knowledge, since using prior knowledge is not in contradictory with my "belief" in the frequentist approach.

Thanks in advance if you could point out errors or blind spots in my understanding of the two approaches!

Note: I asked this question at mathoverflow but was closed. They believe my question is too philosophical, so I ask it here.

  • FWIK many mathematicians share your reservations opposite Bayesian probabilities. E.g. Feller's text book doesn't go there. The Bayesian approach typically seems to appear in applied fields where some more pragmatism might seem warranted. I'm afraid bridging the gap between the two is not as easy as counting imagined frequencies if these are subjectively imagined. – Drux Sep 30 '13 at 14:28
  • @Drux Thanks for suggesting Feller's text. I just checked it, and find that I completely agree with his view (as presented in the beginning pages). But I still don't understand why these days the frequentist view seems to be out of fashion and the Bayesian approach is becoming dominant. – D.F.J. Sep 30 '13 at 16:52
  • 2
    Perhaps it's because the Bayesian approach has much wider potential applicability (ask any Wall-Street quant :) In many situations one cannot derive empirical frequencies: a-priori "subjective" probabilities are a convenient way out. N. N. Taleb has a lot of axes to grind with statistics (although not primarily with Bayesian but e.g. with "low" frequencies) and his books also touch on philosophy: you might enjoy those. At the other end of the spectrum, consult e.g. E. T. Jayes. – Drux Sep 30 '13 at 17:18
  • @Drux Thanks. I read a few paragraphs of Jayes before and find that he likes to attack frequentists in a not-so-pleasant way... – D.F.J. Sep 30 '13 at 17:30
  • Plus IMHO his arguments don't hold up also e.g. in light of what we meanwhile know from, say, behavioral economics. – Drux Sep 30 '13 at 17:32
  1. If we consider statistics as a science, instead of a philosophy or even a psychology, isn't it necessary that a statistical prediction (or the outcome of a statistical calculation) be tested in the real world (at least in principle), as physical sciences usually do?

Ultimately every scheme has some basis in experience, so tests are helpful in principle and necessary in the beginning of construction of such scheme (people played hazardous games and only then the probability theory came around).

However, presently the theory of probability (and statistics) is not a natural science. It is much closer to mathematics and logic and perhaps should be placed next to them. These schemes do not depend on tests for us to believe in their validity.

The probability calculation takes input data and assumptions, and produces probabilities.

What is usually tested here, is not the correctness of the calculation itself, but the validity of the initial data/assumptions.

  1. If statistical calculations need to be tested (at least in principle), is the method of test the same as physical sciences, namely, through experiments and observations?

Yes, but again, in doing the test, most often we question the assumptions (hypotheses), not the calculation itself.

First of all, I know this is more than an year old. However, I can’t leave this without making some observations, which may or may not answer the question (which the poster may or may not care about anymore).

The reason to make this post is two fold. One is to address your direct questions, and the other to address your thought experiment about dinosaurs.

  1. If we consider statistics as a science, instead of a philosophy or even a psychology, isn't it necessary that a statistical prediction (or the outcome of a statistical calculation) be tested in the real world (at least in principle), as physical sciences usually do?
  2. If statistical calculations need to be tested (at least in principle), is the method of test the same as physical sciences, namely, through experiments and observations?

I think it is a fundamental mistake to equate statistics with other “hard” sciences. I say statistics is a kind of “observational” science. It is more like Geology than Chemistry. For a Geological theory one can make hypotheses but only look for clues in order to deduce the correctness of the hypotheses and so on, unlike a Chemistry theory which can be readily tested in laboratory. (I am talking about “classical” Chemistry here, not the blurry Quantum Chemistry and the like). The theory of plate tectonics is a great example of how observational science works. One can never experimentally recreate it, and simulation is not the same thing.

Further, as modern Theoretical Physics has shown, things are kind of probabilistic in nature, at least at the sub atomic level, contrary to what we experience at macroscopic level. The experience humans have at the scale where we can manipulate object teaches us that things are fully observable and deterministic. Although the reality is not quite so, it is so close enough that we can ignore the discrepancies (not that we can perceive it anyway).

In some sense, the Frequentist approach believes in a deterministic world. Per that theory, all one needs to do is observe/measure enough number of times to get close to the “true” parameter defining the event. This kind of conflicts with the nature of the…err…nature as espoused by modern Theoretical Physics, so people might be a bit jittery in accepting it. It gets more puzzling if you imagine that every event that could ever happen has a predestined parameter that defines it.

To put it very loosely, the world is filled with randomness and the purpose of statistics is to quantify that randomness, which effectively precludes physical experimenting to verify because that would be contradictory to the idea of randomness.

For example, suppose a geologist says that "there is a 0.9 probability that dinosaur was extinguished by a meteorite impact some 70 million years ago", a Bayesian may interpret the number 0.9 as the "degree of confidence" of this geologist.[…] Given the current knowledge of the world, especially of the paleontological and geological discoveries, if there is a God who can create the universe for many times, then among the Earths (within these universes) that have been created and share the same features as our present Earth, dinosaurs are killed by meteorite impact in around 90% of them.

This was the real motivation to make this post. I think this formulation is quite wrong, not to mention simplistic and even a bit dishonest. Bringing something clearly imaginary like God into the discussion of science… :-)

Formulating a Matrix-esque scenario complicates the problem and diverges into all kinds of tangents. I would look at the number 0.9 this way. Lets us go back to observational science part. Let us say there are quite a few geological, astronomical, meteorological… events that could have occurred. Let us say each event leaves behind a telltale sign, a "residue". Let us say geologists gather a total of a million such residue. The number 0.9 could be interpreted as 900,000 of the residue pointing to a meteorite strike. Elementary probability theory suffices if you ask me.

If the number 0.9 is not testable, even in principle or in imagination, or it is not meant to be tested, then I think it should not be spoken out in the first place.

There may be some truth to this statement from a practical point of view. The German press currently tells us something like the following

Auf der zweiten Seite von Steinmeiers Prüfberichts steht, die Gesamtplagiatswahrscheinlichkeit liege bei 63 Prozent.

What does this 63%, i.e. the number 0.63 means in this context? It means they are man hunting again in Germany, which always feels strange to me when I see the havoc caused by people like Silvio Berlusconi at the same time.

But back to your question about the meaning of such numbers, this a number automatically generated by some more or less carefully designed procedure. The same is probably also true for probabilities reported by the weather forecast. The point about such automatically generated numbers is that you can try to calibrate the translation of these numbers into probabilities such that the deviation from actual frequentist probabilities is minimized. As such a calibration doesn't really modify the underlying procedure, the reported numbers can't be true testable frequentist probabilities. All we have done by the calibration is to convert the output of the procedure into a "measurement scale" which we know how to interpret (more or less intuitively).

Bayes Law, the principle that Bayesian statistics admits and classical statistics avoids is a true statement about what you are calling frequentist probability. So the distinction is moot.

The real question is whether predictions from psychology can ever really represent probabilities, or whether they are so "meta-fuzzy" that one dares not use them. That is not a question for statistics, it is a question for each domain of application to answer for itself.

In general, it probably only makes sense to begin to rely upon psychological statements of probability once one has 'tuned' the psychology extensively. So yes, all of that outcome testing is in order. But it is in order before using the measure, to confirm that you know the uncertainty in the predictions given by the source is not so high as to warn you off further math.

At that point the definitions converge, and you can do real statistics without arguing interpretation. But at that point you already know most of your answers.

First, to address the original question, I don't see the distinction between question 1 and question 2, and the answer would be: yes, statistical predictions need to be tested against reality. This is routine in statistics, and there are methods for doing some checking with existent data (for example, hold-out test sets, cross-validation, permutation tests, etc), as well as the experimental test of gathering new data.

This applies to both frequentist and Bayesian methods. If the OP means "let's do a bunch of experiments using frequentist and Bayesian techniques to see which performs better in the real world", then things get tricky because Bayesian methods are more flexible because of prior probabilities.

For example, if you use uniform priors in linear regression, you will tend to get answers very close to frequentist linear regression. Uniform priors are arguably not sensible, though. And if you have principled priors, you will get better results.

The OP believes that prior knowledge is not incompatible with frequentism. This is a core part of the (vociferous and long-running) Bayesian-frequentist debate: frequentist procedures assume no prior knowledge. A frequentist no doubt uses prior knowledge to form a hypothesis or to choose a procedure but the procedure itself doesn't incorporate any prior knowledge.

To address the comments:

Feller (1950!) is mentioned but the field of statistics has moved on since then. It would be informative to read de Finetti (1995) to see Bayesian statistics as a continuous version of logic. (Prior = Axiom, etc.) And there are now three Bayesian schools regarding priors: the traditional "subjective", as well as "objective" and "empirical".

Jaynes may be strident, but he was essentially replying in kind. Statisticians lost positions over being Bayesians, and the dismissive attitude of many frequentists was over-the-top, even if Feller managed to keep farther from vociferous than Jaynes.

I'm not sure what behavioral economics "proves" about the frequentist/Bayesian debate.

One of the comments mentions quantum mechanics, which reminds me that many (frequentists) seem to believe in a force of nature called Chance. They speak of things being caused by Chance. Perhaps they are two steps ahead, to the degree that quantum mechanics might affect the macro world. But if we look only one step ahead, Bayesians attribute to "uncertainty" what frequentists attribute to Chance, and that makes a lot more sense to me: we get distributions for things not because of some mysterious force (Chance) but because of things like measurement error or because our model is leaving out variables and our results are integrating over them -- creating distributions rather than points.

I debated leaving this posting alone, but I didn't like any of the answers. Of course, my liking something must be the defining methodology to determine truth!

My difficulty is that the conclusion doesn't follow from the premises in the questions.

My simple questions are:

1) If we consider statistics as a science, instead of a philosophy or even a psychology, isn't it necessary that a statistical prediction (or the outcome of a statistical calculation) be tested in the real world (at least in principle), as physical sciences usually do?

2)If statistical calculations need to be tested (at least in principle), is the method of test the same as physical sciences, namely, through experiments and observations?

My thinking is that if the answer to both questions are yes, then the Bayesian probability can be interpreted as frequentist, and ultimately the two are not so different.

The answer to 1 is yes depending on how you mean "as physical sciences do?" Statistics are often tested by numerical simulation which is a physical simulation in that it is performed using pseudorandom numbers. They are also tested through validation in specific studies, regardless of whether hard or soft.

The answer to 2 may be no, again depending on what you man about experiment and observation. Many sciences have to use observational studies and cannot perform experiments. The public might get angry if meteorologists created a category five hurricane just to see if a hypothesis about it was false. They may get even angrier if it is to test the sufficiency of building materials in Miami. Finally, some statistics don't really need to be "tested," because if their assumptions are met in fact and if their properties are understood, then a test doesn't serve any purpose except to waste resources.

What does not follow is that Bayesian probability can be interpreted as Frequentist and they are not different. This is not remotely true unless you have a sample size that is approaching infinity. In that case, their predictions will match. Subjective beliefs will converge to objective realities and bad objective models will vanish so that the final models will produce identical predictions.

Consider the cookie story in https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval

What is important here is to note that in Winstein's first table, the probabilities sum vertically. This is a Frequentist table. In his fourth table, the same data creates probabilities but they are summed horizontally.

Consider the case in Winstein's first table and assume the null hypothesis is that the type of cookie jar is class A. There is a 1% chance of observing a cookie with no chips given that the null is true. Now consider Winstein's fourth table. Bayesian methods have no null hypothesis to condition on. As a result, the Bayesian probability that the class of cookie jar is A given that no chips were in the cookie that was drawn is 1.9%.

These are very different probability statements. The former is Pr(data|model) whereas the latter is Pr(model|data). This is not at all the same thing. The Frequentist is, in a sense, trying to minimize the maximum possible damage that could be done from having no prior information of using a model to make a wrong choice. The Bayesian is trying to minimize the average damage, using all available information, from using a model to make a wrong choice. The Bayesian can do this because the data is choosing the model rather than the other way around as the Frequentist is.

Consider how the two would solve the simple linear regression problem ax+by+cz+d+e=0, where e is some error term, (x,y,z) are random variables and (a,b,c,d) are unknown constants. The Frequentist would assert that the model is the true model, using Fisher's "no effect hypothesis," would assert that a=b=c=0 as the null. If that is rejected, then the estimated parameters would be used. This is not similar to the Bayesian method.

The Bayesian method would calculate the probability that each of the separate following models is true:

  1. ax+d+e=0
  2. by+d+e=0
  3. cz+d+e=0
  4. d+e=0
  5. ax+by+d+e=0
  6. ax+cz+d+e=0
  7. by+cz+d+e=0
  8. ax+by+cz+d+e=0

A probability weighting would be applied to each one. Now depending upon the cost or utility function, one would either be chosen, or some or all would be averaged together. This can only be accomplished because a prior distribution over the parameter and model space exists. This prior knowledge allows the researcher to let the data dominate the solution. Indeed, if averaging is used, then the researcher truly averages the risk of loss over the entire set of possible outcomes.

There is another set of issues that make them different. Bayesian probabilities are coherent and Frequentist probabilities are not. Fair gambles can be placed on Bayesian predictions because of this. Frequentist probabilities and statistics cannot be gambled on because a crafty opponent can set up a bookie for a sure loss in all possible outcomes. Conversely, a Bayesian cannot discuss statistical power. There is also no guaranteed coverage against false positives. This is due to the fact that the Bayesian model, where Pr(model|data), is driven by only and exactly this one sample.

Power and guarantees against false positives come from the seeing the sample with respect to the sample space, implying infinite repetition. This is only credible under Pr(data|model). It is a valid model that inherits sample properties from this potentially infinite set of possible outcomes.

Because Bayesian methods work in the parameter space, things such as testing logical assertions or gambling on the results of an outcome are reasonable things to do. Because Frequentist methods work in the parameter space, things such as guaranteeing a minimum protection against false positives and a guaranteed level of power are possible.

Bayesian and Frequentist methods use the same data to answer very different types of questions. Bayesian probabilities cannot be interpreted as Frequencies. Conversely, classical Frequencies cannot be interpreted as probabilities as they are worst-case distributions and not actual distributions.

I guess the simple answer maybe: Bayesian and Frequentist define 'probability' in two different ways, so you can't simply do the interpret. In the simplest case, you observe the fact that dinosaurs exterminated, a frequentist can only give probability by saying 'ok, if infinite identical Earth undergoes some events, and became the one we see today, then some fraction of it is caused by comet hitting, while some are caused by some other reasons.' You see, there's no definition of probability for a single event for frequentist, you have to generate a 'infinite identical situations' to calculate the probability. While for Bayesian, there is definition for probability on single event. And I personally recommend this xkcd comic:

enter image description here

  • Yes, I saw that. I think you can define probability for a single event, but then, if your probability is about something in real life, and when it comes to "test" it in real life, namely, to test whether your calculation of the probability of a "single event" is right or not, what would you do? Or, maybe the Bayesian probability is not supposed to be tested? Of course, I understand that in reality many calculations cannot be tested since we don't have an infinite amount of resources. What I am asking is the "mental" model of probability (or degree of belief) in the minds of Bayesians. – D.F.J. Oct 1 '13 at 14:27

I think you're confusing statistics with the various arts of prediction, such as science, which use statistics.

Statistics is a math. It's a collections of axioms and proofs. In mathematics, not only is empirical data superfluous, it can be dangerously misleading. A famous example is Fermat's conjecture that 2^2^n+1 is prime for all n.

Of course, statistics isn't just any math, it's the most earthy. Developed by gamblers, thieves and gambler-thieves (bankers), statistics is the formal language for reasoning under uncertainty, and thus the language of science.

Just like one could use logic to "prove" the existence of a benevolent sky-beard, one could use statistics to compute the "probability" of a dinosaur ending meteorite. The validity of the conclusion depends on the validity of the assumptions.

The difference between Bayesian and frequentist methods is one of assumptions and less so one of machinery (no frequentist denies the truth of Bayes' theorem). Usually problems lend themselves to one method or the other, but with sufficient data, both approaches usually converge to the same solution.

Statisticians tend to be pragmatic about their choice.

I hope that helps.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.