Is Occam's Razor a Prior?

Question

In Bayesian statistics, the posterior probability of a hypothesis is composed of two parts:

• the prior, reflecting our initial belief in a certain hypothesis, and
• the likelihood, which represents how well our observations are explained by each hypothesis.

One way I interpret the prior is that it acts as a tie-breaker for hypothesis which explain the observations/data we have about the world equally well. If this interpretation is correct, is Occam's Razor an example of a generalized prior?

As a small example, consider the following setup: One afternoon I am cooking a steak. Just as I get ready to eat it, the doorbell rings. I take care of the situation, but once I return to kitchen, (i) the plate is shattered on the floor, (ii) the steak is gone, and (iii) my dog is sitting next to the shattered plate, happily wagging its tail.

What happened? Let's say I come up with (for simplicity only) two hypotheses:

1. When I left the room, the dog took the opportunity to steal the steak from the plate. In the process, he knocked the plate off the table. Now he feigns innoncence. Bad dog.

2. When I left the room, aliens teleported into my kitchen, trying to steal the steak. My dog valiantly tried to defend the homestead, knocking the plate off the table in the process. With their technological superiority, the aliens zapped my dog with an amnesia/happiness-beam, stole the steak, and teleported out of the room before I returned.

One of these hypotheses is clearly nonsense (my dog would never steal a steak... hehe), but both explain the three observations I have made. Hence, in a Bayesian sense, their likelihoods (given only these three observations) would be equivalent. The second hypothesis would probably be assigned a low prior for assuming a lot of unsubstantiated space magic, but aside from that, it's also more complex than the first hypothesis. So if I apply Occam's Razor and consider the second hypothesis inferior merely on the basis of complexity... would that constitute a prior?

Answer 1

You might be interested in Epistemic Justification by Richard Swinburne (he was applying Bayes to philosophical issues before Bayesian epistemology became cool), especially the section The Criterion of Simplicity (on pages 83-99 in the first edition, 2001). He would say the answer to your question is "yes".

He identifies four criteria used in comparing theories. The first three are related to the prior probability of a theory in Bayes' theorem (prior to a specific piece of evidence), and the last one is to do with posterior probability conditional on some piece of evidence.

1. Scope (how many objects is your theory trying to say true things about, e.g. "all orbits of heavenly bodies are ellipses" has greater scope than "all orbits of planets are ellipses", so the latter has a greater intrinsic probability)
2. Simplicity (see below)
3. Consonance with background beliefs (how likely is your theory given everything else you believe)
4. Explanatory power (how well does your theory account for some new piece of data)

Swinburne identifies six facets of simplicity (I don't think it's meant to be exhaustive or the only way to measure simplicity, and he does mention other approaches such as the computational approach of Solomonoff). It's not usually possible to compare simplicity precisely, but we do it intuitively. And for each facet of simplicity Swinburne mentions, it should be obvious in some artificial cases where all else is equal, the simpler theory (with respect to that facet) is to be preferred (which, according to Swinburne, is because it's more likely to be true - he defends this claim in the section that immediately follows on pages 99-100).

The six facets of simplicity that he identifies are (the sentences in quotes are Swinburne's):

1. A theory that postulates fewer things is simpler, all else being equal.
2. A theory that postulates fewer kinds of things is simpler, all else being equal.
3. "A formulation of a hypothesis that contains a predicate (descriptive of a property) whose sense can be grasped only by someone who understands some other term (when the latter can be understood without understanding the former) will be less simple that an otherwise equally simple formulation of a hypothesis that contains the latter term instead."
4. A theory with fewer independent laws is simpler. "Kepler's three laws of planetary motion (plus a proposition for each planet, stating its mean distance from the sun) enabling deduction of the paths and periods of all the planets relative to earth was (in this respect) simpler than Copernicus' or Ptolemy's forty or so laws, which also enabled these to be deduced."
5. "A formulation of a theory in which individual laws relate few variables rather than many is simpler"
6. "A mathematically simpler formulation is simpler"; there are two sub-facets involved in mathematical simplicity: i) fewer terms in an equation, and ii) "other things being equal, an equation or description of some state of affairs is mathematically simpler than another in so far as it uses simpler mathematical entities or relations than the other. A mathematical entity or relation or relation Q is simpler than another one Y if Q can be understood by someone who does not understand Y, but Y cannot be understood by anyone who does not understand Q." His definition has the consequence that "multiplication is a less simple relation that addition, power than multiplication, vector product than scalar product; rational numbers are less simple entities than integers, real numbers than rational numbers, tensors than vectors, and so on."

It is the first facet of simplicity (postulating fewer entities) that Swinburne explicitly identifies with Occam's razor, though I've seen some people identify it with both the first and second facet (and others seem to identify Occam's razor with the general concept of simplicity itself).

Answer 2

Without any background information of a specific situation, generally Occam's Razor is not considered a priori first principle but aesthetic and heuristic according to reference here:

The probabilistic (Bayesian) basis for Occam's razor is elaborated by David J. C. MacKay in chapter 28 of his book Information Theory, Inference, and Learning Algorithms, where he emphasizes that a prior bias in favor of simpler models is not required. William H. Jefferys and James O. Berger (1991) generalize and quantify the original formulation's "assumptions" concept as the degree to which a proposition is unnecessarily accommodating to possible observable data. They state, "A hypothesis with fewer adjustable parameters will automatically have an enhanced posterior probability, due to the fact that the predictions it makes are sharp." The use of "sharp" here is not only a tongue-in-cheek reference to the idea of a razor, but also indicates that such predictions are more accurate than competing predictions. The model they propose balances the precision of a theory's predictions against their sharpness, preferring theories that sharply make correct predictions over theories that accommodate a wide range of other possible results. This, again, reflects the mathematical relationship between key concepts in Bayesian inference (namely marginal probability, conditional probability, and posterior probability).

Ludwig Wittgenstein From the Tractatus Logico-Philosophicus: ... "Occam's Razor is, of course, not an arbitrary rule nor one justified by its practical success. It simply says that unnecessary elements in a symbolism mean nothing. Signs which serve one purpose are logically equivalent; signs which serve no purpose are logically meaningless."... "The procedure of induction consists in accepting as true the simplest law that can be reconciled with our experiences."

So clearly there's disagreement about a priori nature of Occam's Razor in application, so it acts more like a heuristic. However, when in some special situations where we have no logical method for settling on one hypothesis amongst an infinite number of equally data-compliant hypotheses, some philosopher like Richard Swinburne argues for its synthetic a priori Kantian nature same as space and time:

According to Swinburne, since our choice of theory cannot be determined by data (see Underdetermination and Duhem–Quine thesis), we must rely on some criterion to determine which theory to use. Since it is absurd to have no logical method for settling on one hypothesis amongst an infinite number of equally data-compliant hypotheses, we should choose the simplest theory: "Either science is irrational [in the way it judges theories and predictions probable] or the principle of simplicity is a fundamental synthetic a priori truth."

So in your described alien case, clearly you can base your chosen hypothesis on Occam's Razor in a priori manner according to Swinburne. But generally speaking it's not a settled universally accepted epistemic conclusion yet.

Answer 3

Occam's razor occurs in the context of science. Very often, the focus is on novel phenomena, so with no priors available. The generalisation, we call: physics. In hypothesis generation to account for novel phenomena, we find this quality that elaborate, contrived explanations tend not to be good.

You can't apply Bayesian statistics to an individual one-off case. But you can, to hypothesis generation in general. I still wouldn't count Occam's razor as a prior, it's not concretely expressible enough. It's more of a heuristic.

Answer 4

(After MacKay, 1992, Neural Comput. 4(3):415-447, doi:10.1162/neco.1992.4.3.415)

No. The Bayesian version of Occam's razor doesn't arise from the prior, it arises from the likelihood, i.e. from the goodness of fit to the empirical data.

The Bayesian Occam's razor tends to disfavour hypotheses not for their complexity per se, nor for merely having lots of adjustable parameters. Instead, it tends to disfavour hypotheses for needing to tune the values of their adjustable parameters finely in order to achieve a good fit to the data.

This works because Bayesian model comparison scores each hypothesis not by the best goodness of fit it can achieve to the data with special values of its adjustable parameters (the "best fit likelihood", "maximum likelihood", or "conditional likelihood"), but by the average goodness of fit it achieves to the data, across all possible values of its adjustable parameters (the "marginal likelihood" or "evidence"). (MacKay coined the name "Occam factor" for the quotient of the marginal likelihood by the best fit likelihood.)

In the case of your toy example, the hypothesis that starts "aliens teleported into my kitchen..." is fairly clearly not an isolated hypothesis that can stand on its own two feet. Instead, it's a special case of the hypothesis that starts "aliens teleported into a room in a house on Earth...". That hypothesis has an adjustable parameter identifying which room in which house. If the value of the adjustable parameter is "@J.Galt's kitchen", then the hypothesis is a very good fit to the data; if the value of the adjustable parameter is "some other room in @J.Galt's house", then the hypothesis is an OK fit to the data; and if the value of the adjustable parameter is "a room in some other house", then the hypothesis is a terrible fit to the data. Since almost all the possible rooms are in houses other than yours, the average goodness of fit of the hypothesis to the data, across all possible values of the adjustable parameter, is terrible.

Answer 5

Depending on the situation, Occam's razor can be helpful to shave the superfluous or do damage and shave off too much or even cause bleeding.

In your case of the steak, it will be probably the aliens who have taken steak because you know that your dog doesn't steal. But considering that aliens are very unlikely to have invaded yet, you might think it reasonable to think that you have come to know a new thing about your dog. In this case, it is helpful to apply Occam's razor (as long as you don't apply it to your dog!). The alien-based reason for the disappearance of the steak seems too far-fetched. It's much simpler to just blame your dog (which is even sitting next to it, wiggling his tail, a strong hint that the guilty one shouldn't be sought in the heavens).
But what if your steak wasn't there anymore and both your cat (assuming you have one) and your dog are in the room. The dog is watching you, his head covered with bloodstains, and your cat is licking herself clean while purring satisfied. In the room of little Henry, it smells steaky.
Now it's not so easy to apply Occam's razor. Well, you could say again that aliens are the cause of the disappearance of the steak, but this can be ruled out by applying the razor again. Now it could be that your dog has used the razor himself, to account for the blood, but this can also be ruled out by applying Occam's razor. you find a number of explanations. Each involves the dog, the cat, and little Henry. Each explanation is based upon the evidence (the bloodstains, the purring, and the smell) and your knowledge of the three involved. You're not sure at all which of these explanations can be true.
So Occam's razor can't be applied yet. It's possible that one of the three is responsible, two of them, or maybe they even played in cahoots. In finding the guilty one(s) it's possible you come to know new things about each one of them. While investigating you come up with scenarios that can be ruled out by the razor. Eventually, you have a number of scenarios. How to choose between them? Can you apply Occam's razor? No. But only one scenario is the real one. So how to know the real one? The dog and the cat can't speak. Little Henry can lie. But if you offer him a sweet... How to know for sure about the dog and the cat? Unless little Henry has seen what happened this will remain quite unsure. Ockham's razor doesn't provide in this case.

So, depending on the situation and explanations given, it can rule out certain explanations, it can't be used at all. situation and explanation will depend on each other though. In a world where aliens are common, the explanation could be reasonable. In the early days of relativity, a big bang would have been shaven of immediately. Which was a pity (the bleeding), as it is seen these days as something that really happened.

Answer 6

Yes, the theory of algorithmic probability describes a "universal prior" that encodes Occam's razor. There exist infinitely many possible hypotheses, and more hypotheses of higher complexity than of lower complexity, so any proper (normalizable) prior must assign asymptotically lower probability to more complex hypotheses.

Algorithmic probability is based on a universal Turing machine, which describes Kolmogorov complexity and represents how hypotheses are used computationally for prediction. This leads to a prior with asymptotic universality and optimality properties.

The catch is that this prior is uncomputable and can only be approximated (e.g., by the minimum description length principle or the similar but explicitly Bayesian minimum message length principle). We may miss a relatively simple hypothesis that explains the data, because a hypothesis may take an unpredictably long time to "run".

Answer 7

Bayesian probability can be viewed as a generalization of Occam's razor.

• Instead of choosing the simplest hypothesis that fits the data, one instead assigns prior probabilities to all hypotheses based on their simplicity, where less simple hypotheses are allowed, but given less weight.
• Instead of giving hypotheses a binary classification of "fitting the observations" or "not fitting the observations", the likelihood of different hypotheses are measured on a continuum.

Finding the simplest hypothesis is, fundamentally, an optimization problem. Let J(m) denote the objective function for this Occam optimization problem. Then, under reasonable assumptions and some math, one finds that the Bayesian posterior is proportional to the function exp(-J(m)). This is discussed in books on inverse problem theory.