# How to justify Occam's Razor?

I'm aware of a few justifications for Occam's (or Ockham) Razor, as it's usually understood that extra factors/complexities should not be added unnecessarily. The only truly compelling justification I have seen appeals to probability theory:

Probabilistic/Mathematical Justification: The probability that A and B are both true can only be equal two or less than the probability that only A is true. If we give both P(A) and P(B) prior probabilities of 0.5, then it's true that P(A) > P(A^B)

However, it seems to me that this principle is sometimes used in a slightly different way, where the author assumes that Occam's Razor is somehow self-justifying or intuitively true. Is there any way to explain Occam's Razor, or perhaps phrase it differently, that makes it axiomatic? If I say that the burden of proof should be borne on the side that has to prove more things - is that necessarily true?

• I apologize if I've run afoul of this issue currently being discussed on the meta: meta.philosophy.stackexchange.com/questions/1686/… Commented Jul 1, 2014 at 23:57
• Could you explain what you hope a justification could look like and/or give us an expression of what you found inadequate in a common defense of it from the literature? I can give you an expression for Occam's razor: "stop wasting my time with pointless things" that is roughly speaking axiomatic. Commented Jul 2, 2014 at 8:33
• @virmaior See Wikipedia's article on Occam's Razor Commented Jul 2, 2014 at 11:36
• see comment above. I'm not asking what Occam's razor is or anything of the sort. I'm asking you to indicate what you think the defense is and then indicate why you find it unsatisfying rather than requiring me to look inside your mind and figure out what you mean in your rejection of it. Commented Jul 2, 2014 at 11:41
• Ockam's razor is not axiomatic, it is heuristic. Commented Jul 2, 2014 at 18:58

The intuitive version (that seems justified to me) is a statement of pragmatism, not truth: if `A` and `B` explain things equally well, and `A` is simpler, why would I bother with the extra headache of `B`?

I think there's a truthier version that is entangled with Kolmogorov complexity and dynamic semantics in deep ways. I've never seen anything approaching a proof of this, but the intuition is that although it is easy to write a true statement, writing a true statement that conveys a lot of information (in the Kolmogorov sense) is very difficult. Almost everything you try to say will either have very little content ("My name is not Joe") or will be wrong ("My name is Fred") or will not take advantage of context and thus be impossibly bulky ("My name is Matt, where by 'name' I mean that verbal utterance and corresponding written string of symbols by which I am commonly referred, in contrast to the legal name on my birth certificate...").

Occam's Razor is then a statement of three things: first, how special it is to find a compact description of anything; second, that we have to a large extent organized language to match causally-separable or independent processes; and third, that we observe that very often there is a single proximal causal process rather than an indecipherable muddle that gives rise to recognizable patterns. With these three together, you have reasonable hope that if you find one of these rare compact but effective descriptions, you're really onto something.

Even so it's just a rule of thumb, but I think it's a deeply and subtly true rule of thumb.

• The difference between the pragmatic/methodological razor and the metaphysical razor is huge. In an infinitely complex universe, the methodological razor could still be valid. It tells us nothing about true ontological complexity. Commented Jul 2, 2014 at 12:48

I like Popper's interpretation. Simplicity is not based on language or aesthetics; it is based on falsifiability.

Regarding your question "the burden of proof should be borne on the side that has to prove more things" I say, yes. Suppose you gather data below,

``````sample 1 is 1 1 0
sample 2 is 2 3 2
sample 3 is 5 1 10
``````

and you make a theory "the third value is two times the difference of the first two values, unless the first value greater than 3, in which case the third value is 10".

Although the theory is falsifiable, I would be skeptical of it, because the complexity of the theory rivals the data itself. Similar criticisms are leveled against string theory. A theory with more complexity has a greater burden of data.

• Per your answer, do you mean that "simplicity" is synonymous with "generality"? Commented Jul 10, 2014 at 7:18
• @Pacerier - I'm no expert, but it seems to me that specificity, generality, and complexity are independent properties. Some simple theories are general, and some are not. I think the OP is asking whether, if we ignore the first two, there is a relationship between the complexity of the theory and the amount of data required to reasonably support it. And I think yes. See also Dave's link to MacKay. Commented Jul 10, 2014 at 19:16

The Razor is based on a more fundamental principle that we should give verifiable evidence wherever possible for any claim, from which it follows that we should prefer to limit the number of unverifiable claims, or unobservable entities, wherever possible. The Razor follows directly from this. For example, suppose that we have an observable phenomenon X. Theory 1 postulates unobservable entity A as the cause of X. Thus only one unobservable entity is required for theory 1.

Theory 2 postulates unobservable entities B and C as the cause of X. Thus two unobservable entities are required for theory 2, rather than just one for theory 1. On the assumption that we want to limit the stuff for which we have no direct evidence (except as postulated explanations for X), we should go for theory 1.

On how to justify the principle that we should prefer verifiable or observable phenomena, and should rely as little as possible on explanations that involve unobservable things, I don't know, but it's a separate question.

• Does fewer unobservable entities mean higher probability the theory to be right ? Commented Jul 13, 2019 at 14:57

The idea embodied in Occam's razor is present in Bayesian descriptions of belief; and this formulation provides a rigorous mathematical formulation of the idea.

Qualitatively, more complex theories have their (prior) probability density spread out over a larger (higher dimensional even) volume, which ends up affecting the inferred probability of the hypothesis in a negative way. Thus a complex hypothesis needs more "lift" (likelihood gain) from the observed data in order to surpass a given threshold in probability.

• This is what I was trying to say, but you said it much better! Commented Jul 3, 2014 at 19:55
• If you are dealing with deterministic phenomenon, why does probability even come in to it? Commented Nov 17, 2018 at 6:02
• – Dave
Commented Nov 18, 2018 at 2:58

Your question has two flaws. (1) Justification is impossible and not desirable. (2) Occam's razor is a badly flawed standard.

Justification, showing an idea is true or probably true, is impossible. If you assess ideas using argument then the arguments have premises and rules of inference and the result of the argument may not be true (or probably true) if the premises and rules of inference are false. You might try to solve this by coming up with a new argument that proves the premises and rules of inference but then you have the same problem with those premises and rules of inference. You might say that some stuff is indubitably true (or probably true), and you can use that as a foundation. But that just means you have cut off a possible avenue of intellectual progress since the foundation can't be explained in terms of anything deeper. And in any case there is nothing that can fill that role. Sense experience won't work since you can misinterpret information from your sense organs, e.g. - optical illusions. Sense organs also fail to record lots of stuff that does exist, e.g. - neutrinos. Scientific instruments aren't infallible either since you can make mistakes in setting them up, in interpreting information from them and so on.

We don't create knowledge (useful or explanatory information) by showing stuff is true or probably true for reasons so how do we create knowledge? We can only create knowledge by finding mistakes in our current ideas and correcting them piecemeal. You notice a problem with your current ideas, propose solutions, criticise the solutions until only one is left and then find a new problem.

Occam's razor is problematic because it claims that you should not multiply entities beyond necessity without specifying what counts as necessity. As a result it has often been given bad interpretations like don't add stuff that isn't justified. It is sometimes true that a better explanation has been created by discarding an idea that people formerly thought was necessary, e.g. - Einstein discarding the ether in special relativity. Any idea should be judged by whether it solves problems that other ideas don't solve, regardless of the number of entities it invokes. For example, it is not a good objection to the atomic theory to claim that it assumes the existence of large numbers of atoms: that's a lot of assumptions ~10 the power 23 for any given object. You could say that's just one assumption but why count it as only one assumption? This is the sort of issue that comes up if you see the number of assumptions as the important issue.

• I think you might be misreading Occam's razor in your dismissal of it here. Yes, it does not make perfectly clear what the necessity is. You might be saying that's its problem, but that's also its strength = eliminate any entity unnecessary to an explanation from your explanation both means you may end up adding entities that were removed when not understood and deleting entities that seem to add nothing to the explanation. Commented Jul 2, 2014 at 11:40

Events (A) and their consequences (B) necessarily require a chain of causation by which the former leads to the latter. This means that the more complex the former, the more complex the unknown chain of causation which you are requiring to be in place outside of your knowledge by which A leads to B.

Any explanation in which A is more complex requires that there be a greater and more complex chain of causation of which you are unaware, which you have failed to see, and therefore it is more lacking. Since you have seen none of it, your best estimator is the minimal estimate.

For example, if you look at a landscape and look away, and I tell you that you didn't see some number of trees, you can estimate there was one tree which you missed or that there was a billion rainforests. Since you failed to see any trees, in the absence of any evidence to the contrary, the smaller number of trees is the best estimator.