How to justify Occam's Razor?

Question

I'm aware of a few justifications for Occam's (or Ockham) Razor, as it's usually understood that extra factors/complexities should not be added unnecessarily. The only truly compelling justification I have seen appeals to probability theory:

Probabilistic/Mathematical Justification: The probability that A and B are both true can only be equal two or less than the probability that only A is true. If we give both P(A) and P(B) prior probabilities of 0.5, then it's true that P(A) > P(A^B)

However, it seems to me that this principle is sometimes used in a slightly different way, where the author assumes that Occam's Razor is somehow self-justifying or intuitively true. Is there any way to explain Occam's Razor, or perhaps phrase it differently, that makes it axiomatic? If I say that the burden of proof should be borne on the side that has to prove more things - is that necessarily true?

Answer 1

The intuitive version (that seems justified to me) is a statement of pragmatism, not truth: if A and B explain things equally well, and A is simpler, why would I bother with the extra headache of B?

I think there's a truthier version that is entangled with Kolmogorov complexity and dynamic semantics in deep ways. I've never seen anything approaching a proof of this, but the intuition is that although it is easy to write a true statement, writing a true statement that conveys a lot of information (in the Kolmogorov sense) is very difficult. Almost everything you try to say will either have very little content ("My name is not Joe") or will be wrong ("My name is Fred") or will not take advantage of context and thus be impossibly bulky ("My name is Matt, where by 'name' I mean that verbal utterance and corresponding written string of symbols by which I am commonly referred, in contrast to the legal name on my birth certificate...").

Occam's Razor is then a statement of three things: first, how special it is to find a compact description of anything; second, that we have to a large extent organized language to match causally-separable or independent processes; and third, that we observe that very often there is a single proximal causal process rather than an indecipherable muddle that gives rise to recognizable patterns. With these three together, you have reasonable hope that if you find one of these rare compact but effective descriptions, you're really onto something.

Even so it's just a rule of thumb, but I think it's a deeply and subtly true rule of thumb.

Answer 2

I like Popper's interpretation. Simplicity is not based on language or aesthetics; it is based on falsifiability.

Regarding your question "the burden of proof should be borne on the side that has to prove more things" I say, yes. Suppose you gather data below,

sample 1 is 1 1 0
sample 2 is 2 3 2
sample 3 is 5 1 10

and you make a theory "the third value is two times the difference of the first two values, unless the first value greater than 3, in which case the third value is 10".

Although the theory is falsifiable, I would be skeptical of it, because the complexity of the theory rivals the data itself. Similar criticisms are leveled against string theory. A theory with more complexity has a greater burden of data.

Answer 3

The Razor is based on a more fundamental principle that we should give verifiable evidence wherever possible for any claim, from which it follows that we should prefer to limit the number of unverifiable claims, or unobservable entities, wherever possible. The Razor follows directly from this. For example, suppose that we have an observable phenomenon X. Theory 1 postulates unobservable entity A as the cause of X. Thus only one unobservable entity is required for theory 1.

Theory 2 postulates unobservable entities B and C as the cause of X. Thus two unobservable entities are required for theory 2, rather than just one for theory 1. On the assumption that we want to limit the stuff for which we have no direct evidence (except as postulated explanations for X), we should go for theory 1.

On how to justify the principle that we should prefer verifiable or observable phenomena, and should rely as little as possible on explanations that involve unobservable things, I don't know, but it's a separate question.

Answer 4

The idea embodied in Occam's razor is present in Bayesian descriptions of belief; and this formulation provides a rigorous mathematical formulation of the idea.

Qualitatively, more complex theories have their (prior) probability density spread out over a larger (higher dimensional even) volume, which ends up affecting the inferred probability of the hypothesis in a negative way. Thus a complex hypothesis needs more "lift" (likelihood gain) from the observed data in order to surpass a given threshold in probability.

c.f. this chapter by D. MacKay

Answer 5

Events (A) and their consequences (B) necessarily require a chain of causation by which the former leads to the latter. This means that the more complex the former, the more complex the unknown chain of causation which you are requiring to be in place outside of your knowledge by which A leads to B.

Any explanation in which A is more complex requires that there be a greater and more complex chain of causation of which you are unaware, which you have failed to see, and therefore it is more lacking. Since you have seen none of it, your best estimator is the minimal estimate.

For example, if you look at a landscape and look away, and I tell you that you didn't see some number of trees, you can estimate there was one tree which you missed or that there was a billion rainforests. Since you failed to see any trees, in the absence of any evidence to the contrary, the smaller number of trees is the best estimator.

Answer 6

Your question has two flaws. (1) Justification is impossible and not desirable. (2) Occam's razor is a badly flawed standard.

Justification, showing an idea is true or probably true, is impossible. If you assess ideas using argument then the arguments have premises and rules of inference and the result of the argument may not be true (or probably true) if the premises and rules of inference are false. You might try to solve this by coming up with a new argument that proves the premises and rules of inference but then you have the same problem with those premises and rules of inference. You might say that some stuff is indubitably true (or probably true), and you can use that as a foundation. But that just means you have cut off a possible avenue of intellectual progress since the foundation can't be explained in terms of anything deeper. And in any case there is nothing that can fill that role. Sense experience won't work since you can misinterpret information from your sense organs, e.g. - optical illusions. Sense organs also fail to record lots of stuff that does exist, e.g. - neutrinos. Scientific instruments aren't infallible either since you can make mistakes in setting them up, in interpreting information from them and so on.

We don't create knowledge (useful or explanatory information) by showing stuff is true or probably true for reasons so how do we create knowledge? We can only create knowledge by finding mistakes in our current ideas and correcting them piecemeal. You notice a problem with your current ideas, propose solutions, criticise the solutions until only one is left and then find a new problem.

Occam's razor is problematic because it claims that you should not multiply entities beyond necessity without specifying what counts as necessity. As a result it has often been given bad interpretations like don't add stuff that isn't justified. It is sometimes true that a better explanation has been created by discarding an idea that people formerly thought was necessary, e.g. - Einstein discarding the ether in special relativity. Any idea should be judged by whether it solves problems that other ideas don't solve, regardless of the number of entities it invokes. For example, it is not a good objection to the atomic theory to claim that it assumes the existence of large numbers of atoms: that's a lot of assumptions ~10 the power 23 for any given object. You could say that's just one assumption but why count it as only one assumption? This is the sort of issue that comes up if you see the number of assumptions as the important issue.