Are more parsimonious theories harder to falsify?

Question

Occam's razor states that, everything else being equal, the theory with the least number of assumptions is more likely to be true. This has been formalized as Solomonoff's theory of inductive inference.

Heuristically it seems reasonable that given two theories, A and B, if A is more parsimonious than B, then A is harder to falsify: the theory that is more likely to be true is less likely to be falsified.

But I do not feel comfortable with just a simple heuristic argument. I am looking for better justification or an argument as to why it is not the case.

Here are some more thoughts, since I do not want to answer my own question, as I do not feel like I have a sufficient answer.

If I understand Solomonoff induction correctly, given two hypotheses A and B, s.t. A and B both have accurately predicted data so far, the shorter of the two hypotheses will have the higher probability of producing a correct result for the next prediction. In other words, the longer one will have a lower probability of predicting a correct result. This is the same as saying that it is more likely to be falsified.

Distinct Theories

It seems that part of the problem in answering this question comes down to a disagreement between what constitutes a unique theory. I would call y = 2x and y = 3x distinct theories. Yes; they fall into the same class, but they are not the same theory. The same is true with y ∝ x. y = 2x and y = 3x are similar theories: it's that theory with an additional assumption: the specific constant proportionality is either 2 or 3.

There are some good reasons why simply varying the parameter should be considered a different theory. For one thing, suppose we have a general power series as our theory. By varying different parameters, we can get any single function which is complex differentiable on an open set. That would include y = e^x, y = 8sin(x), y = 3x^2+2x+1, etc. But clearly those are modeling fairly different phenomena.

Answer 1

Following the general understanding of what a theory is in philosophy of science, this is quite the contrary actually: a simpler theory is more easy to falsify, and this is precisely why, if confirmed, it is more likely to be true (if one think parsimony is an indicator of truth, which is controversial).

Indeed as Popper argues, a complicated theory with many parameters is easy to adjust to fit any data set. For example, you could approximate many more curves or data points with a polynomial of high degree than with a simple line. This could mean that if both fit, the polynomial of high degree is less confirmed by evidence (because, intuitively speaking, its fit was too easy to obtain) than the line. But this is merely an intuitive argument.

Now Solomonoff is not so much concerned with actual scientific theories than with predictive algorithms (that produce sequences of data that match or not a given sequence). This is very unrealistic from the perspective of philosophy of science, but since your question concerns this particular framework, I would say you're mostly right that a simpler algorithm is less likely to be falsified by subsequent data. But here "simpler"means shorter algorithm, not theories that posit less entities (as"parsimony"is generally understood). I suspect the reason for this result is that Solomonoff starts from the assumption that actual data are produced by an algorithm, and a lot of algorithms will produce the same subsequent data than a simple one, whereas complex algorithms will produce "unique"data sets that are less likely to occur. This is a very specific way of weighting data probabilities (in terms of how many algorithms would produce them) which seem to me not metaphysically neutral and unrealistic with regards to what actual scientists are doing.

Regarding the second aspect of your question: I would say all these are different theories indeed although perhaps they could count as scientific theories as generally understood (if x and y are qualified, or at least they could count as observation laws), they don't count as theories for Solomonoff because they do not produce any sequence of data.

Answer 2

Occam's razor states that, everything else being equal, the theory with the least number of assumptions is more likely to be true

Whatever Solomonoff's theory of inductive inference is proving, it's not that.

Firstly, if all other things are equal i.e. both theories have identical pre- and post- diction power then it's impossible to distinguish the level of truth between them. Regardless of how convoluted they are. At best, one may argue that the simpler one is more elegant but that is orthogonal to truth.

Secondly, if we don't really mean all then "because fairies" has precisely one assumption so is hard to beat in the "least assumptions" stakes.

More boldly, Occam's Razor can tell you little about the truth of a theory when comparing to a theory of equivalent power.

Where Occam's Razor has proved valuable is in the utility of a theory. In other words, a theory that fits the facts with less assumptions is more likely, in practice, to afford useful predictions. This may be because of overfitting or overconstraining of the convoluted theory. Or it may be because the extraneous features are incorrect but the current facts aren't sensitive to them.

Or to put another way, Occam's Razor can be a useful predictor of relative theoretical predictive power between theories that have equivalent postdictive power.

But that's not very catchy.

Answer 3

For the title question, the underlying justification is Bayes' theorem.

Suppose for every "theory" S, we assign a weight W(S) which we interpret as being proportional to the probability that S is true.

Now, for an observation T, we want to re-evaluate our theories taking the new observation into account. Let W(S | T) denote the new weight we assign to S.

Bayes' theorem tells us exactly how to do this:

W(S | T) = P(T | S) W(S)

where P(T | S) is the probability that theory S assigns to the given observation T.

Roughly speaking, this means any time a theory that makes a specific prediction consistent with observation, it is rewarded by retaining nearly the same weight. However any theory that admits many possible observations (e.g. because we have room to adjust parameters) is penalized. And, of course, any theory that makes a specific, wrong prediction is extremely harshly penalized.

---

Now onto Solomonoff's formalization of things, the "theories" are all possible algorithms for producing a sequence of values. It's important to note that algorithms often come in two parts:

• A program that, when given input data, produces results
• A set of input data

I think it's even safe to assume that this structure is enforced upon all theories.

Anyways, I believe section 3.1 of Solomonoff's paper can be summarized as assigning probabilities to to future outputs from a sequence of data using the Bayesian inference procedure I've described above.

So, your question is about the prior weights Solomonoff chooses to assign; i.e. what are the values W(S)?

The values he wants to use (there is a technical issue he regularizes away) are generated as follows:

• The lengths of theories are uniformly distributed
• For a fixed length, the theories of that length are uniformly distributed

Assuming we write things in binary, you can see from this that any particular theory of length n should be twice as likely as any particular theory of length n+1.

Why should one be indifferent about the length of the theory? I don't have an argument for that other than "it tends to give useful results". And one of Solomonoff's points is that "it tends to give useful results" is more important than whatever heuristic reasoning went into coming up with the idea.

Answer 4

Things that are true would be impossible to falsify. The exact collection of all truths is huge and not parsimonious, but it would be impossible to falsify -- being by definition a collection of truths. So there is no absolute logic here. And statistically, the shorter statement is more likely to be ambiguous and to fail to make a prediction, so to that degree it is less likely to be found false. No statement makes all predictions. So it is not reliably true statistically either.

But Popper's standard of falsifiability is not about truth or falseness, it is about the specificity of the target event that would lead to a loss of belief if it were true.

So things that are parsimonious have greater falsifiability. The parts of a theory that are redundant or overdetermined allow for other parts to be omitted or weakened in a way that would leave the theory standing. That means that if my theory is not parsimonious, I could correctly look at the theory, interpret its contents, target a premise, an find it false, without leading to falsifying the theory, because the theory has multiply covered the causal uses of the premise I chose. I would need a larger number of properly well-targeted events to result in the holder discarding the theory.

From this point of view the parsimonious theory is "easier to falsify", in the positive sense in which that word is most often used. The faults it has are more easily seen, so it will lead to faster improvement in interpretation of the same data.

Answer 5

If one of two comparative hypotheses is long then reality must be questioning or hypothesis is insufficient. The theory is not about general primitive reasoning. Shortest hypothesis is one proposition. Next proposition can maybe verify all false propositions. The accuracy of a hypothesis is 50%.