Occam's razor states that, everything else being equal, the theory with the least number of assumptions is more likely to be true. This has been formalized as Solomonoff's theory of inductive inference.

Heuristically it seems reasonable that given two theories, A and B, if A is more parsimonious than B, then A is harder to falsify: the theory that is more likely to be true is less likely to be falsified.

But I do not feel comfortable with just a simple heuristic argument. I am looking for better justification or an argument as to why it is not the case.

Here are some more thoughts, since I do not want to answer my own question, as I do not feel like I have a sufficient answer.

If I understand Solomonoff induction correctly, given two hypotheses A and B, s.t. A and B both have accurately predicted data so far, the shorter of the two hypotheses will have the higher probability of producing a correct result for the next prediction. In other words, the longer one will have a lower probability of predicting a correct result. This is the same as saying that it is more likely to be falsified.

Distinct Theories

It seems that part of the problem in answering this question comes down to a disagreement between what constitutes a unique theory. I would call y = 2x and y = 3x distinct theories. Yes; they fall into the same class, but they are not the same theory. The same is true with y ∝ x. y = 2x and y = 3x are similar theories: it's that theory with an additional assumption: the specific constant proportionality is either 2 or 3.

There are some good reasons why simply varying the parameter should be considered a different theory. For one thing, suppose we have a general power series as our theory. By varying different parameters, we can get any single function which is complex differentiable on an open set. That would include y = e^x, y = 8sin(x), y = 3x^2+2x+1, etc. But clearly those are modeling fairly different phenomena.

  • Would this be better suited for math.stackexchange.com? – Daniel Goldman Nov 8 '17 at 20:14
  • 1
    You are oversimplifying Solomonoff's induction. It does not preclude longer hypotheses from having higher probability, the argument is only that the probabilities must converge to 0 with growing length to add up to 1. And in any case, scientists look for highly unlikely hypotheses that are nonetheless true, not more likely ones. This is the main criticism of Bayesianism by falsificationists. – Conifold Nov 9 '17 at 23:09
  • I thought shorter hypotheses are given higher prior probabilities. – Daniel Goldman Nov 10 '17 at 1:58
  • My feeling is that it will depend on the type of theory. I'd agree with Quentin;s answer generally but don't think it's quite this simple. It seems possible that the ease with which a theory may be falsified is more to do with its ramifications and predictions than the number of its necessary hypotheses. . – PeterJ Nov 10 '17 at 17:00
  • As an aside, in the context of the formalism you are considering, I don't think y ∝ x resembles a theory. Instead, you have a family of theories "y ∝ x and we assume y=k when x=1". That is, one kind of theory is a program with three parts: (1) a model with parameters, (2) the algorithm for deciding the parameters from given data, and (3) the sequence of data we feed into the algorithm. – Hurkyl Nov 14 '17 at 8:50

Following the general understanding of what a theory is in philosophy of science, this is quite the contrary actually: a simpler theory is more easy to falsify, and this is precisely why, if confirmed, it is more likely to be true (if one think parcimony is an indicator of truth, which is controversial).

Indeed as Popper argues, a complicated theory with many parameters is easy to adjust to fit any data set. For example, you could approximate many more curves or data points with a polynomial of high degree than with a simple line. This could mean that if both fit, the polynomial of high degree is less confirmed by evidence (because, intuitively speaking, its fit was too easy to obtain) than the line. But this is merely an intuitive argument.

Now Solomonoff is not so much concerned with actual scientific theories than with predictive algorithms (that produce sequences of data that match or not a given sequence). This is very unrealistic from the perspective of philosophy of science, but since your question concerns this particular framework, I would say you're mostly right that a simpler algorithm is less likely to be falsified by subsequent data. But here "simpler"means shorter algorithm, not theories that posit less entities (as"parsimony"is generally understood). I suspect the reason for this result is that Solomonoff starts from the assumption that actual data are produced by an algorithm, and a lot of algorithms will produce the same subsequent data than a simple one, whereas complex algorithms will produce "unique"data sets that are less likely to occur. This is a very specific way of weighting data probabilities (in terms of how many algorithms would produce them) which seem to me not metaphysically neutral and unrealistic with regards to what actual scientists are doing.

Regarding the second aspect of your question: I would say all these are different theories indee although perhaps they could count as scientific theories as generally understood (if x and y are qualified, or at least they could count as observation laws), they don't count as theories for Solomonoff because they do not produce any sequence of data.

  • But again, if you adjust a theory, it's not the same theory. If you need to adjust a theory to remove a contradiction between the theory and the existing data, it's not the same theory. It's a similar theory. – Daniel Goldman Nov 10 '17 at 17:15
  • @DanielGoldman a theory generally has several models. Take Newtonian mechanics for example: when applied to the solar system, you can add as many "hidden"planets as you wish (as scientists did when they postulated the existence of Vulcan to account for the weird trajectory of mercury) but it's still the same theory. – Quentin Ruyant Nov 10 '17 at 18:31
  • I think we are disagreeing on what constitutes distinct theories. Is y = 2x and y ∝ x the same theory? – Daniel Goldman Nov 10 '17 at 19:53
  • Uh they're equations... :-) you mean if x and y describe physical quantities? The second equation is not very specific. Would you accept to reformulate it as y=kx where k is another physical quantity? Then they're different theories. – Quentin Ruyant Nov 10 '17 at 20:29
  • What do you mean "physical quantity?" We have some variable x, and other variable y, and we have some data and we have two theories for the data: one is that y is twice x specifically and the other is that y is proportional to x. – Daniel Goldman Nov 10 '17 at 21:35

Things that are true would be impossible to falsify. The exact collection of all truths is huge and not parsimonious, but it would be impossible to falsify -- being by definition a collection of truths. So there is no absolute logic here. And statistically, the shorter statement is more likely to be ambiguous and to fail to make a prediction, so to that degree it is less likely to be found false. No statement makes all predictions. So it is not reliably true statistically either.

But Popper's standard of falsifiability is not about truth or falseness, it is about the specificity of the target event that would lead to a loss of belief if it were true.

So things that are parsimonious have greater falsifiability. The parts of a theory that are redundant or overdetermined allow for other parts to be omitted or weakened in a way that would leave the theory standing. That means that if my theory is not parsimonious, I could correctly look at the theory, interpret its contents, target a premise, an find it false, without leading to falsifying the theory, because the theory has multiply covered the causal uses of the premise I chose. I would need a larger number of properly well-targeted events to result in the holder discarding the theory.

From this point of view the parsimonious theory is "easier to falsify", in the positive sense in which that word is most often used. The faults it has are more easily seen, so it will lead to faster improvement in interpretation of the same data.

  • Comments are not for extended discussion; this conversation has been moved to chat. – Keelan Nov 13 '17 at 18:51

Occam's razor states that, everything else being equal, the theory with the least number of assumptions is more likely to be true

Whatever Solomonoff's theory of inductive inference is proving, it's not that.

Firstly, if all other things are equal i.e. both theories have identical pre- and post- diction power then it's impossible to distinguish the level of truth between them. Regardless of how convoluted they are. At best, one may argue that the simpler one is more elegant but that is orthogonal to truth.

Secondly, if we don't really mean all then "because fairies" has precisely one assumption so is hard to beat in the "least assumptions" stakes.

More boldly, Occam's Razor can tell you little about the truth of a theory when comparing to a theory of equivalent power.

Where Occam's Razor has proved valuable is in the utility of a theory. In other words, a theory that fits the facts with less assumptions is more likely, in practice, to afford useful predictions. This may be because of overfitting or overconstraining of the convoluted theory. Or it may be because the extraneous features are incorrect but the current facts aren't sensitive to them.

Or to put another way, Occam's Razor can be a useful predictor of relative theoretical predictive power between theories that have equivalent postdictive power.

But that's not very catchy.

  • When you say "whatever Solomonoff's theory of inductive inference is proving, it's not that" are you familiar with the theory and what it proves? Additionally, if we simply go by Occam's razor, if a theory is more likely to be true, is it not less likely to be falsified? – Daniel Goldman Nov 10 '17 at 13:57
  • @DanielGoldman That's just a turn of phrase. The point is that it can't be proving that as nothing could, even in principle, prove that as it's demonstrably false. On your second point, it assumes a monotonic relationship between truth and falsifiability but that's reasonable. Occam's razor though is not monotonic. Hence the fairies. – Alex Nov 10 '17 at 14:25
  • That's not what I am asking. Occam's razor states that, all else being equal, the theory with the least number of assumptions is more likely to be true. I am asking which theory is easier to falsify, all else being equal. – Daniel Goldman Nov 10 '17 at 14:34

For the title question, the underlying justification is Bayes' theorem.

Suppose for every "theory" S, we assign a weight W(S) which we interpret as being proportional to the probability that S is true.

Now, for an observation T, we want to re-evaluate our theories taking the new observation into account. Let W(S | T) denote the new weight we assign to S.

Bayes' theorem tells us exactly how to do this:

W(S | T) = P(T | S) W(S)

where P(T | S) is the probability that theory S assigns to the given observation T.

Roughly speaking, this means any time a theory that makes a specific prediction consistent with observation, it is rewarded by retaining nearly the same weight. However any theory that admits many possible observations (e.g. because we have room to adjust parameters) is penalized. And, of course, any theory that makes a specific, wrong prediction is extremely harshly penalized.

Now onto Solomonoff's formalization of things, the "theories" are all possible algorithms for producing a sequence of values. It's important to note that algorithms often come in two parts:

  • A program that, when given input data, produces results
  • A set of input data

I think it's even safe to assume that this structure is enforced upon all theories.

Anyways, I believe section 3.1 of Solomonoff's paper can be summarized as assigning probabilities to to future outputs from a sequence of data using the Bayesian inference procedure I've described above.

So, your question is about the prior weights Solomonoff chooses to assign; i.e. what are the values W(S)?

The values he wants to use (there is a technical issue he regularizes away) are generated as follows:

  • The lengths of theories are uniformly distributed
  • For a fixed length, the theories of that length are uniformly distributed

Assuming we write things in binary, you can see from this that any particular theory of length n should be twice as likely as any particular theory of length n+1.

Why should one be indifferent about the length of the theory? I don't have an argument for that other than "it tends to give useful results". And one of Solomonoff's points is that "it tends to give useful results" is more important than whatever heuristic reasoning went into coming up with the idea.

  • Yes; this basically summarizes Solomonoff induction, which assigns a higher prior probability shorter "theories." But I'm not seeing an answer to the actual question. We agree that shorter ("simpler") theories are assigned higher prior probabilities, but is either in general "easier" to falsify? – Daniel Goldman Nov 14 '17 at 11:19

If one of two comparative hypotheses is long then reality must be questioning or hypothesis is insufficient. The theory is not about general primitive reasoning. Shortest hypothesis is one proposition. Next proposition can maybe verify all false propositions. The accuracy of a hypothesis is 50%.

  • Yeah; I don't really see how this answers the question. What does fraud have to do with this? – Daniel Goldman Nov 8 '17 at 18:21
  • I think you know what you are getting at, but this answer is not well expressed. I agree with Daniel Goldman that the reference to fraud is unclear. I encourage you to revise the answer and post again. – Mark Andrews Nov 23 '17 at 2:47

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.