6

Occam's Razor states, essentially, that among hypotheses with similar explanatory power, the simplest hypothesis should be favored as the most plausible.

Thermodynamics states, at zero temperature, that a reaction will proceed if the products have lower energy than the reactants, ΔH < 0. The analogy is that the favorable reaction is like a favorable hypothesis: lower energy is akin to simpler (and/or better-fitting).

But at non-zero temperature, it's not so simple: random thermal fluctuations will tend to knock species into different states; if there's more ability to "be random", that will tend to be favored. This ability is quantified as entropy, and favorable reactions now must obey ΔG = ΔH - TΔS < 0, where T is absolute temperature and ΔS is the change in entropy (G is Gibbs Free Energy). (To increase sophistication one step more, one should consider the Boltzmann distribution of states, but I'll stop with entropy.)

There seems to be an analogue in hypothesis space. Suppose there are many not-exactly-simplest hypotheses with approximately the same explanatory power as the simplest hypothesis. (For example, you could add more variables than are needed to explain your data.) Depending on what mechanism might encourage simplicity (i.e. what the "temperature" is), you might be a lot more likely to find the true hypothesis in the pool of not-exactly-simplest hypotheses than to actually find that the simplest hypothesis is true.

Is there any literature, perhaps in philosophy of science, that expands upon this point? It is of particular relevance in fields like biology where the simplest hypothesis is often not true--and when you consider evolution as temperature (specifically, a component of it involves randomly changing stuff), it vaguely suggests that a Gibbs-Free-Energy-like modification to Occam's Razor is advisable.

  • 3
    Don't you need two theories with equally strong explanations of the same thing in order for Occam's Razor to apply? – Lukas Apr 14 '15 at 21:29
  • @Lukas - Yes, I'm assuming that all theories have approximately the same explanatory power. I'll add that to the question. – Rex Kerr Apr 14 '15 at 21:47
  • 2
    "It is of particular relevance in fields like biology where the simplest hypothesis is often not true" - As you point out yourself, the concern isn't simplicity, but maintaining a tradeoff between simplicity and explanatory power. – James Kingsbery Apr 14 '15 at 22:47
  • 1
    I don't know of a particular philosophical reference, but in statistical models there is often a regularization term that penalizes complicated models, which forces any evaluation function to trade-off between simplicity and explanatory power. – James Kingsbery Apr 14 '15 at 22:56
  • Another interesting issue on the way to create a entropy-related philosophical issue +1 – Kentaro Tomono Apr 14 '15 at 23:16
3

In terms of Bayesian hypothesis testing, Occam's razor is incorporated in the prior probabilities of the hypotheses. Often one can interpret differences in the prior probabilities as being "entropic". E.g. imagine two models: H1: 0<=x<0.2, H2: 0.2<x<1.0, based on some parameter 0<=x<1. If we take the maximum entropy (constant) distribution as the prior distribution for x, then the a priori bias towards H2 is due to the fact that there are "more states", i.e. higher entropy, for that model. This straightforward use of Bayesian hypothesis testing does not have the feature of balancing energetic against entropic considerations; or equivalently there isn't the analog of (variable) temperature.

There is research on the statistical mechanics of Bayeisan networks, e.g. variational approaches for estimating the free energy in physical systems can be applied as approximate propagation algorithms (Yeddida is a relevant author). In the end, the thermodynamics is all done at kT=1, so that energy <=> log-likelihood.

When you get to machine learning problems, you can start to see things that look like temperatures in the models; and thus you get more interesting analogies. I've seen papers (don't have reference handy) where during the initial phases of learning Bayesian networks, they used (effectively) a high temperature to prevent over fitting the conditional relationships while the structure was still uncertain, and then lowered the "temperature" until kT=1, and they were finding the maximum likelihood model for the data.

There is what I consider a related application in reinforcement learning where a temperature-like parameter can be used to go between exploration (entropy driven) and exploitation (energy driven).

In terms of the kind of hypothesis testing alluded to in the OP, I haven't seen research that has real thermodynamics (absence of evidence and all...), and I don't see what feature of those kinds of problems map onto the idea of a temperature. However, in various places in machine learning I have seen what look like relevant ideas; this is due to the fact that that these types of problems need to balance model fit (energy analog) against generalization (entropy analog). None of them exactly match the criteria set in the question, but hopefully provide some indications of where related ideas have popped up.

  • Good points! I'm actually familiar with the idea of temperature in ML systems, from explicit (Boltzmann machines) to less-explicit (varying learning rates, etc.). I'll try to check out Yeddida. But I was hoping there was more philosophically-focused discussion of these issues where the epistemology of the matter is thought about more deeply, and where you have ideas of how to apply the take-home message even if you don't have a parameterization of your prior distribution like you do in Bayesian hypothesis testing. – Rex Kerr Apr 15 '15 at 0:57
1

Occam's razor is looking for the "best" answer, and it defines that to be the "most plausible answer." In science, we do not have the luxury of looking for a "true hypothesis" because the definition of a true hypothesis is an ontological one beyond the scope of science (it tends to be more in the domain of religion). Instead, discovery is treated as a slow accumulation of improvements.

In theory, if we opened the gates to consider a wider range of more complex hypothesis, we would be more likely to consider the "true" hypothesis in the first round. However, does that mean we are more likely to act upon the true hypothesis? That's a harder question.

Consider a set of hypothesis which are built from a kernel hypothesis, H. Define a transform (H, n)->H' which creates a new hypothesis by combining the kernel hypothesis with some numerically defined alterations to create a new hypothesis. This could easily take the form of "adding additional degrees of freedom," which you mention.

Now, lets also include a "simplest" hypothesis, S, and let's instantiate a few hypothesis, H1, H2, and H3 (generated from (H, 1), (H, 2), and (H, 3)) respectively. It is clear that we could define as many of these as we want, and "dilute" the effect of any one hypothesis, including S.

In many cases, it is remarkably hard to prove "All hypotheses constructed using (H, n)->H' are less likely to be 'true' than S." This creates a frustrating dilemma: just by counting from 1 on upwards, a single individual could construct an arbitrary number of hypothesis to be tested, one of which might be true.

In an ideal world, where discussing hypothesis is free and takes no time, this is not a concern. One simply discusses the infinite set of hypothesis, draw a conclusion, and move on. However, in the real world, testing a theory is expensive. We do not want to make it easy to cloud the waters.

So science's solution is Occam's razor. Instead of trying to jump to the "true" hypothesis in one jump, science simply tries to jump "closer." Simplicity is something that can be reasonably agreed upon (it's not perfect, but its far easier than getting agreement on "truthfulness"). So science, and Occam's Razor declares a postulate: "Generally speaking, finding a simple and mostly-right hypothesis will lead us to a better solution in the next iteration than we would have gotten from selecting more complex hypothesis." Thus, if the "true" hypothesis, lets choose H3, had an extra variable, science claims that it will arrive at H3 cheaper and faster if it first goes through S, and then extends that to H3. It claims that if it had to jump to H3, the process would be more expensive and slower.

I draw attention to the word "postulate." There is nothing that says "Occam's Razor is the true way to find knowledge." It's just a principle that has been successful enough in scientific inquiry to reach a high level of prestige.

I think you could find literature on the topic in general. The nature of Occam's razor is well defined in terms of optimizing systems. In its traditional usage, Occam's razor is an approach seeking to optimize the value of hypotheses generated against the time spent on them. If you wished to explore alternative optimizations which find the "true" hypothesis faster, you could start by exploring how one would test if a hypothesis is "true," and then see if that leads in directions which call for alternative optimization techniques.

However, I don't think you will see much on the application of Occam's Razor in philosophy in the literature because Occam's razor starts from the premise that that two hypothesis have equal proving power. This is a highly unusual occurrence in philosophy because there is so little agreement on the definition of "proving power." The few places I am familiar with it (phyiscalism vs. dualism vs. idealism being my favorite), we actually do see what you suggest: rather than just starting from one hypothesis and moving forward, philosophy has gone forward with several in parallel.

1

I'm going to suggest two things in answer.

First, Occam's Razor is not well-defined. On one level, it's not at all clear what William of Ockham wrote except that it is some form of principle of parsimony (link). On another level, it's hard to find someone who knowingly offers an explanation with superfluous details. At least, that's my take on it.

Second, my experience from my undergraduate days in chemistry was that chemists always operate under simplifications, and we try to pick the least sufficient simplification (by which I mean the simplest form that is sufficient for solving the problem).

For instance, in many elementary experiments we act as if ΔH = ΔE. Or we consider gases in accordance with PV=nRT rather than the virial equations needed for real gases (http://facstaff.cbu.edu/rprice/lectures/realgas.html). Similarly, we pick systems that are easy to calculate over those that are hard (beginning with using the metric system over imperial units).

But in terms of philosophy of science literature, I don't know anywhere that directly addresses what you're asking (which could be because my AOSes and AOC are not in philosophy of science).

See especially SEP.

  • The insight about Occam's Razor not being well-defined is an important one. The concept is very popular but that doesn't equate to it being precise! (And I agree about choice of simplification in chemistry; it agrees with my undergrad experience also, and with a famous Einstein quote.) – Rex Kerr Apr 18 '15 at 18:13
0

The actual premise seems to be a bit of a dead ringer. Your pool of hypotheses includes the simplest, so being a superset, the probability of finding the best answer in the pool is greater than or equal to the probability of the simplest answer being the best.

As for the applicability of the ideas to philosophy, I've had a little trouble. In theory, it makes sense that Occam's Razor should have some effect here. However, the challenge is that Occam's Razor is that it suggests the "simplest" should be the best. "Simple" is not a simple term in philosophy. People come with different sets of axioms, and the path from them to a theory is not always straightforward. For example, there are many theories which are "simplest" for people who believe in a literal word-for-word translation of the Bible, but which are not "simplest" for an atheist.

I think you could apply a theory along these lines to theories for your own personal use, but I find that the value of Occam's Razor is dwarfed by the complexity of trying to get people to agree on what is simple, so it is harder to apply in a group setting.

  • I guess I wasn't clear enough. Here's a typical example in biology: break A, and stuff happens. Break B, the opposite stuff happens. Break both, and it looks like B. Logic is: A turns off B, B does stuff. Simplest hypothesis. But there are families of hypotheses that look like: A turns on some C, which turns off some D, which turns off some E, which turns off B; B does stuff. Or: A signals to C. B also signals to C. But in the absence of signal, C looks like A-on, B-off. Anyway, the point is that the more-complicated-pool doesn't (by premise) include the simplest answer. – Rex Kerr Apr 15 '15 at 7:23
  • Hmm, makes some sense, but its complicated itself. So I think the reason I wrote my answer the way I did is the wording "the pool" in the second to last paragraph. You might want to edit to expand on that wording. I automatically assumed the pool was generaly-sort-of around the simplest answer. If the pool is intended to exclude the simplest answer rather than include it, then I will write another answer, because in that case, the word "should" in Occam's Razor is the key. Should is a very philosophical word that opens many interpretations beyond the problem set itself. – Cort Ammon Apr 15 '15 at 15:30
  • I've tried to clarify the premise. Do you find that clearer? – Rex Kerr Apr 15 '15 at 19:19
  • Yes, that is much clearer to me. I should be able to come up with another answer post clarification which will hopefully be more in line with what you are looking for. – Cort Ammon Apr 15 '15 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.