Thermodynamics of Occam's Razor

Question

Occam's Razor states, essentially, that among hypotheses with similar explanatory power, the simplest hypothesis should be favored as the most plausible.

Thermodynamics states, at zero temperature, that a reaction will proceed if the products have lower energy than the reactants, ΔH < 0. The analogy is that the favorable reaction is like a favorable hypothesis: lower energy is akin to simpler (and/or better-fitting).

But at non-zero temperature, it's not so simple: random thermal fluctuations will tend to knock species into different states; if there's more ability to "be random", that will tend to be favored. This ability is quantified as entropy, and favorable reactions now must obey ΔG = ΔH - TΔS < 0, where T is absolute temperature and ΔS is the change in entropy (G is Gibbs Free Energy). (To increase sophistication one step more, one should consider the Boltzmann distribution of states, but I'll stop with entropy.)

There seems to be an analogue in hypothesis space. Suppose there are many not-exactly-simplest hypotheses with approximately the same explanatory power as the simplest hypothesis. (For example, you could add more variables than are needed to explain your data.) Depending on what mechanism might encourage simplicity (i.e. what the "temperature" is), you might be a lot more likely to find the true hypothesis in the pool of not-exactly-simplest hypotheses than to actually find that the simplest hypothesis is true.

Is there any literature, perhaps in philosophy of science, that expands upon this point? It is of particular relevance in fields like biology where the simplest hypothesis is often not true--and when you consider evolution as temperature (specifically, a component of it involves randomly changing stuff), it vaguely suggests that a Gibbs-Free-Energy-like modification to Occam's Razor is advisable.

Answer 1

In terms of Bayesian hypothesis testing, Occam's razor is incorporated in the prior probabilities of the hypotheses. Often one can interpret differences in the prior probabilities as being "entropic". E.g. imagine two models: H1: 0<=x<0.2, H2: 0.2<x<1.0, based on some parameter 0<=x<1. If we take the maximum entropy (constant) distribution as the prior distribution for x, then the a priori bias towards H2 is due to the fact that there are "more states", i.e. higher entropy, for that model. This straightforward use of Bayesian hypothesis testing does not have the feature of balancing energetic against entropic considerations; or equivalently there isn't the analog of (variable) temperature.

There is research on the statistical mechanics of Bayeisan networks, e.g. variational approaches for estimating the free energy in physical systems can be applied as approximate propagation algorithms (Yeddida is a relevant author). In the end, the thermodynamics is all done at kT=1, so that energy <=> log-likelihood.

When you get to machine learning problems, you can start to see things that look like temperatures in the models; and thus you get more interesting analogies. I've seen papers (don't have reference handy) where during the initial phases of learning Bayesian networks, they used (effectively) a high temperature to prevent over fitting the conditional relationships while the structure was still uncertain, and then lowered the "temperature" until kT=1, and they were finding the maximum likelihood model for the data.

There is what I consider a related application in reinforcement learning where a temperature-like parameter can be used to go between exploration (entropy driven) and exploitation (energy driven).

In terms of the kind of hypothesis testing alluded to in the OP, I haven't seen research that has real thermodynamics (absence of evidence and all...), and I don't see what feature of those kinds of problems map onto the idea of a temperature. However, in various places in machine learning I have seen what look like relevant ideas; this is due to the fact that that these types of problems need to balance model fit (energy analog) against generalization (entropy analog). None of them exactly match the criteria set in the question, but hopefully provide some indications of where related ideas have popped up.

Answer 2

Occam's razor is looking for the "best" answer, and it defines that to be the "most plausible answer." In science, we do not have the luxury of looking for a "true hypothesis" because the definition of a true hypothesis is an ontological one beyond the scope of science (it tends to be more in the domain of religion). Instead, discovery is treated as a slow accumulation of improvements.

In theory, if we opened the gates to consider a wider range of more complex hypothesis, we would be more likely to consider the "true" hypothesis in the first round. However, does that mean we are more likely to act upon the true hypothesis? That's a harder question.

Consider a set of hypothesis which are built from a kernel hypothesis, H. Define a transform (H, n)->H' which creates a new hypothesis by combining the kernel hypothesis with some numerically defined alterations to create a new hypothesis. This could easily take the form of "adding additional degrees of freedom," which you mention.

Now, lets also include a "simplest" hypothesis, S, and let's instantiate a few hypothesis, H1, H2, and H3 (generated from (H, 1), (H, 2), and (H, 3)) respectively. It is clear that we could define as many of these as we want, and "dilute" the effect of any one hypothesis, including S.

In many cases, it is remarkably hard to prove "All hypotheses constructed using (H, n)->H' are less likely to be 'true' than S." This creates a frustrating dilemma: just by counting from 1 on upwards, a single individual could construct an arbitrary number of hypothesis to be tested, one of which might be true.

In an ideal world, where discussing hypothesis is free and takes no time, this is not a concern. One simply discusses the infinite set of hypothesis, draw a conclusion, and move on. However, in the real world, testing a theory is expensive. We do not want to make it easy to cloud the waters.

So science's solution is Occam's razor. Instead of trying to jump to the "true" hypothesis in one jump, science simply tries to jump "closer." Simplicity is something that can be reasonably agreed upon (it's not perfect, but its far easier than getting agreement on "truthfulness"). So science, and Occam's Razor declares a postulate: "Generally speaking, finding a simple and mostly-right hypothesis will lead us to a better solution in the next iteration than we would have gotten from selecting more complex hypothesis." Thus, if the "true" hypothesis, lets choose H3, had an extra variable, science claims that it will arrive at H3 cheaper and faster if it first goes through S, and then extends that to H3. It claims that if it had to jump to H3, the process would be more expensive and slower.

I draw attention to the word "postulate." There is nothing that says "Occam's Razor is the true way to find knowledge." It's just a principle that has been successful enough in scientific inquiry to reach a high level of prestige.

I think you could find literature on the topic in general. The nature of Occam's razor is well defined in terms of optimizing systems. In its traditional usage, Occam's razor is an approach seeking to optimize the value of hypotheses generated against the time spent on them. If you wished to explore alternative optimizations which find the "true" hypothesis faster, you could start by exploring how one would test if a hypothesis is "true," and then see if that leads in directions which call for alternative optimization techniques.

However, I don't think you will see much on the application of Occam's Razor in philosophy in the literature because Occam's razor starts from the premise that that two hypothesis have equal proving power. This is a highly unusual occurrence in philosophy because there is so little agreement on the definition of "proving power." The few places I am familiar with it (phyiscalism vs. dualism vs. idealism being my favorite), we actually do see what you suggest: rather than just starting from one hypothesis and moving forward, philosophy has gone forward with several in parallel.

Answer 3

I'm going to suggest two things in answer.

First, Occam's Razor is not well-defined. On one level, it's not at all clear what William of Ockham wrote except that it is some form of principle of parsimony (link). On another level, it's hard to find someone who knowingly offers an explanation with superfluous details. At least, that's my take on it.

Second, my experience from my undergraduate days in chemistry was that chemists always operate under simplifications, and we try to pick the least sufficient simplification (by which I mean the simplest form that is sufficient for solving the problem).

For instance, in many elementary experiments we act as if ΔH = ΔE. Or we consider gases in accordance with PV=nRT rather than the virial equations needed for real gases (http://facstaff.cbu.edu/rprice/lectures/realgas.html). Similarly, we pick systems that are easy to calculate over those that are hard (beginning with using the metric system over imperial units).

But in terms of philosophy of science literature, I don't know anywhere that directly addresses what you're asking (which could be because my AOSes and AOC are not in philosophy of science).

See especially SEP.

Answer 4

The actual premise seems to be a bit of a dead ringer. Your pool of hypotheses includes the simplest, so being a superset, the probability of finding the best answer in the pool is greater than or equal to the probability of the simplest answer being the best.

As for the applicability of the ideas to philosophy, I've had a little trouble. In theory, it makes sense that Occam's Razor should have some effect here. However, the challenge is that Occam's Razor is that it suggests the "simplest" should be the best. "Simple" is not a simple term in philosophy. People come with different sets of axioms, and the path from them to a theory is not always straightforward. For example, there are many theories which are "simplest" for people who believe in a literal word-for-word translation of the Bible, but which are not "simplest" for an atheist.

I think you could apply a theory along these lines to theories for your own personal use, but I find that the value of Occam's Razor is dwarfed by the complexity of trying to get people to agree on what is simple, so it is harder to apply in a group setting.