1

So there are various interpretations of probability.

Frequentism is the likelihood of events of say for example if I roll a dice the likelihood of of getting a 5 is 1/6 if repeated over and over.

Propensity is given by how much you know about the system for example if I know the initial conditions of the dice roll well enough then I can predict the odds of a 5 is 100% (for example).

In statistical mechanics the thermodynamic entropy of a system seems to be describe the likelihood of an event but then we also have the notion of information entropy which is related to the second type of probability. Many physicists are comfortable swapping and conflating the two. Going so far that they even equate the two. (See Is information entropy the same as thermodynamic entropy? (PhysicsSE).)

What is the relationship if any according to philosophers between information and thermodynamic entropy?

10
  • Who are those many physicists who equate Shannon entropy and thermodynamic entropy ? While there are superficial similarities between the 2 notions, they are certainly not the same. Take a deck of poker cards sorted in order, shuffle it: while thermodynamically the effect is negligible, you have significantly increased the Shannon entropy of the deck (I.e. You can't say in advance which card you are going to draw anymore)
    – armand
    Mar 11, 2022 at 7:11
  • @armand it's a very common point of view imo physics.stackexchange.com/questions/263197/… Mar 11, 2022 at 7:21
  • What they are saying is the formulas are similar. Yet they are not the same thing, one can grow while the other is stable, as demonstrated by the poker deck example.
    – armand
    Mar 11, 2022 at 11:44
  • 1
    @armand: Thermodynamic entropy is typically described as an application of Shannon's information theory: en.wikipedia.org/wiki/…
    – CriglCragl
    Mar 11, 2022 at 13:11
  • 2
    Added links, clarified question, and added 'modality', 'probability', and 'philosophy-of-information' since the question asks after information theory.
    – J D
    Mar 11, 2022 at 18:03

3 Answers 3

1

You never have 100% predictive power. Quantum events involve fundamental uncertainties, and can in principle effect macroscopic events (eg a powerful cosmic ray changing the momentum of the dice at a critical moment, or even more uncertainly a beta decay of a carbon 14 nucleus). There are finite probabilities of landing on an edge or point also. It's good to think about probability as a way to deal with limited information, but the uncertainty principle and the observer effect (the impact of taking measurements on increasing uncertainty) mean information is always limited, in a way that I would describe as there not being such complete information meaningfully in our universe. Three-body dynamics of large interacting blackholes mean uncertainties below the Planck scale can have have macroscopic effects.

Entropy causes a number of confusions. It is relative not absolute, strictly speaking we can only measure change in it, and if there are hidden microstates or degrees of freedom calculations will be incomplete. We typically idealise systems as closed and close to equilibrium, far-from equilibrium thermodynamics is much harder. And open systems where the Gibbs free-energy is more significant than simple entropy gradients are important to us, because it's part of the thermodynamic understanding of life.

Physicists play a sleight of hand, by defining information as the inverse of entropy. The full analysis of Maxwell's Demon links information-entropy and thermodynamics, so it makes sense even if in ordinary language information means a range of less specific things. Discussed here Is the concept of information nonphysical?

I see modern science as currently property-dualist, picturing everything as fundamentally constituted of energy, and information - which includes entropy. It was expected that information would resolve to being secondary properties of the fundamental things, mass-energy and spacetime. But now increasingly the reverse is expected, with those previously fundamental things pictured as emergent from information and it's propagation. Discussed here: Is information the foundation of reality?

We can understand 'true' as relating to comparing expectation and reality, and specifically in regards to building a tables of probabilities like the dice-rolling example, composed of counterfactuals. Discussed here: Why is a measured true value “TRUE”? Deutsch & Marletto's Constructor Theory looks to relate physics and information-theory in a deeper way, by expanding analysis of computation into sets of outcomes instead of a single floating-point variable, comparable to expanding our understanding of now to include Many Worlds.

0

a) Entropy is not about probabilities (although probabilities might be used to calculate it). In simple terms, entropy is a quantity that measures the potential of disorder. It is not a direct measure of disorder. The fact that low entropy implies high predictability is due to the reduction of the potential disorder, not to the reduction of disorder. At S=0, there is no potential for disorder, so, no disorder.

b) Entropy is a macrostatic quantity. Memorize that. That is: it is a descriptor of facts of perception, that is, ideas, sensations (for example, temperature is a macrostatic quantity: it is a feeling, which (thanks, 0th Law, sorry you didn't were defined first), it can be assessed as a physical quantity).

c) While macrostatic facts are related to ideas, microstatic facts are those related with physical phenomena. For example, microstatically, temperature is a quantity proportional to the statistical kinetic energy load of each molecule in a gas container.

d) Statistical entropy (S=k ln $ \omega $) describes the information carried for some macroscopic state:

S=ln2(64)=6

That is, in order to carry 64 states, 6 bits are necessary (logarithm base 2). So, in this example, the entropy of a system which can be in 64 states is 6.

d) Contrary to entropy, information is a microstatic quantity. Following the previous example, 111000 is the information that corresponds for a particular microstate of the system. Although this information is effectively carried in 6 bits, that is not necessarily the same of the entropy of the system. For example, Gibbs entropy can have a different value for the same 6 bits of information.

e) Thermodynamic entropy is equivalent to statistical entropy: it is a macrostatic quantity that measures the potential of disorder of a particular microstate. Perhaps the best interpretation here is to say that thermodynamic entropy is the measure of energetic order.

f) The big difference between thermodynamic and statistical entropy is that the latter allows measuring each microstate, at the cost of addressing systems always as discrete entities.

In addition to know that the entropy of a 6-bits message is 6, each message (the information) can also be obtained: 111000. Statistical entropy is mostly used to analyze fixed-size messages (but yes, it also allows addressing messages of different sizes). That is, statistical entropy is mostly used not to measure disorder, but how to assess the performance of a process at different values of disorder.

But the for the former, thermodynamic entropy, microstates cannot be measured. How would we know the actual configuration of X molecules having a value of entropy of S=0.nnnnn...? In addition, the absolute value of entropy of any substance cannot be measured directly, the only measurable value is dS=dQ/T, that is, change. So, thermodynamic entropy is mostly used to measure disorder as such.

Don't forget that S=0 is a convention for zero temperature, which experimentally shows that mass tends to order at 0∘K.

0

Information and thermodynamic entropies are in some way parallel concepts. Both describe how the complexity of the system increases over time. More and more information is needed to describe the state of the system.

When the signal to noise ratio decreases over time, the information in the signal does not disappear, it is only diluted in the constant influx of random noise. Random information is added to the signal.

Something like that is happening also in thermodynamic processes. Differences in energy density are parallel to the signal. Thermal noise, random vibration of particles distributes the energy more evenly.

It all boils down to the inherent randomness of a probabilistic universe. In a deterministic universe there would be no noise, complexity would not increase, all entropies would remain constant.

4
  • And you know the last sentence to be true because you yourself exhibit infinite complexity/information? At least nothing in your post allows for such a conclusion.
    – Philip Klöcking
    Mar 11, 2022 at 12:16
  • In determinism there is no noise. This is not a conclusion. This is implied in the definition. Mar 11, 2022 at 12:40
  • Like every kind of determinism was binary, sure...
    – Philip Klöcking
    Mar 11, 2022 at 12:52
  • It's a mistake to picture entropy as all about noise & heat. Consider predicting the outcome of coin faces of n coins shaken in a box. As n increases, the closeness of the average balance of faces will get closer & closer to 50%, & by 10^23 'coins' you get very reliable predictions eg gas laws. Thermal states are molecular spin states, lattice vibrations, kinetic energy etc. You can understand that kind of spread-out energy doesn't tend to concentrate, just like many coins don't all come up heads. The 2nd law of thermodynamics is like saying signals deteriorate, they don't increase >100%.
    – CriglCragl
    Mar 11, 2022 at 18:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .