What Is Entropy?
Competing definitions of entropy—from “energy unavailable for work” to “missing information about a system”—highlight how the same mathematical quantity underpins both thermodynamics and information theory. Commenters debate whether entropy is fundamentally a physical property or an observer‑dependent measure of uncertainty, and how concepts like microstates, probability distributions, and Shannon’s formula tie the perspectives together. The thread also touches on the limits of intuitive explanations, the arrow of time, and why linking entropy across physics and information theory is both powerful and conceptually treacherous.
Competing Definitions of Entropy
- Several definitions surface:
- Thermodynamic: “energy unavailable for useful work.”
- Statistical: log of the number of microstates compatible with a macrostate.
- Information-theoretic: a functional on a probability distribution, typically (-\sum p_i \log p_i), interpreted as missing information or uncertainty.
- Some argue the information-theoretic notion is most fundamental, with physical entropy as its application; others see this as misleading or only “shallowly” connected.
- Strong pushback against definitions tying entropy directly to “potential,” “pressure,” or treating it as a force that “creates” attraction/repulsion.
Thermodynamics vs Information Theory
- One camp emphasizes close mathematical equivalence:
- Boltzmann/Gibbs and Shannon entropies coincide for appropriate ensembles.
- Thermodynamic entropy can be derived via maximum-entropy principles.
- Links discussed via statistical mechanics, Liouville’s theorem, Maxwell’s demon.
- Another camp stresses interpretive differences:
- Thermodynamic entropy is tied to macrostates, irreversibility, and the second/third laws.
- Information entropy lacks direct analogues of these laws and applies to any probabilistic setting (algorithms, data, language).
Subjective vs Objective Entropy
- Debate over whether entropy is a property of the system or of an observer’s knowledge.
- Examples: RNG with known vs unknown seed; different observers attaching different distributions.
- Some insist physical entropy is objective (measured via calorimetry, independent of what we know).
- Others maintain probabilities — and thus entropies — are inherently tied to information.
- Cross-entropy and KL divergence are highlighted as tools relating “true” distributions to subjective beliefs.
Arrow of Time and the Second Law
- Discussion of why entropy tends to increase despite time-symmetric microphysics.
- The “Past Hypothesis” (universe starting in a low-entropy state) is cited as needed to get a time-directed second law.
- One view calls the second law almost tautological: systems evolve toward more probable (higher-entropy) macrostates.
Pedagogy and Intuition
- Frustration with vague or mystical treatments; advocacy for starting from the precise ( -\sum p \log p ) definition.
- Others argue that without macro/micro-state intuition, that formula alone is not very illuminating.
- Practical intuitions discussed: entropy as ideal compression limit, “bits you don’t have,” broken vs unbroken egg, and large-deviation (balls-in-bins) viewpoints.