What Is Entropy?

Competing Definitions of Entropy

• Several definitions surface:
  • Thermodynamic: “energy unavailable for useful work.”
  • Statistical: log of the number of microstates compatible with a macrostate.
  • Information-theoretic: a functional on a probability distribution, typically (-\sum p_i \log p_i), interpreted as missing information or uncertainty.
• Some argue the information-theoretic notion is most fundamental, with physical entropy as its application; others see this as misleading or only “shallowly” connected.
• Strong pushback against definitions tying entropy directly to “potential,” “pressure,” or treating it as a force that “creates” attraction/repulsion.

Thermodynamics vs Information Theory

• One camp emphasizes close mathematical equivalence:
  • Boltzmann/Gibbs and Shannon entropies coincide for appropriate ensembles.
  • Thermodynamic entropy can be derived via maximum-entropy principles.
  • Links discussed via statistical mechanics, Liouville’s theorem, Maxwell’s demon.
• Another camp stresses interpretive differences:
  • Thermodynamic entropy is tied to macrostates, irreversibility, and the second/third laws.
  • Information entropy lacks direct analogues of these laws and applies to any probabilistic setting (algorithms, data, language).

Subjective vs Objective Entropy

• Debate over whether entropy is a property of the system or of an observer’s knowledge.
  • Examples: RNG with known vs unknown seed; different observers attaching different distributions.
• Some insist physical entropy is objective (measured via calorimetry, independent of what we know).
• Others maintain probabilities — and thus entropies — are inherently tied to information.
• Cross-entropy and KL divergence are highlighted as tools relating “true” distributions to subjective beliefs.

Arrow of Time and the Second Law

• Discussion of why entropy tends to increase despite time-symmetric microphysics.
• The “Past Hypothesis” (universe starting in a low-entropy state) is cited as needed to get a time-directed second law.
• One view calls the second law almost tautological: systems evolve toward more probable (higher-entropy) macrostates.

Pedagogy and Intuition

• Frustration with vague or mystical treatments; advocacy for starting from the precise ( -\sum p \log p ) definition.
• Others argue that without macro/micro-state intuition, that formula alone is not very illuminating.
• Practical intuitions discussed: entropy as ideal compression limit, “bits you don’t have,” broken vs unbroken egg, and large-deviation (balls-in-bins) viewpoints.