Entropy & it's connection with information

Entropy as Information: Deriving the Gibbs-Shannon Formula

What exactly is entropy? While often loosely defined as "disorder," statistical mechanics offers a much more precise definition: Entropy is the expected value of information. By treating physical events as probabilistic outcomes, we can derive the fundamental equations of thermodynamics from pure logic.

1. Information Associated with an Event

For any physical event occurring with probability \( p \), we associate a quantity called Information, \( I(p) \). To make physical sense, this quantity must satisfy three logical properties:

• (i) Non-negativity: \[ I(p) \ge 0 \]
  Physical meaning: Observing an event cannot reduce your knowledge. You can learn something, or you can learn nothing, but you cannot be "negatively surprised."
• (ii) Additivity for Independent Events: \[ I(p_1 p_2) = I(p_1) + I(p_2) \]
  Physical meaning: If two events are statistically independent, the "surprise" of observing both is the sum of the individual surprises. This mirrors the property of extensivity in thermodynamics.
• (iii) Zero Information for Certainty: \[ I(1) = 0 \]
  Physical meaning: If an event is guaranteed to happen (\( p=1 \)), observing it tells you nothing new. It is informationally empty.

2. The Functional Form

The only mathematical function that satisfies the additivity property \( f(xy) = f(x) + f(y) \) is the logarithm. To ensure positivity (since \( \log p \) is negative for \( 0 \le p \le 1 \)), we require a negative coefficient:

\[ I(p) = -k \log p \]

Here, \( k > 0 \) is a constant (Boltzmann's constant in physics). This formula tells us that rare events (small \( p \)) carry high information ("High Surprise").

3. Expected Information = Entropy

Suppose a system has multiple possible microstates labeled by \( n \), occurring with probabilities \( \{p_n\} \). The total Entropy \( S \) is defined as the weighted average (expectation value) of the information:

\[ \langle I \rangle = \sum_n p_n I_n = -k \sum_n p_n \log p_n \]

This leads to the general definition of entropy in statistical mechanics (Gibbs Entropy):

\[ S = -k \sum_n p_n \log p_n \]

4. Physical Interpretation

This definition corrects common misconceptions:

• Not just disorder: A "messy" room might actually be a very specific, low-probability state.
• Measure of Spread: \( S \) measures the width of the probability distribution. It quantifies our ignorance about the precise microstate of the system.

5. Connection to Thermodynamics

Case A: Perfect Crystal (Sharp Distribution)

If the system is definitely in one state (say state 1), then \( p_1 = 1 \) and all other \( p = 0 \).

\[ S = -k(1 \log 1) = 0 \]

This corresponds to the Third Law of Thermodynamics (zero entropy at absolute zero).

Case B: Microcanonical Ensemble (Uniform Distribution)

If the system is equally likely to be in any of \( W \) available states, then \( p_n = 1/W \).

\[ S = -k \sum_{n=1}^{W} \frac{1}{W} \log \left(\frac{1}{W}\right) \] \[ S = -k \left( W \cdot \frac{1}{W} \right) (-\log W) \]

This simplifies to the famous Boltzmann equation:

\[ S = k \log W \]