Actually, I'm not sure this is mostly about a mathematical vs computational perspective. Let's look at how entropy is defined. It's not just the number of pos

Actually, I'm not sure this is mostly about a mathematical vs computational perspective. Let's look at how entropy is defined. It's not just the number of possibilities, it's how likely they are. For example, consider two functions: A: 1% chance of 1, 1% chance of 2, 98% chance of 3 B: 50% chance of 1, 50% chance of 2 The entropy of a distribution, in bits, is: H = - sum over all possibilities of (p * log_2 p) So for A it is: H_A = -(2 * 0.01 * log_2(0.01) + 0.98 * log_2(0.98)) = 0.16 And for B it is: H_B = -(2 * 0.5 * log_2(0.5)) = 1 So you can see that A has lower entropy than B, because it's more concentrated. For continuous distributions this is defined with an integral instead of a sum, but it's the same idea: H = - integral from -inf to + inf of (p * log_2 p) Whether there's more entropy in a sample drawn from a uniform or normal distribution, then, depends on how wide your distributions are. Consider: C: uniform distribution from -1000 to +1000 D: normal distribution centered on 0 with standard deviation 10 E: uniform distribution from -1 to 1 We then have: H_C = - log_2(2000) / log_2(2) = 10.97 H_D = 1/2 (log_2(2pi * 10^2) + 1) = 5.15 H_E = 1 Note that continuous and discrete entropies mean different things, and their values can't be compared directly. That H_E and H_B are both 1 is not meaningful.