4 ms·

> a function drawing from a normal distribution is less random than one drawing from a uniform distribution That's some mathematical nonsense. A normal distri

by eterm 3y ago

> a function drawing from a normal distribution is less random than one drawing from a uniform distribution That's some mathematical nonsense. A normal distribution is continuous from negative infinity to positive infinity, so even by your flawed judgement of what makes something "random", a sample from the normal distrubtion has far more possibilities. The standard normal distribution is almost the definition of random.

jefftk 3y ago
Again, it depends on whether you are looking at this from a pure math perspective or a "math as implemented on computers" perspective. I'm only claiming that this use of more/less random makes sense in a computational context. But in that context, there is more entropy in a float32 drawn from a uniform distribution than a float32 drawn from a normal distribution, no?
- jefftk 3y ago
  Actually, I'm not sure this is mostly about a mathematical vs computational perspective. Let's look at how entropy is defined. It's not just the number of possibilities, it's how likely they are. For example, consider two functions: A: 1% chance of 1, 1% chance of 2, 98% chance of 3 B: 50% chance of 1, 50% chance of 2 The entropy of a distribution, in bits, is: H = - sum over all possibilities of (p * log_2 p) So for A it is: H_A = -(2 * 0.01 * log_2(0.01) + 0.98 * log_2(0.98)) = 0.16 And for B it is: H_B = -(2 * 0.5 * log_2(0.5)) = 1 So you can see that A has lower entropy than B, because it's more concentrated. For continuous distributions this is defined with an integral instead of a sum, but it's the same idea: H = - integral from -inf to + inf of (p * log_2 p) Whether there's more entropy in a sample drawn from a uniform or normal distribution, then, depends on how wide your distributions are. Consider: C: uniform distribution from -1000 to +1000 D: normal distribution centered on 0 with standard deviation 10 E: uniform distribution from -1 to 1 We then have: H_C = - log_2(2000) / log_2(2) = 10.97 H_D = 1/2 (log_2(2pi * 10^2) + 1) = 5.15 H_E = 1 Note that continuous and discrete entropies mean different things, and their values can't be compared directly. That H_E and H_B are both 1 is not meaningful.
  - eterm 3y ago
    I think you've changed what you're meaning by entropy, it isn't really a meaningful measure of "randomness". Afterall, how can U(0,1) be considered to be "less random" than U(0,1000) when U(0,1000) = 1000*U(0,1). A constant factor can't change how "random" something is.
    - jefftk 3y ago
      If you look at this from a computers ("everything is discrete eventually") perspective, U(0,1) is less random than U(0,1000) because U(0,1000) != 1000U(0,1). For example, if you're using a fixed-point encoding that allocates values uniformly, it will have 1000x as many possible values for [0,1000] as [0,1], and so getting any specific value from U(0,1) is 1000x more likely than any specific value from U(0,1000). If you draw from U(0,1) and then multiply by 1000 you'll never produce most values that U(0,1000) could produce. If you look at it from a math ("continuous is actually possible") perspective I have a much weaker handle on it, but I think the problem is that the entropy of 1000U(0,1) is not the same as the entropy of U(0,1)?

by eterm 3y ago

jefftk 3y ago

Again, it depends on whether you are looking at this from a pure math perspective or a "math as implemented on computers" perspective. I'm only claiming that this use of more/less random makes sense in a computational context. But in that context, there is more entropy in a float32 drawn from a uniform distribution than a float32 drawn from a normal distribution, no?

jefftk 3y ago
Actually, I'm not sure this is mostly about a mathematical vs computational perspective. Let's look at how entropy is defined. It's not just the number of possibilities, it's how likely they are. For example, consider two functions: A: 1% chance of 1, 1% chance of 2, 98% chance of 3 B: 50% chance of 1, 50% chance of 2 The entropy of a distribution, in bits, is: H = - sum over all possibilities of (p * log_2 p) So for A it is: H_A = -(2 * 0.01 * log_2(0.01) + 0.98 * log_2(0.98)) = 0.16 And for B it is: H_B = -(2 * 0.5 * log_2(0.5)) = 1 So you can see that A has lower entropy than B, because it's more concentrated. For continuous distributions this is defined with an integral instead of a sum, but it's the same idea: H = - integral from -inf to + inf of (p * log_2 p) Whether there's more entropy in a sample drawn from a uniform or normal distribution, then, depends on how wide your distributions are. Consider: C: uniform distribution from -1000 to +1000 D: normal distribution centered on 0 with standard deviation 10 E: uniform distribution from -1 to 1 We then have: H_C = - log_2(2000) / log_2(2) = 10.97 H_D = 1/2 (log_2(2pi * 10^2) + 1) = 5.15 H_E = 1 Note that continuous and discrete entropies mean different things, and their values can't be compared directly. That H_E and H_B are both 1 is not meaningful.
- eterm 3y ago
  I think you've changed what you're meaning by entropy, it isn't really a meaningful measure of "randomness". Afterall, how can U(0,1) be considered to be "less random" than U(0,1000) when U(0,1000) = 1000*U(0,1). A constant factor can't change how "random" something is.
  - jefftk 3y ago
    If you look at this from a computers ("everything is discrete eventually") perspective, U(0,1) is less random than U(0,1000) because U(0,1000) != 1000U(0,1). For example, if you're using a fixed-point encoding that allocates values uniformly, it will have 1000x as many possible values for [0,1000] as [0,1], and so getting any specific value from U(0,1) is 1000x more likely than any specific value from U(0,1000). If you draw from U(0,1) and then multiply by 1000 you'll never produce most values that U(0,1000) could produce. If you look at it from a math ("continuous is actually possible") perspective I have a much weaker handle on it, but I think the problem is that the entropy of 1000U(0,1) is not the same as the entropy of U(0,1)?