3 ms·
I don't understand this, can you help me understand? I can give you a uniformly random byte. I can also give you a byte distributed from normal distribution suc
by gnulinux 6y ago
I don't understand this, can you help me understand? I can give you a uniformly random byte. I can also give you a byte distributed from normal distribution such that ~127 is more likely than 0 and 255. They'll both be a byte (same bits of information) but getting 0 in one of them will be be a lot more surprising than the other. How is unpredictability a measure of message's information content?
- jevogel 6y agoI believe the comment you responded to is referring to the concept of entropy in information theory. https://en.wikipedia.org/wiki/Entropy_(information_theory) https://en.wikipedia.org/wiki/Entropy_(information_theory)
- gnulinux 6y agoHmm I believe you're correct. The answer to my problem above seems to be that even though information is encoded in 8 bits in both cases, one of them is a lot of more dense.
- carapace 6y agoYes, it's a fundamental result from Information Theory. In the situation you describe the receiver has to know which distribution you're using. See also https://en.wikipedia.org/wiki/Kolmogorov_complexity https://en.wikipedia.org/wiki/Kolmogorov_complexity
- ThrustVectoring 6y agoPredictability and compressibility are the same thing. For large enough quantities of bytes from that distribution, it'd be more efficient to transmit a compression scheme that mirrors the distribution of data you have to send, matching up shorter words on-the-wire with more common bytes.