4 ms·

"A bit isn't a divisible unit" What ? Sure it is. When you measure the information content (or, the entropy) of a message, you very frequently get non-intege

• kev009 10y ago
  That is expressing a ratio. A more concrete example, .46 a person doesn't exist even if that ratio is useful for expressing statistics.
  • drdeca 10y ago
    I think "gaining half a bit of information" about something can correspond to stuff that lets you update your probability distribution about it? I'm not sure.
    • kristjankalm 10y ago
      yes, but your gain is still an integer number. the non-integer ratio is just a statistic (gain per number of units).
      • jamessb 10y ago
        Why is it an integer number? Entropy (in bits) is defined as - \sum_x (p(x) log_2 p(x)) There is no reason this has to be an integer, since probabilities are not restricted to being reciprocals of powers of 2. Consider also that you can simply use a different logarithm base to get a different unit (e.g. use the natural logarithm to obtain the entropy in nats). It would be bizarre if the arbitrary choice of 2 as the base gave a unit that was indivisible. I think this whole confusion comes down to the difference between a bit as a "unit of information in the sense of information theory" [divisible] and a bit as a "single physical one or zero" [not divisible]. The relationship between the two is that the entropy of a random variable is a lower bound on the average number of bits required to represent it.
        • kristjankalm 10y ago
          yes, you are quite right -- i was referring to the binary digit rather than the shannon
      • tripzilch 10y ago
        only when you consider bits to be the final, indivisible, fundamental unit of information. which they aren't. if you have a data storage thingy that can store any of three values, a ternary digit, it is exactly equivalent to log2(3) = ln(3) / ln(2) ~= 1.585 bits. kind of like US pop-science articles like to say stuff like "a volume 1.5 olympic-size swimming pools" (because a megagallon is just weird), even though obviously, can never have half of such a pool or it would empty. (ok after some consideration, you could have the bottom half)
        • kristjankalm 10y ago
          you are obviously right, but i think that in the specific case described above -- computer code -- we have binary digits as final and indivisible units.
        • pixl97 10y ago
          Filling the top half would be more fun https://what-if.xkcd.com/6/ https://what-if.xkcd.com/6/
• PeCaN 10y ago
  > you very frequently get non-integer numbers of bits per (character/unit/message/whatever) That'd be an average, and it's like when you have 2.58 people per household. Presumably most people do not keep about 6/10 of a person around. The point of a bit (in information theory) is that it is the smallest possible (read: not divisible) unit of information.
• akvadrako 10y ago
  Or to express it another way, a variable with 3 possible states has 0.5 bits more capacity than one with 2.
  • stephencanon 10y ago
    0.585... bits more capacity, but yeah (bits = log2(states)).