3 ms·
"Also, let's say you have it narrowed down to four people, but the last bit of information is common to all of them." The definition of a bit is something whic
by jerf 15y ago
"Also, let's say you have it narrowed down to four people, but the last bit of information is common to all of them."
The definition of a bit is something which removes half the possibilities. If you have 4 people and acquire a "bit" of information that breaks them into two categories, one with 4 people and one with 0 people, you, by definition, in fact have 0 bits.
Fractional bits are not only possible, they are by far the common case. With a lg2 in the definition of the bit, it's pretty uncommon to have integral bits.
Critical insight: What we call a "bit" in a computer and a "bit" in information theory are related but not the same thing. You can't have a fraction of a bit stored in your computer's RAM, the words are meaningless. It is best to simply flush your idea of what you think a bit is and start over again from scratch when studying information theory, then when you are comfortable with it the connections will become obvious. Starting from the RAM side is actively harmful.
- orthecreedence 15y agoThis makes sense, thanks for explaining the semantics. So basically, you're confirming that you'd need 33 "bits" of information, which doesn't necessarily mean you can use 33 pieces of information because a bit is a lot more specific than a one-off piece of information.
- pohl 15y ago33 bits of information can amount to fewer than 33 pieces of information. Quoting from the article: "...knowing your hometown gives me 16 bits of entropy about you". Edit: a relevant anecdote: when I was in high school, a friend went as an exchange student to Costa Rica. As an experiment, I had her send me a blank postcard with nothing on it but "pohl 68320 USA". It arrived in my post office box without a hitch.
- deleted 15y ago[deleted]
- repsilat 15y agoConversely, though, lots of pieces of data can end up not adding a lot of information, because a lot of data is correlated. Consider the following three facts: 1. A person is Chinese (~2 bits of information) 2. A person lives in Asia (<1 bit of information) 3. A person lives in the USA (4 or 5 bits of information) Now consider combinations of these facts. 1&2 together probably doesn't give you much more than two bits of information still. Conversely, though, 1&3 together give you 10 or 11 bits of information - significantly more than the information of those facts added together. This is because facts 1&2 have a strong positive correlation, and facts 1&3 have a strong negative correlation.