3 ms·

> The theory implies that pieces of information are comparable with other pieces of information but this is only true inside a strict set of boundaries defining

> The theory implies that pieces of information are comparable with other pieces of information but this is only true inside a strict set of boundaries defining what can possibly be encoded. sure, but those boundaries are the known universe photons are fungible

benjaminjosephw 6y ago
How much information is in the story of Little Red Riding Hood? You could measure the bits encoded in the text but is that really measuring the amount of information communicated? I'm sure that story has additional meaning (and less surprise) to folk in Europe who recognize the symbolism and analogues in the story to relationships to other parts of culture. Does this story have more or less information in it than Goldilocks and the Three Bears? You could measure the amount of information encoded in the telling of the story to find out but that would give you a very unsatisfactory answer.
- milesvp 6y ago
  I have a feeling you may be conflating some concepts in information theory, and possibly redefining information in multiple places. Goldilocks and Red Riding Hood require a codec to decipher. Otherwise they are literally just strings of bits. A simple codec can take the bitstring and encode it as characters. So that you, may use your codec (your eyes) to send an encoding to another codec (your brain) to decipher and finally intepret the meaning. Codecs are kind of a tricky thing, in that they can compress a lot of information into a single bit, but to set up that compression requires a lot of ‘work’ outside the scope of a given transmission window. To answer your question as to which story contains more information, you need to first ask ‘how sophisticated is my codec?’ I can tell you from watching children grow up, the same story can have vastly different information conveyed depending on when the story is consumed.
- coldtea 6y ago
  >How much information is in the story of Little Red Riding Hood? You could measure the bits encoded in the text but is that really measuring the amount of information communicated? Yes, in several important ways. It's not about putting a number to the "subjective experience" of reading/appreciating the story however.
- coderenegade 6y ago
  I don't see any reason why information theory has to be limited to encoded information in bits. It deals generally with a quantity p(x), with no qualification for what p(x) is. You could, for example, tokenize words and figure out how complex the language used is, given some measure over the distribution of words in the english language. Similarly, you could go higher and work with characters, and figure out the surprise factor of using bears, but again you would need some model or measure of the distribution of character types in english literature. Figuring out the entropy of a coin toss is fairly trivial, but figuring out the entropy of, say, a story book, requires coming up with a model of the distribution over story books, which is the hard part. In that regard, information theory is quite powerful and gives us the tools to do all of that, it's just that the models are highly non-trivial, or otherwise missing.
- abdullahkhalids 6y ago
  > I'm sure that story has additional meaning (and less surprise) to folk in Europe Indeed, information is a receiver dependent quantity, as you say, and is formalized as such in any information theory textbook. Really, the information in a message about variable X is how much the receiver's subjective probability distribution about X is constrained by the message. In practice, creating a complete model of the system i.e. enumerating all the possible variables and their prior probabilities is impossible. However, this doesn't mean the entire field is useless. In fact, the work of Shannon and others helps us understand the limits of what is possible to model and gain information about, and by how much. What those limits mean requires grappling with philosophical questions far above my paygrade.

benjaminjosephw 6y ago

How much information is in the story of Little Red Riding Hood? You could measure the bits encoded in the text but is that really measuring the amount of information communicated? I'm sure that story has additional meaning (and less surprise) to folk in Europe who recognize the symbolism and analogues in the story to relationships to other parts of culture. Does this story have more or less information in it than Goldilocks and the Three Bears? You could measure the amount of information encoded in the telling of the story to find out but that would give you a very unsatisfactory answer.

milesvp 6y ago
I have a feeling you may be conflating some concepts in information theory, and possibly redefining information in multiple places. Goldilocks and Red Riding Hood require a codec to decipher. Otherwise they are literally just strings of bits. A simple codec can take the bitstring and encode it as characters. So that you, may use your codec (your eyes) to send an encoding to another codec (your brain) to decipher and finally intepret the meaning. Codecs are kind of a tricky thing, in that they can compress a lot of information into a single bit, but to set up that compression requires a lot of ‘work’ outside the scope of a given transmission window. To answer your question as to which story contains more information, you need to first ask ‘how sophisticated is my codec?’ I can tell you from watching children grow up, the same story can have vastly different information conveyed depending on when the story is consumed.
coldtea 6y ago
>How much information is in the story of Little Red Riding Hood? You could measure the bits encoded in the text but is that really measuring the amount of information communicated? Yes, in several important ways. It's not about putting a number to the "subjective experience" of reading/appreciating the story however.
coderenegade 6y ago
I don't see any reason why information theory has to be limited to encoded information in bits. It deals generally with a quantity p(x), with no qualification for what p(x) is. You could, for example, tokenize words and figure out how complex the language used is, given some measure over the distribution of words in the english language. Similarly, you could go higher and work with characters, and figure out the surprise factor of using bears, but again you would need some model or measure of the distribution of character types in english literature. Figuring out the entropy of a coin toss is fairly trivial, but figuring out the entropy of, say, a story book, requires coming up with a model of the distribution over story books, which is the hard part. In that regard, information theory is quite powerful and gives us the tools to do all of that, it's just that the models are highly non-trivial, or otherwise missing.
abdullahkhalids 6y ago
> I'm sure that story has additional meaning (and less surprise) to folk in Europe Indeed, information is a receiver dependent quantity, as you say, and is formalized as such in any information theory textbook. Really, the information in a message about variable X is how much the receiver's subjective probability distribution about X is constrained by the message. In practice, creating a complete model of the system i.e. enumerating all the possible variables and their prior probabilities is impossible. However, this doesn't mean the entire field is useless. In fact, the work of Shannon and others helps us understand the limits of what is possible to model and gain information about, and by how much. What those limits mean requires grappling with philosophical questions far above my paygrade.