5 ms·

I love maths when it takes complicated problems and models them in simple terms. I think "Information Theory" actually does the opposite - it's taken a problem

I love maths when it takes complicated problems and models them in simple terms. I think "Information Theory" actually does the opposite - it's taken a problem about *data encoding* and articulated that problem as something that can tell us about the properties and nature of *information* itself. The theory implies that pieces of information are comparable with other pieces of information but this is only true inside a strict set of boundaries defining what can possibly be encoded. Investigating the nature of information and finding ways to measure it is a subject that will never sit comfortably in the sciences. That's okay, let's not force it! Maybe it's time we reconsider labeling this with the very grand title of "Information Theory".

mrow84 6y ago
Information theory provides a way to measure the "size" of probability distributions, which has applications wherever we are interested in quantifying our uncertainties. For example, it can be used to target observations for reducing forecast uncertainties, such as in [0], or [1]. [0] https://www.sciencedirect.com/science/article/abs/pii/0167715284900208 https://www.sciencedirect.com/science/article/abs/pii/016771... [1] https://www.jmlr.org/papers/volume9/krause08a/krause08a.pdf https://www.jmlr.org/papers/volume9/krause08a/krause08a.pdf
coldtea 6y ago
>The theory implies that pieces of information are comparable with other pieces of information but this is only true inside a strict set of boundaries defining what can possibly be encoded. Information theory is concerned exactly to formalize this "strict set of boundaries". It's not some grandiose claim or some kind of philosophical musings, if you imply that, it's math. At least the information theory as known from Shannon and co.
infosurp 6y ago
> The theory implies that pieces of information are comparable with other pieces of information but this is only true inside a strict set of boundaries defining what can possibly be encoded. sure, but those boundaries are the known universe photons are fungible
- benjaminjosephw 6y ago
  How much information is in the story of Little Red Riding Hood? You could measure the bits encoded in the text but is that really measuring the amount of information communicated? I'm sure that story has additional meaning (and less surprise) to folk in Europe who recognize the symbolism and analogues in the story to relationships to other parts of culture. Does this story have more or less information in it than Goldilocks and the Three Bears? You could measure the amount of information encoded in the telling of the story to find out but that would give you a very unsatisfactory answer.
  - milesvp 6y ago
    I have a feeling you may be conflating some concepts in information theory, and possibly redefining information in multiple places. Goldilocks and Red Riding Hood require a codec to decipher. Otherwise they are literally just strings of bits. A simple codec can take the bitstring and encode it as characters. So that you, may use your codec (your eyes) to send an encoding to another codec (your brain) to decipher and finally intepret the meaning. Codecs are kind of a tricky thing, in that they can compress a lot of information into a single bit, but to set up that compression requires a lot of ‘work’ outside the scope of a given transmission window. To answer your question as to which story contains more information, you need to first ask ‘how sophisticated is my codec?’ I can tell you from watching children grow up, the same story can have vastly different information conveyed depending on when the story is consumed.
  - coldtea 6y ago
    >How much information is in the story of Little Red Riding Hood? You could measure the bits encoded in the text but is that really measuring the amount of information communicated? Yes, in several important ways. It's not about putting a number to the "subjective experience" of reading/appreciating the story however.
  - coderenegade 6y ago
    I don't see any reason why information theory has to be limited to encoded information in bits. It deals generally with a quantity p(x), with no qualification for what p(x) is. You could, for example, tokenize words and figure out how complex the language used is, given some measure over the distribution of words in the english language. Similarly, you could go higher and work with characters, and figure out the surprise factor of using bears, but again you would need some model or measure of the distribution of character types in english literature. Figuring out the entropy of a coin toss is fairly trivial, but figuring out the entropy of, say, a story book, requires coming up with a model of the distribution over story books, which is the hard part. In that regard, information theory is quite powerful and gives us the tools to do all of that, it's just that the models are highly non-trivial, or otherwise missing.
henrikeh 6y ago
Richard Hamming (known for the Hamming code among others) has said that "information theory" ought to be called "communication theory". Obviously, communication theory is a field in itself, but I think the gist of his argument holds somewhat true in that information theory tells little about what information is intrinsic but instead tells about what information can be _learned_. But I think the intuition breaks down. What is mutual information then? Classically, it is the information which can be reliably transmitted between a transmitter and receiver (given some model of both.) However, the unit is still bits.
- superfist 6y ago
  Do you know exact source of that quote? Might be useful for my future paper around this area. Norman Abramson also had similar opinion.
  - henrikeh 6y ago
    Yes, it is from Hamming's "The Art of Doing Science and Engineering". From page 89: > Information Theory was created by C.E.Shannon in the late 1940s. The management of Bell Telephone Labs wanted him to call it “'Communication Theory” as that is a far more accurate name, but for obvious publicity reasons “Information Theory” has a much greater impact—this Shannon chose and so it is known to this day. The title suggests the theory deals with information—and therefore it must be important since we are entering more and more deeply into the information age. Then later on on page 90: > First, we have not defined “information”, we merely gave a formula for measuring the amount. Second, the measure depends on surprise, and while it does match, to a reasonable degree, the situation with machines, say the telephone system, radio, television, computers, and such, it simply does not represent the normal human attitude towards information. Third, it is a relative measure, it depends on the state of your knowledge. If you are looking at a stream of “random numbers” from a random source then you think each number comes as a surprise, but if you know the formula for computing the “random numbers” then the next number contains no surprise at all, hence contains no information! Thus, while the definition Shannon made for information is appropriate in many respects for machines, it does not seem to fit the human use of the word. This is the reason it should have been called “Communication Theory”, and not “Information Theory”. It is too late to undo the definition (which produced so much of its initial popularity, and still makes people think it handles “information”) so we have to live with it, but you should clearly realize how much it distorts the common view of information and deals with something else, which Shannon took to be surprise. > This is a point which needs to be examined whenever any definition is offered. How far does the proposed definition, for example Shannon’s definition of information, agree with the original concepts you had, and how far does it differ? Almost no definition is exactly congruent with your earlier intuitive concept, but in the long run it is the definition which determines the meaning of the concept—hence the formalization of something via sharp definitions always produces some distortion.

mrow84 6y ago

Information theory provides a way to measure the "size" of probability distributions, which has applications wherever we are interested in quantifying our uncertainties. For example, it can be used to target observations for reducing forecast uncertainties, such as in [0], or [1]. [0] https://www.sciencedirect.com/science/article/abs/pii/0167715284900208 https://www.sciencedirect.com/science/article/abs/pii/016771... [1] https://www.jmlr.org/papers/volume9/krause08a/krause08a.pdf https://www.jmlr.org/papers/volume9/krause08a/krause08a.pdf

coldtea 6y ago

>The theory implies that pieces of information are comparable with other pieces of information but this is only true inside a strict set of boundaries defining what can possibly be encoded. Information theory is concerned exactly to formalize this "strict set of boundaries". It's not some grandiose claim or some kind of philosophical musings, if you imply that, it's math. At least the information theory as known from Shannon and co.

infosurp 6y ago

benjaminjosephw 6y ago
How much information is in the story of Little Red Riding Hood? You could measure the bits encoded in the text but is that really measuring the amount of information communicated? I'm sure that story has additional meaning (and less surprise) to folk in Europe who recognize the symbolism and analogues in the story to relationships to other parts of culture. Does this story have more or less information in it than Goldilocks and the Three Bears? You could measure the amount of information encoded in the telling of the story to find out but that would give you a very unsatisfactory answer.
- milesvp 6y ago
  I have a feeling you may be conflating some concepts in information theory, and possibly redefining information in multiple places. Goldilocks and Red Riding Hood require a codec to decipher. Otherwise they are literally just strings of bits. A simple codec can take the bitstring and encode it as characters. So that you, may use your codec (your eyes) to send an encoding to another codec (your brain) to decipher and finally intepret the meaning. Codecs are kind of a tricky thing, in that they can compress a lot of information into a single bit, but to set up that compression requires a lot of ‘work’ outside the scope of a given transmission window. To answer your question as to which story contains more information, you need to first ask ‘how sophisticated is my codec?’ I can tell you from watching children grow up, the same story can have vastly different information conveyed depending on when the story is consumed.
- coldtea 6y ago
  >How much information is in the story of Little Red Riding Hood? You could measure the bits encoded in the text but is that really measuring the amount of information communicated? Yes, in several important ways. It's not about putting a number to the "subjective experience" of reading/appreciating the story however.
- coderenegade 6y ago
  I don't see any reason why information theory has to be limited to encoded information in bits. It deals generally with a quantity p(x), with no qualification for what p(x) is. You could, for example, tokenize words and figure out how complex the language used is, given some measure over the distribution of words in the english language. Similarly, you could go higher and work with characters, and figure out the surprise factor of using bears, but again you would need some model or measure of the distribution of character types in english literature. Figuring out the entropy of a coin toss is fairly trivial, but figuring out the entropy of, say, a story book, requires coming up with a model of the distribution over story books, which is the hard part. In that regard, information theory is quite powerful and gives us the tools to do all of that, it's just that the models are highly non-trivial, or otherwise missing.

henrikeh 6y ago

Richard Hamming (known for the Hamming code among others) has said that "information theory" ought to be called "communication theory". Obviously, communication theory is a field in itself, but I think the gist of his argument holds somewhat true in that information theory tells little about what information is intrinsic but instead tells about what information can be _learned_. But I think the intuition breaks down. What is mutual information then? Classically, it is the information which can be reliably transmitted between a transmitter and receiver (given some model of both.) However, the unit is still bits.

superfist 6y ago
Do you know exact source of that quote? Might be useful for my future paper around this area. Norman Abramson also had similar opinion.
- henrikeh 6y ago
  Yes, it is from Hamming's "The Art of Doing Science and Engineering". From page 89: > Information Theory was created by C.E.Shannon in the late 1940s. The management of Bell Telephone Labs wanted him to call it “'Communication Theory” as that is a far more accurate name, but for obvious publicity reasons “Information Theory” has a much greater impact—this Shannon chose and so it is known to this day. The title suggests the theory deals with information—and therefore it must be important since we are entering more and more deeply into the information age. Then later on on page 90: > First, we have not defined “information”, we merely gave a formula for measuring the amount. Second, the measure depends on surprise, and while it does match, to a reasonable degree, the situation with machines, say the telephone system, radio, television, computers, and such, it simply does not represent the normal human attitude towards information. Third, it is a relative measure, it depends on the state of your knowledge. If you are looking at a stream of “random numbers” from a random source then you think each number comes as a surprise, but if you know the formula for computing the “random numbers” then the next number contains no surprise at all, hence contains no information! Thus, while the definition Shannon made for information is appropriate in many respects for machines, it does not seem to fit the human use of the word. This is the reason it should have been called “Communication Theory”, and not “Information Theory”. It is too late to undo the definition (which produced so much of its initial popularity, and still makes people think it handles “information”) so we have to live with it, but you should clearly realize how much it distorts the common view of information and deals with something else, which Shannon took to be surprise. > This is a point which needs to be examined whenever any definition is offered. How far does the proposed definition, for example Shannon’s definition of information, agree with the original concepts you had, and how far does it differ? Almost no definition is exactly congruent with your earlier intuitive concept, but in the long run it is the definition which determines the meaning of the concept—hence the formalization of something via sharp definitions always produces some distortion.