7 ms·

How Claude Shannon Re-Invented Information (2017)

160 points by dnetesn 8y ago

dang 8y ago
Discussed at the time: https://news.ycombinator.com/item?id=14978399 https://news.ycombinator.com/item?id=14978399
- a_imho 8y ago
  It is submitted by the very same user.
  - de_watcher 8y ago
    He keeps us informed.
    - oli5679 8y ago
      Maybe he's adding redundancy, because he's worried that Hacker news is a noisy channel.
      - stronglikedan 8y ago
        Or, maybe he's adding redundancy, because he's worried that Hacker news is a noisy channel.
  - deleted 8y ago
    [deleted]
  - woodandsteel 8y ago
    And I am so happy he did, since I missed it the first time.
woodandsteel 8y ago
This is the first thing I have read about information theory that really made sense to me. I think I now understand it about ten times as well as I did before. >What does information really measure? It measures the uncertainty we overcome My thought here is that matters because human beings have to make decisions all the time, and reducing the uncertainty regarding the situation makes it more likely you will make the correct decision.
- hyperpallium 8y ago
  > The signicant aspect is that the actual message is one selected from a set of possible messages > If the number of messages in the set is finite then this number [...] can be regarded as a measure of the information produced when one message is chosen from the set, all choices being equally likely. From the abstract of his famous paper http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...
  - bhnmmhmd 8y ago
    It's interesting how this paper was first titled "A mathematical theory of communication" and then renamed to "THE mathematical ...".
- yantrams 8y ago
  Here is a great primer on entropy that you might find helpful - http://colah.github.io/posts/2015-09-Visual-Information/ http://colah.github.io/posts/2015-09-Visual-Information/
- lisper 8y ago
  There's a pithy slogan that sums it up nicely: information is a correlation between the state of two systems. For example: if you measure a voltage using a volt-meter then, if the volt-meter is working properly, there will be a correlation between the readout on the meter and the actual voltage, and so the meter contains information about the voltage. Note that information is symmetric: the voltage in this case also contains information about the meter.
  - jonnydubowsky 8y ago
    Your comment and the one above it are awesome summaries of some of Shannon's concepts. Re:your statement on information being symmetrical, how does zero knowledge encryption, Homomorphic encryption and the like fit into this when there is asymmetrical knowledge and validation without full access to the information?
mkagenius 8y ago
If anyone from Nautilus is here: Would you consider having different prices for different countries? The current pricing is too high for India. A reduced price would bring customers from various countries.
- criddell 8y ago
  A lifetime subscription is $99. If you factor in life expectancy, they are actually charging more to people in India than they are to Americans.
  - Anon84 8y ago
    Yes, if you look at raw dollars per year... but you also have to factor in expected lifetime earnings. What fraction of your expected income is $99? I would wager that it's a much higher fraction in India than in the US.
flopska 8y ago
I recently read the Shannon biography 'A Mind at Play' [0] which I guess is the full version of the article (same authors). It does (IMHO) a very good job at explaining entropy / information. [0] https://www.simonandschuster.com/books/A-Mind-at-Play/Jimmy-Soni/9781476766690 https://www.simonandschuster.com/books/A-Mind-at-Play/Jimmy-...
- goldfeld 8y ago
  I enjoyed the story but found the author(s) practically peddling Shannon's importance as if any of that was needed. All in all the writing in form and substance seemed to me of low quality and in search of dramatic revision and editing. Great history job and lazy writing,
  - woodandsteel 8y ago
    >I enjoyed the story but found the author(s) practically peddling Shannon's importance as if any of that was needed. The vast majority of the public have never heard of Shannon, or have but don't really understand how important he was, so yes, it was needed.

160 points by dnetesn 8y ago

dang 8y ago

Discussed at the time: https://news.ycombinator.com/item?id=14978399 https://news.ycombinator.com/item?id=14978399

a_imho 8y ago
It is submitted by the very same user.
- de_watcher 8y ago
  He keeps us informed.
  - oli5679 8y ago
    Maybe he's adding redundancy, because he's worried that Hacker news is a noisy channel.
    - stronglikedan 8y ago
      Or, maybe he's adding redundancy, because he's worried that Hacker news is a noisy channel.
- deleted 8y ago
  [deleted]
- woodandsteel 8y ago
  And I am so happy he did, since I missed it the first time.

woodandsteel 8y ago

This is the first thing I have read about information theory that really made sense to me. I think I now understand it about ten times as well as I did before. >What does information really measure? It measures the uncertainty we overcome My thought here is that matters because human beings have to make decisions all the time, and reducing the uncertainty regarding the situation makes it more likely you will make the correct decision.

hyperpallium 8y ago
> The signicant aspect is that the actual message is one selected from a set of possible messages > If the number of messages in the set is finite then this number [...] can be regarded as a measure of the information produced when one message is chosen from the set, all choices being equally likely. From the abstract of his famous paper http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...
- bhnmmhmd 8y ago
  It's interesting how this paper was first titled "A mathematical theory of communication" and then renamed to "THE mathematical ...".
yantrams 8y ago
Here is a great primer on entropy that you might find helpful - http://colah.github.io/posts/2015-09-Visual-Information/ http://colah.github.io/posts/2015-09-Visual-Information/
lisper 8y ago
There's a pithy slogan that sums it up nicely: information is a correlation between the state of two systems. For example: if you measure a voltage using a volt-meter then, if the volt-meter is working properly, there will be a correlation between the readout on the meter and the actual voltage, and so the meter contains information about the voltage. Note that information is symmetric: the voltage in this case also contains information about the meter.
- jonnydubowsky 8y ago
  Your comment and the one above it are awesome summaries of some of Shannon's concepts. Re:your statement on information being symmetrical, how does zero knowledge encryption, Homomorphic encryption and the like fit into this when there is asymmetrical knowledge and validation without full access to the information?

mkagenius 8y ago

If anyone from Nautilus is here: Would you consider having different prices for different countries? The current pricing is too high for India. A reduced price would bring customers from various countries.

criddell 8y ago
A lifetime subscription is $99. If you factor in life expectancy, they are actually charging more to people in India than they are to Americans.
- Anon84 8y ago
  Yes, if you look at raw dollars per year... but you also have to factor in expected lifetime earnings. What fraction of your expected income is $99? I would wager that it's a much higher fraction in India than in the US.

flopska 8y ago

I recently read the Shannon biography 'A Mind at Play' [0] which I guess is the full version of the article (same authors). It does (IMHO) a very good job at explaining entropy / information. [0] https://www.simonandschuster.com/books/A-Mind-at-Play/Jimmy-Soni/9781476766690 https://www.simonandschuster.com/books/A-Mind-at-Play/Jimmy-...

goldfeld 8y ago
I enjoyed the story but found the author(s) practically peddling Shannon's importance as if any of that was needed. All in all the writing in form and substance seemed to me of low quality and in search of dramatic revision and editing. Great history job and lazy writing,
- woodandsteel 8y ago
  >I enjoyed the story but found the author(s) practically peddling Shannon's importance as if any of that was needed. The vast majority of the public have never heard of Shannon, or have but don't really understand how important he was, so yes, it was needed.