4 ms·

Information Theory for Intelligent People (2018) [pdf]

451 points by oska 7y ago

jbhatab 7y ago
I'm a pretty confused on how this all ties together. Are these these deep dives on examples on information theory? Would love some perspective that knows more on the topic.
- boyobo 7y ago
  It's an introduction to the fundamental concepts of information theory (entropy, KL divergence, conditional entropy), using 20 questions as an ongoing source of example applications and interpretations of these concepts. The author also sprinkles in various interpretations coming from their own research focus (the social sciences). Definitely not a deep dive. The details can probably be found by chasing down the citations.
  - jbhatab 7y ago
    Awesome thanks so much
Ericson2314 7y ago
Why the title?
- throwawayjava 7y ago
  I think it's meant in a humble way rather than a self-important way. "Here's a treatment that will be useful for any smart-enough person [even if they aren't a phd in math/theoretical CS]". Perhaps a better title could be "for Any Intelligent Person".
  - js8 7y ago
    I had a similar feeling as OP at first, and then I realized the same as you. I would personally suggest a title "for curious people".
  - llamaz 7y ago
    I thought it was ironic, because it starts out almost like a children's story and is very approachable. I'd rather read this than a textbook, that's for sure.
- jldugger 7y ago
  You know the book series 'Computers for Dummies?' It's a play on that. See also Zytrax's 'X for Rocket Scientists' which covers complicated stuff like LDAP and DNS and explains why they are the way they are.
  - dredmorbius 7y ago
    Joe Celko's SQL for Smarties (https://www.oreilly.com/library/view/joe-celkos-sql/9780128007617/ https://www.oreilly.com/library/view/joe-celkos-sql/97801280...) and "Learn X the Hard Way" (Python, Java, C/C++, etc.) also play on the "for idiots/dummies" motif.
  - Myrmornis 7y ago
    I didn't even make the "for dummies" connection! The name is a huge fail but it looks like the author meant it well, perhaps someone should tell him that it comes across as intellectual elitism and exclusionary to people who perhaps feel a little insecure reading material on the internet that is associated with fancy universities and graduate schools.
jonmc12 7y ago
For another perspective of "the study of information in the social world", check out "Variational ecology and the physics of sentient systems" (https://www.sciencedirect.com/science/article/pii/S157106451930003X https://www.sciencedirect.com/science/article/pii/S157106451...). Also based on information theory, released just a few months after op paper.
hos234 7y ago
Why do they use the word Release? Release of entropy suddenly drops into the picture when talking abt ice to water. Why not just say info/entropy/uncertainty of the system is increasing or decreasing? This subject is new to me but I like the 20 questions analogy. Thinking of probability distribution of N words as a measure of info in someone's mind is interesting. Disciplined minds that have certain concepts on their finger tips clearly have spent time and effort producing that distribution.
- n4r9 7y ago
  True, it does seem a bit weird to use "release" in reference to entropy. I've googled around a bit and can't find a similar phrasing anywhere. A lot of places talk about "energy" being released into the surroundings, and this article talks about using the entropy of the surroundings to predict whether or not ice will melt: http://www.4college.co.uk/a/O/entsurr.php http://www.4college.co.uk/a/O/entsurr.php .
mac01021 7y ago
The original exposition by Claude Shannon, though long, is one of the most readable mathematical publications I've ever encountered. It's hard to imagine anyone improving on it. http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...
Vaslo 7y ago
This is the opposite of pdf articles on HN. Many are titled with something like "beginners" or "for everyone", then I open up a Latex formatted document that is impenetrable from the first page using math symbols I have never seen before. this is the opposite, much easier to understand and not loaded up with mathematics symbol vomit all over the page.
simondedeo 7y ago
Super-pleased to see the attention to this little piece. It gets updated from time to time, and it's always useful to see comments. I appreciate it. Some responses: 1. hos234 asks "Why do they use the word Release? Release of entropy suddenly drops into the picture when talking abt ice to water. Why not just say info/entropy/uncertainty of the system is increasing or decreasing?" It can be useful (or, at the very least, fun) to think of uncertainty as a fluid that moves through a physical system. Imagine an air-conditioner, for example; it is doing work, but not changing the overall energy of the system (at least in approximation and if it is working very slowly). It's just separating out particles. The more advanced version is Maxwell's Demon; Rolf Landauer at IBM did the full analysis of this system, which you can think of as creating useful energy by "consuming" a pristine hard-drive (or, equivalently, dumping entropy into it.) http://www.w2agz.com/Library/Limits%20of%20Computation/Landauer%20Article,%20Physics%20Today%2044,%205,%2023%20(1991).pdf http://www.w2agz.com/Library/Limits%20of%20Computation/Landa... or https://www.sciencedirect.com/science/article/pii/0375960196004537 https://www.sciencedirect.com/science/article/pii/0375960196... 2. I called it "for intelligent people" because I wanted to emphasize that just because you don't know what information theory is, or are not particularly mathy, it doesn't mean the author (me) thinks you're dumb. The hope was to counteract the usual physics-style arrogance that xkcd captures in "why does your field need a journal?" https://xkcd.com/793/ https://xkcd.com/793/ And yes, per jldugger, it was a play on the "for Dummies" series. Ironically for me, the humanities people that it was aimed at often took it amiss. Dan Dennett told me as much. throwawayjava thank you for the suggestion; I'll use it in a footnote in the next iteration! 3. if you liked "information theory for intelligent people", you'll LOVE "Bayesian reasoning for intelligent people", the shocking prequel featuring characters that will ruin everything for you and plus be mildly racist? http://santafe.edu/~simon/br.pdf http://santafe.edu/~simon/br.pdf
- boyobo 7y ago
  Ironically for me, the humanities people that it was aimed at often took it amiss. Dan Dennett told me as much. throwawayjava thank you for the suggestion; I'll use it in a footnote in the next iteration! Yes, I thought this article was a very well written introduction, but I held back on sharing this link with some non-math friends because I didn't want them to feel dumb if they find it difficult to get through.
oarabbus_ 7y ago
Hmm, Information Theory has always done an excellent job of making me feel non-intelligent, so not sure how applicable this PDF is to me.
shitgoose 7y ago
try this: https://colah.github.io/posts/2015-09-Visual-Information/ https://colah.github.io/posts/2015-09-Visual-Information/ his blog is pretty cool. he is original.
- rramadass 7y ago
  Nice Article! Much reading to do on his blog :-)

451 points by oska 7y ago

I'm a pretty confused on how this all ties together. Are these these deep dives on examples on information theory? Would love some perspective that knows more on the topic.

boyobo 7y ago
It's an introduction to the fundamental concepts of information theory (entropy, KL divergence, conditional entropy), using 20 questions as an ongoing source of example applications and interpretations of these concepts. The author also sprinkles in various interpretations coming from their own research focus (the social sciences). Definitely not a deep dive. The details can probably be found by chasing down the citations.
- jbhatab 7y ago
  Awesome thanks so much

Ericson2314 7y ago

Why the title?

throwawayjava 7y ago
I think it's meant in a humble way rather than a self-important way. "Here's a treatment that will be useful for any smart-enough person [even if they aren't a phd in math/theoretical CS]". Perhaps a better title could be "for Any Intelligent Person".
- js8 7y ago
  I had a similar feeling as OP at first, and then I realized the same as you. I would personally suggest a title "for curious people".
- llamaz 7y ago
  I thought it was ironic, because it starts out almost like a children's story and is very approachable. I'd rather read this than a textbook, that's for sure.
jldugger 7y ago
You know the book series 'Computers for Dummies?' It's a play on that. See also Zytrax's 'X for Rocket Scientists' which covers complicated stuff like LDAP and DNS and explains why they are the way they are.
- dredmorbius 7y ago
  Joe Celko's SQL for Smarties (https://www.oreilly.com/library/view/joe-celkos-sql/9780128007617/ https://www.oreilly.com/library/view/joe-celkos-sql/97801280...) and "Learn X the Hard Way" (Python, Java, C/C++, etc.) also play on the "for idiots/dummies" motif.
- Myrmornis 7y ago
  I didn't even make the "for dummies" connection! The name is a huge fail but it looks like the author meant it well, perhaps someone should tell him that it comes across as intellectual elitism and exclusionary to people who perhaps feel a little insecure reading material on the internet that is associated with fancy universities and graduate schools.

jonmc12 7y ago

For another perspective of "the study of information in the social world", check out "Variational ecology and the physics of sentient systems" (https://www.sciencedirect.com/science/article/pii/S157106451930003X https://www.sciencedirect.com/science/article/pii/S157106451...). Also based on information theory, released just a few months after op paper.

hos234 7y ago

Why do they use the word Release? Release of entropy suddenly drops into the picture when talking abt ice to water. Why not just say info/entropy/uncertainty of the system is increasing or decreasing? This subject is new to me but I like the 20 questions analogy. Thinking of probability distribution of N words as a measure of info in someone's mind is interesting. Disciplined minds that have certain concepts on their finger tips clearly have spent time and effort producing that distribution.

n4r9 7y ago
True, it does seem a bit weird to use "release" in reference to entropy. I've googled around a bit and can't find a similar phrasing anywhere. A lot of places talk about "energy" being released into the surroundings, and this article talks about using the entropy of the surroundings to predict whether or not ice will melt: http://www.4college.co.uk/a/O/entsurr.php http://www.4college.co.uk/a/O/entsurr.php .

mac01021 7y ago

The original exposition by Claude Shannon, though long, is one of the most readable mathematical publications I've ever encountered. It's hard to imagine anyone improving on it. http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...

Vaslo 7y ago

This is the opposite of pdf articles on HN. Many are titled with something like "beginners" or "for everyone", then I open up a Latex formatted document that is impenetrable from the first page using math symbols I have never seen before. this is the opposite, much easier to understand and not loaded up with mathematics symbol vomit all over the page.

simondedeo 7y ago

Super-pleased to see the attention to this little piece. It gets updated from time to time, and it's always useful to see comments. I appreciate it. Some responses: 1. hos234 asks "Why do they use the word Release? Release of entropy suddenly drops into the picture when talking abt ice to water. Why not just say info/entropy/uncertainty of the system is increasing or decreasing?" It can be useful (or, at the very least, fun) to think of uncertainty as a fluid that moves through a physical system. Imagine an air-conditioner, for example; it is doing work, but not changing the overall energy of the system (at least in approximation and if it is working very slowly). It's just separating out particles. The more advanced version is Maxwell's Demon; Rolf Landauer at IBM did the full analysis of this system, which you can think of as creating useful energy by "consuming" a pristine hard-drive (or, equivalently, dumping entropy into it.) http://www.w2agz.com/Library/Limits%20of%20Computation/Landauer%20Article,%20Physics%20Today%2044,%205,%2023%20(1991).pdf http://www.w2agz.com/Library/Limits%20of%20Computation/Landa... or https://www.sciencedirect.com/science/article/pii/0375960196004537 https://www.sciencedirect.com/science/article/pii/0375960196... 2. I called it "for intelligent people" because I wanted to emphasize that just because you don't know what information theory is, or are not particularly mathy, it doesn't mean the author (me) thinks you're dumb. The hope was to counteract the usual physics-style arrogance that xkcd captures in "why does your field need a journal?" https://xkcd.com/793/ https://xkcd.com/793/ And yes, per jldugger, it was a play on the "for Dummies" series. Ironically for me, the humanities people that it was aimed at often took it amiss. Dan Dennett told me as much. throwawayjava thank you for the suggestion; I'll use it in a footnote in the next iteration! 3. if you liked "information theory for intelligent people", you'll LOVE "Bayesian reasoning for intelligent people", the shocking prequel featuring characters that will ruin everything for you and plus be mildly racist? http://santafe.edu/~simon/br.pdf http://santafe.edu/~simon/br.pdf

boyobo 7y ago
Ironically for me, the humanities people that it was aimed at often took it amiss. Dan Dennett told me as much. throwawayjava thank you for the suggestion; I'll use it in a footnote in the next iteration! Yes, I thought this article was a very well written introduction, but I held back on sharing this link with some non-math friends because I didn't want them to feel dumb if they find it difficult to get through.

oarabbus_ 7y ago

Hmm, Information Theory has always done an excellent job of making me feel non-intelligent, so not sure how applicable this PDF is to me.

shitgoose 7y ago

try this: https://colah.github.io/posts/2015-09-Visual-Information/ https://colah.github.io/posts/2015-09-Visual-Information/ his blog is pretty cool. he is original.

rramadass 7y ago
Nice Article! Much reading to do on his blog :-)