4 ms·

Journey into Information Theory

79 points by gshrikant 11y ago

murbard2 11y ago
"History of the alphebet" [sic] Not mentioned is the curious fact that there seems to have been only one independent discovery of the alphabet. All alphabets are descended from the same root, proto-sinaitic or have been made up by people who knew about alphabets (Korean). Irish runes and Khmer script thus share a common ancestor.
gshrikant 11y ago
I found Hartley's paper [1] (also discussed in the video sequence) on quantifying information surprisingly accessible and a good read in general. [1] http://www.dotrose.com/etext/90_Miscellaneous/transmission_of_information_1928b.pdf http://www.dotrose.com/etext/90_Miscellaneous/transmission_o...
anirudt 11y ago
Elements of Information Theory, by Thomas Cover is also a very good read for fundamentals.
- stared 11y ago
  By J. Thomas AND Thomas Cover. Even worse combination than Stephan-Boltzmann law or Albert-Barabasi networks. The book itself is great, IMHO the best intro to information theory and entropy.
yen223 11y ago
Information Theory is one of my most favourite courses that I took at uni. The idea that you can take something as abstract as "information", and quantify it in a useful manner was eye-opening.
- bandrami 11y ago
  Same here. Particularly the idea that "information" and "entropy" are the same quantity. That was an eye-opener.
beagle3 11y ago
Shannon's original paper, titled "A Mathematical Theory of Communication"[0] which basically founded Information Theory as its own coherent and well defined field is also very readable. Some of the concepts were well known before he published it, some were original -- but the coherent formulation, notation and concentration in that paper is what established the field now known as "Information Theory". [0] http://worrydream.com/refs/Shannon%20-%20A%20Mathematical%20Theory%20of%20Communication.pdf http://worrydream.com/refs/Shannon%20-%20A%20Mathematical%20...
- gshrikant 11y ago
  Thanks for pointing me to http://worrydream.com/refs http://worrydream.com/refs. It is a treasure trove of wonderful writings!
mrcactu5 11y ago
I think it's insane that Khan Academy should attempt to cover information theory giving the Coursera course has a 1% passing rate. I am two videos in, biting my lip to not make technical comments that don't matter at this level. They do a great job of simplifying a very complex subject.

79 points by gshrikant 11y ago

"History of the alphebet" [sic] Not mentioned is the curious fact that there seems to have been only one independent discovery of the alphabet. All alphabets are descended from the same root, proto-sinaitic or have been made up by people who knew about alphabets (Korean). Irish runes and Khmer script thus share a common ancestor.

gshrikant 11y ago

I found Hartley's paper [1] (also discussed in the video sequence) on quantifying information surprisingly accessible and a good read in general. [1] http://www.dotrose.com/etext/90_Miscellaneous/transmission_of_information_1928b.pdf http://www.dotrose.com/etext/90_Miscellaneous/transmission_o...

anirudt 11y ago

Elements of Information Theory, by Thomas Cover is also a very good read for fundamentals.

stared 11y ago
By J. Thomas AND Thomas Cover. Even worse combination than Stephan-Boltzmann law or Albert-Barabasi networks. The book itself is great, IMHO the best intro to information theory and entropy.

yen223 11y ago

Information Theory is one of my most favourite courses that I took at uni. The idea that you can take something as abstract as "information", and quantify it in a useful manner was eye-opening.

bandrami 11y ago
Same here. Particularly the idea that "information" and "entropy" are the same quantity. That was an eye-opener.

beagle3 11y ago

Shannon's original paper, titled "A Mathematical Theory of Communication"[0] which basically founded Information Theory as its own coherent and well defined field is also very readable. Some of the concepts were well known before he published it, some were original -- but the coherent formulation, notation and concentration in that paper is what established the field now known as "Information Theory". [0] http://worrydream.com/refs/Shannon%20-%20A%20Mathematical%20Theory%20of%20Communication.pdf http://worrydream.com/refs/Shannon%20-%20A%20Mathematical%20...

gshrikant 11y ago
Thanks for pointing me to http://worrydream.com/refs http://worrydream.com/refs. It is a treasure trove of wonderful writings!

mrcactu5 11y ago

I think it's insane that Khan Academy should attempt to cover information theory giving the Coursera course has a 1% passing rate. I am two videos in, biting my lip to not make technical comments that don't matter at this level. They do a great job of simplifying a very complex subject.