31 ms·

If anyone is on the fence about reading this, or worried about their ability to comprehend the content, I would tell you to go ahead and give it a chance. Shann

If anyone is on the fence about reading this, or worried about their ability to comprehend the content, I would tell you to go ahead and give it a chance. Shannon's writing is remarkably lucid and transparent. The jargon is minimal, and his exposition is fantastic. As many other commentators has mentioned, it is impressive that such an approachable paper would lay the foundations for a whole field. I actually find that many subsequent textbooks seem to obfuscate the simplicity of the idea of entropy. Two examples from the paper really stuck with me. In one, he discusses the importance of spaces for encoding language, something which I had never really considered before. In the second, he discusses how it is the redundancy of language that allows for crosswords, and that a less redundant language would make it harder to design these (unless we started making them 3D!). It made me think more deeply about communication as a whole.

jp42 2y ago
I am one of the few who is on the fence. This comment motivates me to give it try to this paper. Thanks ImageXav!
jessriedel 2y ago
Agreed. This is one of the all time great papers in that it both launched an entire field (information theory) and remains very accessible and pedagogical. A true gem.
Anon84 2y ago
He also creates the first (at least that I could find) instance of a auto-regressive (markovian) language model as a clarifying example in the first 10 pages :)
contingencies 2y ago
> Two examples from the paper really stuck with me. In one, he discusses the importance of spaces for encoding language, something which I had never really considered before. As a westerner who has studied quite a few writing systems this is kind of hard to interpret. Verbally however, the timing of pauses are important in all languages I've learned. This would be a more coherent argument to place at the pan-lingual level than one related to written representation, which is pretty arbitrary (many languages have migrated scripts over the years, see for example the dual mode devanagari/arabic hindu/urdu divide, many other languages migrating to arabic, phagspa, vietnamese moving from chinese to french diacritics, etc.). > In the second, he discusses how it is the redundancy of language that allows for crosswords, and that a less redundant language would make it harder to design these (unless we started making them 3D!). It made me think more deeply about communication as a whole. Yeah, good luck making a Chinese crossword. Not sure "redundancy" is the right term, however. Perhaps "frequent [even tediously repetitive?] glyph reuse".
kragen 2y ago
my experience is that papers that lay the foundations for a whole field are usually very approachable. i'm not sure why this is: - maybe being better at breaking new intellectual ground requires some kind of ability that can also be applied to explaining things? like maybe some people are just smarter than others, either inherently or as a result of their training and experience, in a way that generalizes to both tasks - maybe the things that most strongly impede people from breaking new intellectual ground also impede them from explaining them clearly? candidates might include emotional insecurity, unthinking devotion to tradition, and intellectual vanity (wanting to look right rather than be right) - maybe the people who suck at explaining their own ideas don't get access to the cutting-edge developments that they would need to break new intellectual ground? shannon had the great good fortune, for example, to spend a lot of the war at bell labs conducting cryptanalysis, rather than sleeping under a dunghill on the battlefield or on the assembly line making artillery shells
- jdewerd 2y ago
  Jargon and shared context are barriers for newbies. In a new field they simply don't exist yet. The avenues for accidentally excluding people (or intentionally but I like to be charitable) don't exist yet.
  - kragen 2y ago
    this is an excellent point, and in retrospect obvious
- akvadrako 2y ago
  I think it's because they are older and that led to less of the modern publication pressures. You didn't need to publish if you didn't have something interesting to say and you could just make your point, without worrying about drowning in a sea of mediocrity.
  - kragen 2y ago
    i thought about that, and certainly i have read a lot of terribly written papers (twps) in recent years, but i think this is partly survival bias; there were lots of twps in the older literature (ol) too, but they don't get cited, so you have to do things like find an entire ol journal issue (maybe one containing a single well-known paper) to read through and find twps. but the well-written papers from the ol seem to be much, much better written than the current well-written papers. i think something about the current publishing pipeline acts as a filter against good writing, something that didn't use to be there
teleforce 2y ago
Shannon is the closest equivalent to Einstein in contribution but for engineering fields. He pioneered several foundational research in the engineering field including communication entropy, cryptography, chess engine, robotic intelligence, digital boolean, LLM and modern AI in general. An outstanding engineer, or engineer's engineer in a true sense of word.
keepamovin 2y ago
I think it’s worth saying, thank you for such a great description of why Shannon’s writing is good, and how approachable his foundational papers are. I could’ve just upboted but it’s nice to know that what you’re doing really resonates with other people and they really appreciate your way of describing it. Thank you. Haha! :)
baq 2y ago
> The jargon is minimal Just pointing out the obvious here - it’s impossible to have any jargon in this paper since it literally created the field. Any jargon is necessarily invented later. On the whole I agree: just go read the paper. While you’re at it, queue up Lamport’s The Part-Time Parliament of Paxos.
- a_wild_dandan 2y ago
  new jargon != any jargon Plenty of STEM jargon existed which Shannon chose to not use.

jp42 2y ago

I am one of the few who is on the fence. This comment motivates me to give it try to this paper. Thanks ImageXav!

jessriedel 2y ago

Agreed. This is one of the all time great papers in that it both launched an entire field (information theory) and remains very accessible and pedagogical. A true gem.

Anon84 2y ago

He also creates the first (at least that I could find) instance of a auto-regressive (markovian) language model as a clarifying example in the first 10 pages :)

contingencies 2y ago

> Two examples from the paper really stuck with me. In one, he discusses the importance of spaces for encoding language, something which I had never really considered before. As a westerner who has studied quite a few writing systems this is kind of hard to interpret. Verbally however, the timing of pauses are important in all languages I've learned. This would be a more coherent argument to place at the pan-lingual level than one related to written representation, which is pretty arbitrary (many languages have migrated scripts over the years, see for example the dual mode devanagari/arabic hindu/urdu divide, many other languages migrating to arabic, phagspa, vietnamese moving from chinese to french diacritics, etc.). > In the second, he discusses how it is the redundancy of language that allows for crosswords, and that a less redundant language would make it harder to design these (unless we started making them 3D!). It made me think more deeply about communication as a whole. Yeah, good luck making a Chinese crossword. Not sure "redundancy" is the right term, however. Perhaps "frequent [even tediously repetitive?] glyph reuse".

kragen 2y ago

my experience is that papers that lay the foundations for a whole field are usually very approachable. i'm not sure why this is: - maybe being better at breaking new intellectual ground requires some kind of ability that can also be applied to explaining things? like maybe some people are just smarter than others, either inherently or as a result of their training and experience, in a way that generalizes to both tasks - maybe the things that most strongly impede people from breaking new intellectual ground also impede them from explaining them clearly? candidates might include emotional insecurity, unthinking devotion to tradition, and intellectual vanity (wanting to look right rather than be right) - maybe the people who suck at explaining their own ideas don't get access to the cutting-edge developments that they would need to break new intellectual ground? shannon had the great good fortune, for example, to spend a lot of the war at bell labs conducting cryptanalysis, rather than sleeping under a dunghill on the battlefield or on the assembly line making artillery shells

jdewerd 2y ago
Jargon and shared context are barriers for newbies. In a new field they simply don't exist yet. The avenues for accidentally excluding people (or intentionally but I like to be charitable) don't exist yet.
- kragen 2y ago
  this is an excellent point, and in retrospect obvious
akvadrako 2y ago
I think it's because they are older and that led to less of the modern publication pressures. You didn't need to publish if you didn't have something interesting to say and you could just make your point, without worrying about drowning in a sea of mediocrity.
- kragen 2y ago
  i thought about that, and certainly i have read a lot of terribly written papers (twps) in recent years, but i think this is partly survival bias; there were lots of twps in the older literature (ol) too, but they don't get cited, so you have to do things like find an entire ol journal issue (maybe one containing a single well-known paper) to read through and find twps. but the well-written papers from the ol seem to be much, much better written than the current well-written papers. i think something about the current publishing pipeline acts as a filter against good writing, something that didn't use to be there

teleforce 2y ago

Shannon is the closest equivalent to Einstein in contribution but for engineering fields. He pioneered several foundational research in the engineering field including communication entropy, cryptography, chess engine, robotic intelligence, digital boolean, LLM and modern AI in general. An outstanding engineer, or engineer's engineer in a true sense of word.

keepamovin 2y ago

I think it’s worth saying, thank you for such a great description of why Shannon’s writing is good, and how approachable his foundational papers are. I could’ve just upboted but it’s nice to know that what you’re doing really resonates with other people and they really appreciate your way of describing it. Thank you. Haha! :)

baq 2y ago

> The jargon is minimal Just pointing out the obvious here - it’s impossible to have any jargon in this paper since it literally created the field. Any jargon is necessarily invented later. On the whole I agree: just go read the paper. While you’re at it, queue up Lamport’s The Part-Time Parliament of Paxos.

a_wild_dandan 2y ago
new jargon != any jargon Plenty of STEM jargon existed which Shannon chose to not use.