13 ms·

Claude Shannon: Tinkerer, Prankster, and Father of Information Theory (2016)

178 points by phonebucket 5y ago

noneeeed 5y ago
Funny timing. Tim Harford's Cautionary Tales podcast had an episode about Shannon. He sounds like a fun guy to have been around. https://timharford.com/2021/05/cautionary-tales-fritterin-away-genius/ https://timharford.com/2021/05/cautionary-tales-fritterin-aw...
- phonebucket 5y ago
  The timing is a little less funny if you know that I listened to the podcast shortly before finding the IEEE article. The podcast was great! I would think that many on HN would find it interesting.
  - doublejay1999 5y ago
    There's a passable movie called "the bit player" currently on amazon and I guess elsewhere. I'd never heard of him, and that's embarrassing, but he probably prefers it that way :-).
  - vaughandroid 5y ago
    Cautionary Tales is my favourite podcast. Most episodes concern either interesting (mis)use of statistics or cognitive biases, both of which are topics I find inherently interesting, but Tim Harford and team do a brilliant job of the storytelling too. Tim Harford's other podcasts are great too - More or Less, and 50 Things Which Made the Modern Economy. Does anyone have any recommendations along similar lines?
    - anitil 5y ago
      He also hosted 'How to Vaccinate the World', and interviewed some heavy-hitters like Bill Gates. The series is over now but was very interesting.
- anitil 5y ago
  It was so endearing to head about Shannon spending his time as an illustrious academic learning to juggle, then unicycle, then both at the same time.
op03 5y ago
From what I understand, before Shannon showed up no one knew how to calculate the Channel Capacity of a medium and how to deal with noise in the medium. So people just yelled louder or yelled repeatedly to get things across with better reliability. And when channel capacity was exceeded with everyone yelling too much, no one knew it had exceeded and lot of energy and time was wasted with increasing errors in the system. Doesn't that feel like a repeat with whats happening in social media and news media these days? Or is it just different things.
- rusk 5y ago
  I think it’s a different thing but an interesting analog. In conventional information theory you can add another channel. I guess the social media metaphor is like crowded wifi or if I’m being clever alohanet... question is, how do we add another channel here?if you’re talking about social discourse rather than a physical medium are we talking about the necessity of class hierarchy? Is this what the Tower of Babel was about?
- TeMPOraL 5y ago
  > Doesn't that feel like a repeat with whats happening in social media and news media these days? Or is it just different things. Maybe. It can feel like this, but applying mathematical models to social phenomena is tricky. You have to map formal parameters to fuzzy, poorly understood factors. How do you define, precisely, what is a "channel" on a social network? What is "signal", and what is "noise"? People can and do prove anything by using slightly different (or inconsistent) definitions here, getting the numbers to line up just like they want them to. My understanding is that a better way to approach such mapping would be to shove in probability distributions in place of hard-to-map exact parameters - abstracting away choices and measurements lets you see the wider context here. This pushes the problem to defining the appropriate distributions, but I think that's more tamper-proof. Unfortunately, the results may come out next to useless - e.g. probability distributions so wide you could sail a carrier strike group through them. I'd love to know what's considered the correct, robust approach to such problems.
  - sig-reduction 5y ago
nathias 5y ago
'shannonigans'
llaolleh 5y ago
Shannon was awesome. It seemed like after he wrote his groundbreaking information theory paper he did whatever the hell he wanted for the rest of his life in Cambridge. Made all kinds of contraptions and devices.
Shatnerz 5y ago
I recommend reading "A Mind at Play", a biography about Shannon. I found it to be a decent read. Funnily enough, I knew about him more through the unicycling community than I did through computer science. It wasn't until a few years ago when I realised what sort of impact he had on the world.
- idatum 5y ago
  I also enjoyed "A Mind at Play". I had forgotten (or never fully appreciated) what Shannon did and what other early pioneers did in computing. I have a better appreciation for all of them now. One other highlight was Shannon's early childhood and using electrified fencing to communicate with others in his farmland neighborhood.
ShaneCurran 5y ago
Many know Shannon as the author of "A Mathematical Theory of Communication" and the founder of information theory, as well as the brain behind "Communication Theory of Secrecy Systems" Both papers derive from a technical report, "A Mathematical Theory of Cryptography"[0], written by Shannon in 1945 — 3 years before his breakthrough papers in the Bell Labs Technical Journal. [0]: https://evervault.com/papers/shannon.pdf https://evervault.com/papers/shannon.pdf
lqet 5y ago
> In 1973, he recalled, he persuaded Shannon to give the first annual Shannon lecture at the International Information Theory Symposium, but Shannon almost backed out at the last minute. “I never saw a guy with so much stage fright,” Berlekamp said. “In this crowd, he was viewed as a godlike figure, and I guess he was worried he wouldn’t live up to his reputation.” > In 1985 he made an unexpected appearance at the International Information Theory Symposium in Brighton, England. The meeting was proceeding smoothly, if uneventfully, when news raced through the halls and lecture rooms that the snowy-haired man with the shy grin who was wandering in and out of the sessions was none other than Claude Shannon. Some of those at the conference had not even known he was still alive. > At the banquet, the meeting’s organizers somehow persuaded Shannon to address the audience. He spoke for a few minutes and then—fearing that he was boring his audience, he recalled later—pulled three balls out of his pockets and began juggling. The audience cheered and lined up for autographs.
- harveywi 5y ago
  > At the banquet, the meeting’s organizers somehow persuaded Shannon to address the audience. He spoke for a few minutes and then—fearing that he was boring his audience, he recalled later—pulled three balls out of his pockets and began juggling. Shannon was famous for his entropy.
  - 1cvmask 5y ago
    The alleged origins of the naming of entropy: When Shannon first derived his famous formula for information, he asked von Neumann what he should call it and von Neumann replied “You should call it entropy for two reasons: first because that is what the formula is in statistical mechanises but second and more important, as nobody knows what entropy is, whenever you use the term you will always be at an advantage! From: http://www.spatialcomplexity.info/what-von-neumann-said-to-shannon/ http://www.spatialcomplexity.info/what-von-neumann-said-to-s...
    - 2snakes 5y ago
      Yeah, I would characterize it as a mathematical theory of -symbols- and that information has different definitions. However, the argument can be made that symbolic representation is a principle of reality.
tekstar 5y ago
A really good book is "The Idea Factory" by Jon Gertner. It describes Bell Labs in this period, and through the book basically tells the story of where modern life was born: everything from growing crystals to transistors to cell phones to radar was created at Bell Labs.
ansible 5y ago
Curiosity Stream has a recent video centering around a fictionalized interview with Claude Shannon: https://curiositystream.com/video/3831 https://curiositystream.com/video/3831 If you sign up through a YouTube promotional link, you also get access to Nebula for free: https://curiositystream.com/legaleagle/index.html https://curiositystream.com/legaleagle/index.html
- Wistar 5y ago
  Here's a very so-so promo site for The Bit Player. I saw it about a year ago (Curiosity Stream) and thought it was pretty decent. I somewhat sheepishly admit that, for a while, I thought that they had somehow interviewed the real Claude Shannon. https://thebitplayer.com/ https://thebitplayer.com/ Although not devoted to Claude Shannon, Jim Al-Khalili's documentary "The Story of Information" does discuss Shannon's significant contributions and is how I first became aware of Claude Shannon. https://topdocumentaryfilms.com/story-information/ https://topdocumentaryfilms.com/story-information/
sillysaurusx 5y ago
Shannon was cursed by being the inventor of information theory. It seems like this wonderful thing; who wouldn’t want to invent information theory? But as an inventor, you’ll never top it. You’ll forever live in your own shadow. And for someone that enjoys inventing things, that’s... not a happy thing.
- mycologos 5y ago
  Nothing I've read about Shannon suggests that he felt "cursed" in his later life. Is there some reason to think otherwise, or is this just assuming that anybody who does something big early feels sad about it later?
  - jecel 5y ago
    In Shannon's case, given how fundamental his 1937 thesis was (you can use Boolean Algebra to design digital circuits), most other people might have been afraid of not being able to top that. My impression was that he was always looking forward to bigger things through his own work and through his students. https://en.wikipedia.org/wiki/A_Symbolic_Analysis_of_Relay_and_Switching_Circuits https://en.wikipedia.org/wiki/A_Symbolic_Analysis_of_Relay_a...
vieclamdailoan 5y ago
realy?
legohead 5y ago
another recommended book: The Idea Factory: Bell Labs and the great age of american innovation. Includes a lot of interesting information, including Claude Shannon of course, who worked at Bell Labs. Bell Labs was a remarkable place.
JoeDaDude 5y ago
Claude Shannon was also an early pioneer in artificial intelligence, the minimax algorithm applied to game trees was one of his inventions. He built several machines to play games and demonstrate his ideas, some of which can be seen in the article photos. I compiled a list of his game gadgets, most of which still exist in the care of the MIT museum. https://boardgamegeek.com/geeklist/143233/claude-shannon-man-games-and-machines https://boardgamegeek.com/geeklist/143233/claude-shannon-man...
krosaen 5y ago
An interesting tidbit from Hamming in The Art of Doing Science and Engineering [1][2] on the name "Information Theory" > Information Theory was created by C.E.Shannon in the late 1940s. The management of Bell Telephone Labs wanted him to call it “Communication Theory” as that is a far more accurate name, but for obvious publicity reasons “Information Theory” has a much greater impact—thus Shannon chose and so it is known to this day. [1] https://press.stripe.com/#the-art-of-doing-science-and-engineering https://press.stripe.com/#the-art-of-doing-science-and-engin... [2] http://worrydream.com/refs/Hamming-TheArtOfDoingScienceAndEngineering.pdf http://worrydream.com/refs/Hamming-TheArtOfDoingScienceAndEn...

178 points by phonebucket 5y ago

Funny timing. Tim Harford's Cautionary Tales podcast had an episode about Shannon. He sounds like a fun guy to have been around. https://timharford.com/2021/05/cautionary-tales-fritterin-away-genius/ https://timharford.com/2021/05/cautionary-tales-fritterin-aw...

phonebucket 5y ago
The timing is a little less funny if you know that I listened to the podcast shortly before finding the IEEE article. The podcast was great! I would think that many on HN would find it interesting.
- doublejay1999 5y ago
  There's a passable movie called "the bit player" currently on amazon and I guess elsewhere. I'd never heard of him, and that's embarrassing, but he probably prefers it that way :-).
- vaughandroid 5y ago
  Cautionary Tales is my favourite podcast. Most episodes concern either interesting (mis)use of statistics or cognitive biases, both of which are topics I find inherently interesting, but Tim Harford and team do a brilliant job of the storytelling too. Tim Harford's other podcasts are great too - More or Less, and 50 Things Which Made the Modern Economy. Does anyone have any recommendations along similar lines?
  - anitil 5y ago
    He also hosted 'How to Vaccinate the World', and interviewed some heavy-hitters like Bill Gates. The series is over now but was very interesting.
anitil 5y ago
It was so endearing to head about Shannon spending his time as an illustrious academic learning to juggle, then unicycle, then both at the same time.

op03 5y ago

From what I understand, before Shannon showed up no one knew how to calculate the Channel Capacity of a medium and how to deal with noise in the medium. So people just yelled louder or yelled repeatedly to get things across with better reliability. And when channel capacity was exceeded with everyone yelling too much, no one knew it had exceeded and lot of energy and time was wasted with increasing errors in the system. Doesn't that feel like a repeat with whats happening in social media and news media these days? Or is it just different things.

rusk 5y ago
I think it’s a different thing but an interesting analog. In conventional information theory you can add another channel. I guess the social media metaphor is like crowded wifi or if I’m being clever alohanet... question is, how do we add another channel here?if you’re talking about social discourse rather than a physical medium are we talking about the necessity of class hierarchy? Is this what the Tower of Babel was about?
TeMPOraL 5y ago
> Doesn't that feel like a repeat with whats happening in social media and news media these days? Or is it just different things. Maybe. It can feel like this, but applying mathematical models to social phenomena is tricky. You have to map formal parameters to fuzzy, poorly understood factors. How do you define, precisely, what is a "channel" on a social network? What is "signal", and what is "noise"? People can and do prove anything by using slightly different (or inconsistent) definitions here, getting the numbers to line up just like they want them to. My understanding is that a better way to approach such mapping would be to shove in probability distributions in place of hard-to-map exact parameters - abstracting away choices and measurements lets you see the wider context here. This pushes the problem to defining the appropriate distributions, but I think that's more tamper-proof. Unfortunately, the results may come out next to useless - e.g. probability distributions so wide you could sail a carrier strike group through them. I'd love to know what's considered the correct, robust approach to such problems.
- sig-reduction 5y ago

nathias 5y ago

'shannonigans'

llaolleh 5y ago

Shannon was awesome. It seemed like after he wrote his groundbreaking information theory paper he did whatever the hell he wanted for the rest of his life in Cambridge. Made all kinds of contraptions and devices.

Shatnerz 5y ago

I recommend reading "A Mind at Play", a biography about Shannon. I found it to be a decent read. Funnily enough, I knew about him more through the unicycling community than I did through computer science. It wasn't until a few years ago when I realised what sort of impact he had on the world.

idatum 5y ago
I also enjoyed "A Mind at Play". I had forgotten (or never fully appreciated) what Shannon did and what other early pioneers did in computing. I have a better appreciation for all of them now. One other highlight was Shannon's early childhood and using electrified fencing to communicate with others in his farmland neighborhood.

ShaneCurran 5y ago

Many know Shannon as the author of "A Mathematical Theory of Communication" and the founder of information theory, as well as the brain behind "Communication Theory of Secrecy Systems" Both papers derive from a technical report, "A Mathematical Theory of Cryptography"[0], written by Shannon in 1945 — 3 years before his breakthrough papers in the Bell Labs Technical Journal. [0]: https://evervault.com/papers/shannon.pdf https://evervault.com/papers/shannon.pdf

lqet 5y ago

> In 1973, he recalled, he persuaded Shannon to give the first annual Shannon lecture at the International Information Theory Symposium, but Shannon almost backed out at the last minute. “I never saw a guy with so much stage fright,” Berlekamp said. “In this crowd, he was viewed as a godlike figure, and I guess he was worried he wouldn’t live up to his reputation.” > In 1985 he made an unexpected appearance at the International Information Theory Symposium in Brighton, England. The meeting was proceeding smoothly, if uneventfully, when news raced through the halls and lecture rooms that the snowy-haired man with the shy grin who was wandering in and out of the sessions was none other than Claude Shannon. Some of those at the conference had not even known he was still alive. > At the banquet, the meeting’s organizers somehow persuaded Shannon to address the audience. He spoke for a few minutes and then—fearing that he was boring his audience, he recalled later—pulled three balls out of his pockets and began juggling. The audience cheered and lined up for autographs.

harveywi 5y ago
> At the banquet, the meeting’s organizers somehow persuaded Shannon to address the audience. He spoke for a few minutes and then—fearing that he was boring his audience, he recalled later—pulled three balls out of his pockets and began juggling. Shannon was famous for his entropy.
- 1cvmask 5y ago
  The alleged origins of the naming of entropy: When Shannon first derived his famous formula for information, he asked von Neumann what he should call it and von Neumann replied “You should call it entropy for two reasons: first because that is what the formula is in statistical mechanises but second and more important, as nobody knows what entropy is, whenever you use the term you will always be at an advantage! From: http://www.spatialcomplexity.info/what-von-neumann-said-to-shannon/ http://www.spatialcomplexity.info/what-von-neumann-said-to-s...
  - 2snakes 5y ago
    Yeah, I would characterize it as a mathematical theory of -symbols- and that information has different definitions. However, the argument can be made that symbolic representation is a principle of reality.

tekstar 5y ago

A really good book is "The Idea Factory" by Jon Gertner. It describes Bell Labs in this period, and through the book basically tells the story of where modern life was born: everything from growing crystals to transistors to cell phones to radar was created at Bell Labs.

ansible 5y ago

Curiosity Stream has a recent video centering around a fictionalized interview with Claude Shannon: https://curiositystream.com/video/3831 https://curiositystream.com/video/3831 If you sign up through a YouTube promotional link, you also get access to Nebula for free: https://curiositystream.com/legaleagle/index.html https://curiositystream.com/legaleagle/index.html

Wistar 5y ago
Here's a very so-so promo site for The Bit Player. I saw it about a year ago (Curiosity Stream) and thought it was pretty decent. I somewhat sheepishly admit that, for a while, I thought that they had somehow interviewed the real Claude Shannon. https://thebitplayer.com/ https://thebitplayer.com/ Although not devoted to Claude Shannon, Jim Al-Khalili's documentary "The Story of Information" does discuss Shannon's significant contributions and is how I first became aware of Claude Shannon. https://topdocumentaryfilms.com/story-information/ https://topdocumentaryfilms.com/story-information/

sillysaurusx 5y ago

Shannon was cursed by being the inventor of information theory. It seems like this wonderful thing; who wouldn’t want to invent information theory? But as an inventor, you’ll never top it. You’ll forever live in your own shadow. And for someone that enjoys inventing things, that’s... not a happy thing.

mycologos 5y ago
Nothing I've read about Shannon suggests that he felt "cursed" in his later life. Is there some reason to think otherwise, or is this just assuming that anybody who does something big early feels sad about it later?
- jecel 5y ago
  In Shannon's case, given how fundamental his 1937 thesis was (you can use Boolean Algebra to design digital circuits), most other people might have been afraid of not being able to top that. My impression was that he was always looking forward to bigger things through his own work and through his students. https://en.wikipedia.org/wiki/A_Symbolic_Analysis_of_Relay_and_Switching_Circuits https://en.wikipedia.org/wiki/A_Symbolic_Analysis_of_Relay_a...

vieclamdailoan 5y ago

realy?

legohead 5y ago

another recommended book: The Idea Factory: Bell Labs and the great age of american innovation. Includes a lot of interesting information, including Claude Shannon of course, who worked at Bell Labs. Bell Labs was a remarkable place.

JoeDaDude 5y ago

Claude Shannon was also an early pioneer in artificial intelligence, the minimax algorithm applied to game trees was one of his inventions. He built several machines to play games and demonstrate his ideas, some of which can be seen in the article photos. I compiled a list of his game gadgets, most of which still exist in the care of the MIT museum. https://boardgamegeek.com/geeklist/143233/claude-shannon-man-games-and-machines https://boardgamegeek.com/geeklist/143233/claude-shannon-man...

krosaen 5y ago

An interesting tidbit from Hamming in The Art of Doing Science and Engineering [1][2] on the name "Information Theory" > Information Theory was created by C.E.Shannon in the late 1940s. The management of Bell Telephone Labs wanted him to call it “Communication Theory” as that is a far more accurate name, but for obvious publicity reasons “Information Theory” has a much greater impact—thus Shannon chose and so it is known to this day. [1] https://press.stripe.com/#the-art-of-doing-science-and-engineering https://press.stripe.com/#the-art-of-doing-science-and-engin... [2] http://worrydream.com/refs/Hamming-TheArtOfDoingScienceAndEngineering.pdf http://worrydream.com/refs/Hamming-TheArtOfDoingScienceAndEn...