3 ms·

Claude Shannon's "The Mathematical Theory of Communication" (not mentioned by name, but referenced in the PDF) is a really pleasant little read. This is a foun

Claude Shannon's "The Mathematical Theory of Communication" (not mentioned by name, but referenced in the PDF) is a really pleasant little read. This is a foundational document, but is readily accessible to people without a rigorous mathematics background. https://openlibrary.org/works/OL2296213W/The_mathematical_theory_of_communication https://openlibrary.org/works/OL2296213W/The_mathematical_th...

crystal_revenge 1y ago
Shannon is also a great way to get started understanding mathematical reasoning. Largely because he derives information entropy from basic desiderata regarding how a mathematical model of information should behave. That is, information entropy doesn’t initially have any meaning, it just fits Shannon’s requirements. What’s brilliant about this is that Shannon accidentally arrives at a definition that is essentially identical to thermodynamic entropy (it was actually Von Neumann who pointed this out and gave it the name). My experience is that many people fail to understand mathematics because mathematics often follows from “what do such and such rules imply” rather than building an intuitive model of the world (which is closer to where Physics traditionally falls). Interestingly enough though, Shannon only establishes the framework which makes coding theory possible. He doesn’t actually implement any of these examples in that paper. I used to run a book club many years ago at a startup that worked through the book version of the paper. A great way for anyone to understand both information theory specifically and mathematics in general.
- ath92 1y ago
  Interestingly Shannon did write about entropy relating to the English language, and how given a sequence of tokens, the next token can be predicted using the probabilities of finding that token after a certain sequence in other bodies of text: http://medientheorie.com/doc/shannon_redundancy.pdf http://medientheorie.com/doc/shannon_redundancy.pdf This is from 1950. I wonder what he would have to say about today’s LLMs.

crystal_revenge 1y ago

Shannon is also a great way to get started understanding mathematical reasoning. Largely because he derives information entropy from basic desiderata regarding how a mathematical model of information should behave. That is, information entropy doesn’t initially have any meaning, it just fits Shannon’s requirements. What’s brilliant about this is that Shannon accidentally arrives at a definition that is essentially identical to thermodynamic entropy (it was actually Von Neumann who pointed this out and gave it the name). My experience is that many people fail to understand mathematics because mathematics often follows from “what do such and such rules imply” rather than building an intuitive model of the world (which is closer to where Physics traditionally falls). Interestingly enough though, Shannon only establishes the framework which makes coding theory possible. He doesn’t actually implement any of these examples in that paper. I used to run a book club many years ago at a startup that worked through the book version of the paper. A great way for anyone to understand both information theory specifically and mathematics in general.

ath92 1y ago
Interestingly Shannon did write about entropy relating to the English language, and how given a sequence of tokens, the next token can be predicted using the probabilities of finding that token after a certain sequence in other bodies of text: http://medientheorie.com/doc/shannon_redundancy.pdf http://medientheorie.com/doc/shannon_redundancy.pdf This is from 1950. I wonder what he would have to say about today’s LLMs.