5 ms·

Information Theory, Inference, and Learning Algorithms (2003)

237 points by malshe 4y ago

ml_basics 4y ago
Great book by a great teacher. I read this during my masters and was my route into machine learning. David did many great things but sadly passed away too young in 2016, surely would have continued to have a great positive impact in the world had he lived longer. His other projects included the sustainable energy book https://www.withouthotair.com/ https://www.withouthotair.com/ and information theory inspired keyboard software for people with severe disabilities https://www.bltt.org/software/dasher/ https://www.bltt.org/software/dasher/ I had the honour of being a PhD student in Cambridge during his last year of life, and saw him give talks a few times in our lab. It's truly a remarkable experience to see someone who is a passionate explainer in action. Like how I imagine it must have been to see Feynman in the flesh. This is a great short video to get a feel for his style of explaining: https://youtu.be/UR8wRSp2IXs https://youtu.be/UR8wRSp2IXs (Writing this post feels wrong because he's had such a positive impact on my life and those of many others, and this post definitely can't do it justice)
- azalemeth 4y ago
  His books are fantastic and his writing is captivating. It's such a shame he passed -- cancer is a bitch, but the blog post about the radiators in the hospice that he died in are one of the most heart rending pieces of writing I've come across from a mathematician.
  - ml_basics 4y ago
    Is this the post you mean? http://itila.blogspot.com/2016/04/appendix-two-open-letter-to-directors.html?m=1 http://itila.blogspot.com/2016/04/appendix-two-open-letter-t...
    - azalemeth 4y ago
      Yes!
- malshe 4y ago
  Thanks for sharing your experience. The video link is fantastic!
- sideshowb 4y ago
  As a follow on from Sustaianble Energy he also drove the original version of the my2050 game, which communicates the tradeoffs we need to make in decarbonizing to the general public very well. More recent update here https://my2050.beis.gov.uk/?levers=433233211331111 https://my2050.beis.gov.uk/?levers=433233211331111 I've taken this as inspiration for my own transport models, though I have a model with some sliders to drag around I don't have a flashy game yet!
- jll29 4y ago
  A great communicator and a kind man, indeed. It's been a privilege to see him lecturing at the Cavendish in 2002. Here's his last video interview: https://www.youtube.com/watch?v=sCyidsxIDtQ https://www.youtube.com/watch?v=sCyidsxIDtQ
diego898 4y ago
I absolutely adore this book and its approach to information theory. I will also note the author made available the (non compilable) latex source [1] of the text for easy searching! [1]http://www.inference.org.uk/mackay/itprnn/booktext.txt http://www.inference.org.uk/mackay/itprnn/booktext.txt
dang 4y ago
Related: Information Theory, Inference, and Learning Algorithms (2003) [pdf] - https://news.ycombinator.com/item?id=11289488 https://news.ycombinator.com/item?id=11289488 - March 2016 (9 comments) Information Theory, Inference, and Learning Algorithms (free ebook edition) - https://news.ycombinator.com/item?id=852562 https://news.ycombinator.com/item?id=852562 - Sept 2009 (21 comments)
- p1esk 4y ago
  I remember him (David Mackay) leaving comments here on HN, but I don’t remember the username. I wonder if you can find them somehow.
  - ml_basics 4y ago
    If anyone can find this I'd be very grateful, would be cool to read his comments
    - madcaptenor 4y ago
      Tried Googling "mackay site:news.ycombinator.com" and found a lot of times where this book has been mentioned; I didn't see anything that clearly said "I am David Mackay" but that might be a good start. (You'll also find the occasional reference to Charles Mackay's book Extraordinary Popular Delusions and the Madness of Crowds.)
  - dang 4y ago
    I did some searches but couldn't find anything. If someone else does and the thread is closed for replying, let me know at hn@ycombinator.com and I'll reopen it.
ckrapu 4y ago
This book shows up every year or so on HN and I will never stop upvoting it. My favorite textbook of all time, along with my favorite quote: “Marginalization is a macho activity, appropriate only for those fluent in definite integration”
- agileAlligator 4y ago
  "Exact marginalization over continuous parameters is a macho activity enjoyed by those who are fluent in definite integration." http://www.inference.org.uk/itprnn/book.pdf#page=331 http://www.inference.org.uk/itprnn/book.pdf#page=331
a-dub 4y ago
one of my favorites (even though the pdf is free i keep a copy of the book). very clear introductory chapters in probability leading into a really nice blend of applied math and signal processing, finishing out with neural networks. his father published was something of a figure in the fields of perception and cognition as well.
- deleted 4y ago
  [deleted]
mindcrime 4y ago
There's also a nice set of recorded lectures available by Professor MacKay covering the core material from the book, available on videolectures.net. http://videolectures.net/course_information_theory_pattern_recognition/ http://videolectures.net/course_information_theory_pattern_r...
- i_like_apis 4y ago
  Fantastic. Thank you for sharing this!
- noud 4y ago
  "Your browser does not support playback of available video formats. Please install Adobe Flash player or upgrade to a more modern browser." ...Okay? I found the videos on youtube as well: https://www.youtube.com/watch?v=BCiZc0n6COY&list=PLruBu5BI5n4aFpG32iMbdWoRVAA-Vcso6 https://www.youtube.com/watch?v=BCiZc0n6COY&list=PLruBu5BI5n...
  - scythmic_waves 4y ago
    From lecture 1 [1]: > Ok, so, we have an answer of 1000 plus or minus 30 thanks to "Mr. Binomial", the inventor of the distribution that bears his name. Lol, this is great. [1]: https://www.youtube.com/watch?v=BCiZc0n6COY&t=971s https://www.youtube.com/watch?v=BCiZc0n6COY&t=971s
  - mindcrime 4y ago
    Huh. Weird. I'm using Firefox 109.0 on Linux and the videos on videolectures.net play fine for me.
- ml_basics 4y ago
  Yes these are so good! Highly recommend.
smath 4y ago
I attended an amazing talk by David circa 2006 at Princeton where he demoed his probabilistic text entry system called Dasher [1,3]. I think it was made into an iOS app later. I also fondly recall his videos on Gaussian processes - best explanation on GPs I ever came across. [2] [1] https://www.inference.org.uk/dasher/ https://www.inference.org.uk/dasher/ [2] https://www.youtube.com/watch?v=NegVuuHwa8Q https://www.youtube.com/watch?v=NegVuuHwa8Q [3] https://dasher.acecentre.net/about/ https://dasher.acecentre.net/about/ EDIT: Fixed the youtube link in [2], added [3].
- zzz95 4y ago
  Would love to see the GP video, but [2] is points to an unrelated video.
  - smath 4y ago
    Fixed
Tomte 4y ago
The book compares quite well to Harry Potter, unless you need the Latin translation: https://www.inference.org.uk/mackay/itila/Potter.html https://www.inference.org.uk/mackay/itila/Potter.html
Schiphol 4y ago
Cybernetics is alive the same way punk is alive: people say it's dead, but then there are punk bands and punk records and just like a truckload of punks. This book, like those written by his dad, is cybernetic theory in everything but name. I absolutely adore it.
sriram_malhar 4y ago
His books have taught me soooo much; in addition to information theory, I have learnt many tricks of LaTeX typesetting (the info theory book's LaTeX source is open) and how to estimate (from his other book on sustainable energy) David was a fantastic ultimate frisbee player. He taught me how to properly throw forehand and overhand.
wiz21c 4y ago
From the preface: "My work has depended on the generosity of free software authors. I wrote the book in LATEX 2ε. Three cheers for Donald Knuth and Leslie Lamport! Our computers run the GNU/Linux operating system. I use emacs, perl, and gnuplot every day. Thank you Richard Stallman, thank you Linus Torvalds, thank you everyone"
optimalsolver 4y ago
Was the David MacKay on Stargate: Atlantis named after the author?

237 points by malshe 4y ago

Great book by a great teacher. I read this during my masters and was my route into machine learning. David did many great things but sadly passed away too young in 2016, surely would have continued to have a great positive impact in the world had he lived longer. His other projects included the sustainable energy book https://www.withouthotair.com/ https://www.withouthotair.com/ and information theory inspired keyboard software for people with severe disabilities https://www.bltt.org/software/dasher/ https://www.bltt.org/software/dasher/ I had the honour of being a PhD student in Cambridge during his last year of life, and saw him give talks a few times in our lab. It's truly a remarkable experience to see someone who is a passionate explainer in action. Like how I imagine it must have been to see Feynman in the flesh. This is a great short video to get a feel for his style of explaining: https://youtu.be/UR8wRSp2IXs https://youtu.be/UR8wRSp2IXs (Writing this post feels wrong because he's had such a positive impact on my life and those of many others, and this post definitely can't do it justice)

azalemeth 4y ago
His books are fantastic and his writing is captivating. It's such a shame he passed -- cancer is a bitch, but the blog post about the radiators in the hospice that he died in are one of the most heart rending pieces of writing I've come across from a mathematician.
- ml_basics 4y ago
  Is this the post you mean? http://itila.blogspot.com/2016/04/appendix-two-open-letter-to-directors.html?m=1 http://itila.blogspot.com/2016/04/appendix-two-open-letter-t...
  - azalemeth 4y ago
    Yes!
malshe 4y ago
Thanks for sharing your experience. The video link is fantastic!
sideshowb 4y ago
As a follow on from Sustaianble Energy he also drove the original version of the my2050 game, which communicates the tradeoffs we need to make in decarbonizing to the general public very well. More recent update here https://my2050.beis.gov.uk/?levers=433233211331111 https://my2050.beis.gov.uk/?levers=433233211331111 I've taken this as inspiration for my own transport models, though I have a model with some sliders to drag around I don't have a flashy game yet!
jll29 4y ago
A great communicator and a kind man, indeed. It's been a privilege to see him lecturing at the Cavendish in 2002. Here's his last video interview: https://www.youtube.com/watch?v=sCyidsxIDtQ https://www.youtube.com/watch?v=sCyidsxIDtQ

diego898 4y ago

I absolutely adore this book and its approach to information theory. I will also note the author made available the (non compilable) latex source [1] of the text for easy searching! [1]http://www.inference.org.uk/mackay/itprnn/booktext.txt http://www.inference.org.uk/mackay/itprnn/booktext.txt

dang 4y ago

Related: Information Theory, Inference, and Learning Algorithms (2003) [pdf] - https://news.ycombinator.com/item?id=11289488 https://news.ycombinator.com/item?id=11289488 - March 2016 (9 comments) Information Theory, Inference, and Learning Algorithms (free ebook edition) - https://news.ycombinator.com/item?id=852562 https://news.ycombinator.com/item?id=852562 - Sept 2009 (21 comments)

p1esk 4y ago
I remember him (David Mackay) leaving comments here on HN, but I don’t remember the username. I wonder if you can find them somehow.
- ml_basics 4y ago
  If anyone can find this I'd be very grateful, would be cool to read his comments
  - madcaptenor 4y ago
    Tried Googling "mackay site:news.ycombinator.com" and found a lot of times where this book has been mentioned; I didn't see anything that clearly said "I am David Mackay" but that might be a good start. (You'll also find the occasional reference to Charles Mackay's book Extraordinary Popular Delusions and the Madness of Crowds.)
- dang 4y ago
  I did some searches but couldn't find anything. If someone else does and the thread is closed for replying, let me know at hn@ycombinator.com and I'll reopen it.

ckrapu 4y ago

This book shows up every year or so on HN and I will never stop upvoting it. My favorite textbook of all time, along with my favorite quote: “Marginalization is a macho activity, appropriate only for those fluent in definite integration”

agileAlligator 4y ago
"Exact marginalization over continuous parameters is a macho activity enjoyed by those who are fluent in definite integration." http://www.inference.org.uk/itprnn/book.pdf#page=331 http://www.inference.org.uk/itprnn/book.pdf#page=331

a-dub 4y ago

one of my favorites (even though the pdf is free i keep a copy of the book). very clear introductory chapters in probability leading into a really nice blend of applied math and signal processing, finishing out with neural networks. his father published was something of a figure in the fields of perception and cognition as well.

deleted 4y ago
[deleted]

mindcrime 4y ago

There's also a nice set of recorded lectures available by Professor MacKay covering the core material from the book, available on videolectures.net. http://videolectures.net/course_information_theory_pattern_recognition/ http://videolectures.net/course_information_theory_pattern_r...

i_like_apis 4y ago
Fantastic. Thank you for sharing this!
noud 4y ago
"Your browser does not support playback of available video formats. Please install Adobe Flash player or upgrade to a more modern browser." ...Okay? I found the videos on youtube as well: https://www.youtube.com/watch?v=BCiZc0n6COY&list=PLruBu5BI5n4aFpG32iMbdWoRVAA-Vcso6 https://www.youtube.com/watch?v=BCiZc0n6COY&list=PLruBu5BI5n...
- scythmic_waves 4y ago
  From lecture 1 [1]: > Ok, so, we have an answer of 1000 plus or minus 30 thanks to "Mr. Binomial", the inventor of the distribution that bears his name. Lol, this is great. [1]: https://www.youtube.com/watch?v=BCiZc0n6COY&t=971s https://www.youtube.com/watch?v=BCiZc0n6COY&t=971s
- mindcrime 4y ago
  Huh. Weird. I'm using Firefox 109.0 on Linux and the videos on videolectures.net play fine for me.
ml_basics 4y ago
Yes these are so good! Highly recommend.

smath 4y ago

I attended an amazing talk by David circa 2006 at Princeton where he demoed his probabilistic text entry system called Dasher [1,3]. I think it was made into an iOS app later. I also fondly recall his videos on Gaussian processes - best explanation on GPs I ever came across. [2] [1] https://www.inference.org.uk/dasher/ https://www.inference.org.uk/dasher/ [2] https://www.youtube.com/watch?v=NegVuuHwa8Q https://www.youtube.com/watch?v=NegVuuHwa8Q [3] https://dasher.acecentre.net/about/ https://dasher.acecentre.net/about/ EDIT: Fixed the youtube link in [2], added [3].

zzz95 4y ago
Would love to see the GP video, but [2] is points to an unrelated video.
- smath 4y ago
  Fixed

Tomte 4y ago

The book compares quite well to Harry Potter, unless you need the Latin translation: https://www.inference.org.uk/mackay/itila/Potter.html https://www.inference.org.uk/mackay/itila/Potter.html

Schiphol 4y ago

Cybernetics is alive the same way punk is alive: people say it's dead, but then there are punk bands and punk records and just like a truckload of punks. This book, like those written by his dad, is cybernetic theory in everything but name. I absolutely adore it.

sriram_malhar 4y ago

His books have taught me soooo much; in addition to information theory, I have learnt many tricks of LaTeX typesetting (the info theory book's LaTeX source is open) and how to estimate (from his other book on sustainable energy) David was a fantastic ultimate frisbee player. He taught me how to properly throw forehand and overhand.

wiz21c 4y ago

From the preface: "My work has depended on the generosity of free software authors. I wrote the book in LATEX 2ε. Three cheers for Donald Knuth and Leslie Lamport! Our computers run the GNU/Linux operating system. I use emacs, perl, and gnuplot every day. Thank you Richard Stallman, thank you Linus Torvalds, thank you everyone"

optimalsolver 4y ago

Was the David MacKay on Stargate: Atlantis named after the author?