David MacKay was an absolute rockstar and this book is grossly underrated among beginners in machine learning. This should be THE complementary reference for an

David MacKay was an absolute rockstar and this book is grossly underrated among beginners in machine learning. This should be THE complementary reference for anyone who uses the Bishop book. Both these books follow a philosophy which some people may not completely agree with, that from the "church of Bayes". My guess is that information theory went through its hype phase and much of the ideas are so pervasive across real systems that people forget how important those connections are. To tell you its importance, skim this work on information-theoretic probing [1]. I find this so satisfying. Its most famous alternative, linear probing, always felt inelegant. This paper experimentally shows how terribly linear probing fails. [1] https://arxiv.org/abs/2003.12298 https://arxiv.org/abs/2003.12298