4 ms·
It's really neat to find people on HN who've been working on those structures for such a long time. If you can indulge me, what is a lesser known or obscure bo
by hikingsimulator 5y ago
It's really neat to find people on HN who've been working on those structures for such a long time.
If you can indulge me, what is a lesser known or obscure book on neural networks or an adjacent topic that you think would deserve to be read?
- mark_l_watson 5y agoPerhaps some of the really old texts by Kahonen, Carver Mead, etc.? For more modern material, there are a few new good books on Transformers. Transformers are interesting because they were designed for efficiency: layers the same size, encoding both data and time sequencing information in each sample (so recurrent NNs aren’t required), etc.
- pfd1986 5y agoMind sharing some titles?
- moyix 5y agoI have been enjoying Natural Language Processing with Transformers [1]. It's largely focused on the Huggingface library, but Chapter 3 has a very nice walkthrough that builds up the encoder portion of an encoder-decoder Transformer from "scratch" (it still uses some primitives found in PyTorch like nn.Embedding). The decoder portion is covered in less depth and they instead refer folks to Karpathy's awesome minGPT [2], which implements a decoder-only (GPT-style) Transformer in ~300 lines of nicely-commented Python+PyTorch code. For a higher-level conceptual view of how Transformers work, you can check out the now-classic "Illustrated Transformer" series [3] and this programmer-oriented explanation (with code in Rust) from someone at Anthropic [4]. [1] https://www.oreilly.com/library/view/natural-language-processing/9781098103231/ https://www.oreilly.com/library/view/natural-language-proces... [2] https://github.com/karpathy/minGPT https://github.com/karpathy/minGPT [3] https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-transformer/ [4] https://blog.nelhage.com/post/transformers-for-software-engineers/ https://blog.nelhage.com/post/transformers-for-software-engi...
- jointpdf 5y agoNot OP, but I think this is a great (and obscure?) book on neural networks: https://page.mi.fu-berlin.de/rojas/neural/ https://page.mi.fu-berlin.de/rojas/neural/ Published in 1996, but the content is all foundational.
- sjg007 5y agoRead up on the calculus of variations if you want to be obscure with respect to neural networks. Genetic algorithms for hyper-parameter search is also interesting. The "deep" part of DNNs has basically thrown mathematicians and statisticians into an infinite loop that they can't quite compute yet. It's a brand new world and we need them to participate.