8 ms·
Alice's adventures in a differentiable wonderland
- TheRealNGenius 3y ago[dead]
- glonq 3y agoI wonder if the usage of Alice & Wonderland takes inspiration from Douglas Hofstadter's "Gödel, Escher, Bach: an Eternal Golden Braid" ?
- iainmerrick 3y agoIt's a pretty common trope, especially for math-related books, e.g. Alex Bellos' "Alex's Adventures in Numberland".
- devnonymous 3y agoNo, I think the inspiration is more direct https://duckduckgo.com/?q=lewis+carroll+alice+in+wonderland+math https://duckduckgo.com/?q=lewis+carroll+alice+in+wonderland+...
- abdullahkhalids 3y agoLewis Carrol's Alice in Wonderland features a number of logical and mathematical puzzles [1] He also wrote What the Tortoise Said to Achilles (1895) in which the paradoxes of Zeno are discussed. So it's more correct to say that GEB and this article are originally inspired by Lewis Carrol's work. [1] I wrote a short article for my university magazine a long time ago. Some interesting references at the end https://abd.tiddlyspot.com/#%5B%5BMathematical%20Adventures%20in%20Wonderland%5D%5D https://abd.tiddlyspot.com/#%5B%5BMathematical%20Adventures%...
- xanderlewis 3y ago> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name; deep learning is simply the application of sequences of operations that are nonlinear (and hence capable of encoding arbitrary complexity) but nonetheless differentiable and so efficiently optimisable.
- andoando 3y agoWhat does "differentiable primitives" mean here?
- CobrastanJorji 3y agoContinuous mathematical functions which have derivatives.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- xanderlewis 3y agoI think it’s referring to ‘primitive functions’ in the sense that they’re the building blocks of more complicated functions. If f and g are differentiable, f+g, fg, f/g (as long as g is never zero)… and so on are differentiable too. Importantly, f composed with g is also differentiable, and so since the output of the whole network as a function of its input is a composition of these ‘primitives’ it’s differentiable too. The actual primitive functions in this case would be things like the weighted sums of activations in the previous layer to get the activation of a given layer, and the actual ‘activation functions’ (traditionally something like a sigmoid function; these days a ReLU) associated with each layer. ‘Primitives’ is also sometimes used as a synonym for antiderivatives, but I don’t think that’s what it means here. Edit: it just occurred to me from a comment below that you might have meant to ask what the ‘differentiable’ part means. See https://en.wikipedia.org/wiki/Differentiable_function https://en.wikipedia.org/wiki/Differentiable_function.
- andoando 3y agoIs this function composition essentially lambda calculus then?
- Eduard 3y agobetter remove all the Disney-based Alice in Wonderland character intellectual property from the book.
- astrodust 3y agoI was just thinking that's "cease and desist" bait right there.
- iainmerrick 3y agoAlice in Wonderland (the book) is in the public domain. The old Disney movie is still in copyright, and the cover image does look very much like it's from the movie, but that character design is from John Tenniel's illustrations which are also in the public domain.
- astrodust 3y agoThe character design is. The image, however, is clearly Disney flavoured if not traced directly. His version, for example, does not have the distinctive bow. The art style is also completely different.
- iainmerrick 3y agoTrue - it would be a good idea to use a Tenniel piece instead. Edit to add: I was mostly trying to push back on the implication that Disney owns Alice in Wonderland (and Peter Pan, Winnie the Pooh, etc). Now I re-read the original comment, they did specify "Disney-based", so maybe I'm over-reacting!
- deleted 3y ago[deleted]
- p1esk 3y agoAnd then you learn about binary or ternary networks where gradients don’t really exist anywhere, and you start to wonder about the importance of this differentiability.
- whimsicalism 3y agobinary networks don't really work well unless you do a relaxation first
- ubj 3y ago...And then you start learning about generalizations of the notion of "gradient" to scenarios where the classical gradient doesn't exist :)
- gfaure 3y agoIn the literature, they're usually called convolutional layers (I think you can pretty much search and replace all uses of "convolutive" in the text).
- seanhunter 3y agoWow, just skimmed a bit, but this book looks amazing so far. Really understandable but with an intuitive presentation of the underlying maths that invites the reader to go deeper if they want to by giving them what they need to get started.