6 ms·
Hacker's guide to Neural Networks (2012)
- du_bing 9y agoWonderful guide, thanks for sharing!
- NegatioN 9y agoThis has been submitted quite a few times in the past: https://hn.algolia.com/?query=karpathy.github.io%2Fneuralnets&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=karpathy.github.io%2Fneuralnet...
- sillysaurus3 9y agoAw. And I was hoping Chapter 3 and 4 would be finished sometime soonish. It's the only part of the guide that you can learn from by example.
- larryfreeman 9y agoIt would be awesome if there was a section in Hacker News called Classics where this could be posted.
- imranq 9y agoWhat would be Classic and what would not? I guess it would be a combination of votes, times posted, and a minimum age
- larryfreeman 9y agoTotal votes would qualify and then perhaps a combination of views and comments that occur at least one month or more later. The goal is to track posts that have an extended relevancy beyond the current month. Ideally, beyond the current 3 months. Posts that are more than 3 years old but still get high votes when submitted should also be given special attention.
- finchisko 9y agothanks for sharing, apparently i missed past submits
- adamkochanowicz 9y agoThank you for posting this! I hadn't seen it and have been looking for a simple guide like this one.
- debacle 9y agoStatic neural networks on Rosetta Code for basic things like Hello World, etc, would do a lot to aid in people's understanding of neural networks. It would be interesting to visualize different trained solutions.
- frenchie4111 9y agoI've read so many of these, none of them include the information I need. If someone wrote a "Hackers guide to Tuning Hyperparameters" or "Hackers guide to building models for production" I would ready/share the shit out of those.
- kleebeesh 9y agoThis might be helpful:http://nmarkou.blogspot.com/2017/02/the-black-magic-of-deep-learning-tips.html?m=1 http://nmarkou.blogspot.com/2017/02/the-black-magic-of-deep-...
- joaoaccarvalho 9y agoNot sure if this is what you want, but they are aiming to self tune parameters for you. http://www.ml4aad.org/ http://www.ml4aad.org/
- stared 9y agoBoth are tricky subject and lot of cooking-like thinking (there are popular recipes, general rules, but in all practical cases to master it there is a lot of non-transferable experience and knowledge). Some pointers: * http://rishy.github.io/ml/2017/01/05/how-to-train-your-dnn/ http://rishy.github.io/ml/2017/01/05/how-to-train-your-dnn/ * http://www.alexirpan.com/2017/06/27/hyperparam-spectral.html http://www.alexirpan.com/2017/06/27/hyperparam-spectral.html At deepsense.io we are developing Neptune to facilitate that process (compare models, version-control them), here: https://go.neptune.deepsense.io https://go.neptune.deepsense.io (a public release in less that a month!).
- putnam 9y agoAnother resource - https://sigopt.com/ https://sigopt.com/, in the vein of Whetlab. e.g. For NLP: * https://aws.amazon.com/blogs/ai/fast-cnn-tuning-with-aws-gpu-instances-and-sigopt/ https://aws.amazon.com/blogs/ai/fast-cnn-tuning-with-aws-gpu... Disclaimer: they did share free trials with some of our students.
- 9y ago
- amelius 9y agoHmm, I've just scanned through this, but it seems this gets the concept of stochastic gradient descent (SGD) completely wrong. The nice part of SGD is that you can backpropagate even functions that are not differentiable. This is totally missed here.
- kleebeesh 9y agoHow do you compute the gradient of a non differentiable function? I'm not an expert, but that contradicts anything I've learned about gradient descent.
- Houshalter 9y agoPseudo gradients are a thing. You can pretend its a continuous function and get gradients that push it in the right direction.
- Helmet 9y agoCan you explain? I thought the entire point of back-propagation was to differentiate and calculate the weights that are contributing more to the error and thus will be "changed" more when you do GD.
- deleted 9y ago[deleted]
- albertzeyer 9y agoNo, stochastic gradient descent (SGD) needs a differentiable loss function (differentiable with respect to the trainable parameters). The stochastic bit is that you want to optimize the loss for the whole dataset but what you actually do is to do each gradient descent step on a small mini-batch of the dataset. There might be extensions where you have some non-differentiable part where you just assume some gradient in back-propagation. Like Gumble-Softmax (https://arxiv.org/abs/1611.01144 https://arxiv.org/abs/1611.01144). Or similar is the reparametrization trick in VAEs (https://arxiv.org/abs/1606.05908 https://arxiv.org/abs/1606.05908). But that are special cases. Maybe you are thinking about reinforcement learning?
- 9y ago
- stared 9y agoWhen it comes to backpropagation, PyTorch introduction contains some valuable parts: http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html http://pytorch.org/tutorials/beginner/deep_learning_60min_bl...
- GoldDust 9y agoAs someone who is quite new to this field and also a software developer I really look forward to seeing this progress. I write and look at code all day so for me this is much easier to read than the dry math!
- deleted 9y ago[deleted]
- postit 9y agoA good sit in probability theory and multivariate calculus is the first thing you should spend your time if you want to understand NN, ML and most of AI for once. These hacker guides only scratch the surface of the subject which, in part, contributes to creating this aura of black magic that haunts the field; I'm not saying that is a bad thing though, but it needs to be a complementary material, not the way to go.
- deleted 9y ago[deleted]
- nategri 9y agoKnew this wasn't for me when he had to introduce what a derivative was with a weird metaphor. I like this approach to teaching things (it's Feynman-y) but half the time I end up hung up on trying to understand a particular author's hand-waving for a concept I already grok.