4 ms·
"from scratch" but uses autograd and glosses over backpropagation.
by alephxyz 5y ago
"from scratch" but uses autograd and glosses over backpropagation.
- bigdict 5y agoimport torch as scratch from scratch import nn
- charcircuit 5y agoHe doesn't implement matrix operations, floating point addition / multiplication either.
- weberer 5y agoI bet he didn't even create the universe.
- godelski 5y agoThe difference is that autograd isn't something you should already know if you're learning neural networks. Many "from scratch" tutorials implement backprop because this is a key part. I think your comment is a bit facetious and you're not acting in good faith.
- charcircuit 5y agoI could definitely see an argument that knowing gradient descent is a requisite just as much as knowing matrix multiplication. Both of these mathematical concepts are being abstracted over and are being used by the author's neutral network implementation. Now whether you consider using other people's libraries for this as being "from scratch" is up to you.
- deleted 5y ago[deleted]
- Imnimo 5y agoYeah, the autograd choice struck me as odd. Given how simple the model is, it feels like it would have been easy to show how to compute gradients. The whole benefit of having this super simple toy problem is that we can reason about the meaning of individual weights - it's a perfect opportunity to build clear intuition about gradients and weight updates. Switching to torch is just substituting one black box for another - to a novice reader, the torch code is just magical incantations.
- ogogmad 5y agoThis could be the start of a breath-first approach, where you start with very little code, and then dig deep into things like autograd or "backprop" as you get interested in such details. It seems to me that trying to give explicit formulas for gradients is just swamping the beginner with unnecessary details that don't help to build intuition. I think the author made exactly the right choices. It used to be that some NN tutorials would swamp the beginner with backprop formulas, which beginners were forced by their professors to memorise. I don't think this succeeded at doing much; it only made the subject seem more complicated than it needed to be; and I think it should all be abstracted away into autograd.