7 ms·

Honestly, the best thing you can do is try to implement your own shitty neural net with only Python + Numpy, from scratch, with only a basic understanding of th

by jclos 9y ago

Honestly, the best thing you can do is try to implement your own shitty neural net with only Python + Numpy, from scratch, with only a basic understanding of the math. It will make most of the math very concrete very fast.

srean 9y ago
Cannot upvote this enough. I think this is the only way to really grok backpropagation. The hours of staring at the update formula till your eyes glaze over the subscripts and superscripts and the summations would not give you as good an understanding as implementing a toy neural net with just a single hidden layer. Its actually a whole lot easier than parsing those low-level notation. It can be done better with high level notation but then you would need familiarity with the relevant mathematical abstractions.
- jclos 9y ago
  Yes, once you've dealt with implementing it, the notation just makes sense as the most compact way of formalizing what you just did.
- annnnd 9y ago
  I respectfully disagree. Of course you need to really understand backpropagation for any advanced stuff, but it is easier to ignore it at the beginning. Take Keras container, copy some MNIST example from somewhere and tweak it. Then, when you have a general feel of how training NNs works, gradually learn about each concept - BP should of course be one of the first. By the time you get into math stuff it will probably make much more sense because you will understand how it applies to your case. But I guess the approach depends on how you best learn, so there is no wrong answer. Just jump in!
harel 9y ago
Good idea! I'll give it a go
- RandyRanderson 9y ago
  Don't bother. The guy that wrote Encog has several yt tutorials on how he did it and there are hundreds of others. I learned nothing from impl my own other than why all NN libs break when you input values > 1. Any decent NN lib will be way better that whatever you could write in a week full time. Pick your fav lang, find the most used NN lib and try a kaggle competition. If you can get to about rank 50% your training is complete.
doktrin 9y ago
Naive question : is this strategy equally effective for all ML models?
- kgwgk 9y ago
  This strategy is equally effective for most things in life.
  - doktrin 9y ago
    > This strategy is equally effective for most things in life. Not if we understand 'effective' to also mean 'cost & time effective' You'd learn a lot by building a nuclear reactor from first principles, but it's not the most effective way to develop an intuition about how one operates.
    - yorwba 9y ago
      I think you want to talk about whether the strategy is efficient, which I agree it is not. However, if you already tried understanding several general descriptions and it didn't work out, implementing something from scratch is an inefficient but effective way of really grokking it.
      - doktrin 9y ago
        > I think you want to talk about whether the strategy is efficient effective and efficient are synonyms in this context http://www.thesaurus.com/browse/efficient?s=t http://www.thesaurus.com/browse/efficient?s=t http://www.dictionary.com/browse/efficient?s=t http://www.dictionary.com/browse/efficient?s=t http://www.dictionary.com/browse/effective?s=t http://www.dictionary.com/browse/effective?s=t
- jclos 9y ago
  Some require a lot more understanding than others (for instance I'm not sure I'd be comfortable implementing a kernelized SVM from scratch, even though intuitively I know how it works) but basic neural networks (simple perceptron, simple feedforward network, simple recurrent network) are quite easy to grasp, and backpropagation is very intuitive. You can even use finite difference approximation [1] to bypass the derivatives when you're starting (at the cost of some efficiency) and figure out the rest as you go. [1] https://en.wikipedia.org/wiki/Finite_difference https://en.wikipedia.org/wiki/Finite_difference

by jclos 9y ago

srean 9y ago

Cannot upvote this enough. I think this is the only way to really grok backpropagation. The hours of staring at the update formula till your eyes glaze over the subscripts and superscripts and the summations would not give you as good an understanding as implementing a toy neural net with just a single hidden layer. Its actually a whole lot easier than parsing those low-level notation. It can be done better with high level notation but then you would need familiarity with the relevant mathematical abstractions.

jclos 9y ago
Yes, once you've dealt with implementing it, the notation just makes sense as the most compact way of formalizing what you just did.
annnnd 9y ago
I respectfully disagree. Of course you need to really understand backpropagation for any advanced stuff, but it is easier to ignore it at the beginning. Take Keras container, copy some MNIST example from somewhere and tweak it. Then, when you have a general feel of how training NNs works, gradually learn about each concept - BP should of course be one of the first. By the time you get into math stuff it will probably make much more sense because you will understand how it applies to your case. But I guess the approach depends on how you best learn, so there is no wrong answer. Just jump in!

harel 9y ago

Good idea! I'll give it a go

RandyRanderson 9y ago
Don't bother. The guy that wrote Encog has several yt tutorials on how he did it and there are hundreds of others. I learned nothing from impl my own other than why all NN libs break when you input values > 1. Any decent NN lib will be way better that whatever you could write in a week full time. Pick your fav lang, find the most used NN lib and try a kaggle competition. If you can get to about rank 50% your training is complete.

doktrin 9y ago

Naive question : is this strategy equally effective for all ML models?

kgwgk 9y ago
This strategy is equally effective for most things in life.
- doktrin 9y ago
  > This strategy is equally effective for most things in life. Not if we understand 'effective' to also mean 'cost & time effective' You'd learn a lot by building a nuclear reactor from first principles, but it's not the most effective way to develop an intuition about how one operates.
  - yorwba 9y ago
    I think you want to talk about whether the strategy is efficient, which I agree it is not. However, if you already tried understanding several general descriptions and it didn't work out, implementing something from scratch is an inefficient but effective way of really grokking it.
    - doktrin 9y ago
      > I think you want to talk about whether the strategy is efficient effective and efficient are synonyms in this context http://www.thesaurus.com/browse/efficient?s=t http://www.thesaurus.com/browse/efficient?s=t http://www.dictionary.com/browse/efficient?s=t http://www.dictionary.com/browse/efficient?s=t http://www.dictionary.com/browse/effective?s=t http://www.dictionary.com/browse/effective?s=t
jclos 9y ago
Some require a lot more understanding than others (for instance I'm not sure I'd be comfortable implementing a kernelized SVM from scratch, even though intuitively I know how it works) but basic neural networks (simple perceptron, simple feedforward network, simple recurrent network) are quite easy to grasp, and backpropagation is very intuitive. You can even use finite difference approximation [1] to bypass the derivatives when you're starting (at the cost of some efficiency) and figure out the rest as you go. [1] https://en.wikipedia.org/wiki/Finite_difference https://en.wikipedia.org/wiki/Finite_difference