5 ms·

I understand and am sympathetic towards the sentiment, but honestly, its a bit like asking for a tutorial on swimming that does not involve water. Something li

by srean 9y ago

I understand and am sympathetic towards the sentiment, but honestly, its a bit like asking for a tutorial on swimming that does not involve water. Something like that can be written, but it wont be very useful. Most of the simple stuff, the non-mathematical parts would get automated away. You don't want to be in a position where you are competing with someone's commodity script (sometimes just a for loop), unless the situation calls for desperate measures. A bodybuilder got to lift them weights. One genuine scenario could be that you personally do not do ML but want to evaluate/understand what your hires are doing. Even then, its hard to do avoid the math if you want to do a semi-decent job.

harel 9y ago
I get it, and I've actually used the same metaphor just a couple days ago. What I mean though, is that the maths are OK, and although the heavy notation goes over my head (as I'm not academic), I understand the ideas. But I'm lacking some practical examples of how this is put to use. I understand how neural networks work for example, but I do not understand the maths behind how they work. There was (is?) a site called ai-junkie which used (does?) have very practical and layman docs about A.I. before ML became the buzz word du jour.
- srean 9y ago
  Oh I do understand you. The manual for driving a car has to be different from the manual for designing a car. I believe you are looking for a manual to drive the car where you don't really need a lot of visibility into the inner workings. A problem is that ML is not quite as mature as a car yet, so the driving manuals will be a bit on the thinner / shallower side. > I understand how neural networks work Quickly write that down please, that would do the world a favor. Researchers are still grappling with the question 'why the hell does this freaking thing work as well as it does, when it does'.
  - harel 9y ago
    Haha, I meant, the practical idea behind their usage. The example of OCR via a NN was very good in drilling that concept into my head (many inputs leading down to output). The HOW (in capitals) they work bit - I'm not going there. This thread here got me searching and I'm reading through the basic TensorFlow docs. That's, so far, sinking in.
- jclos 9y ago
  Honestly, the best thing you can do is try to implement your own shitty neural net with only Python + Numpy, from scratch, with only a basic understanding of the math. It will make most of the math very concrete very fast.
  - srean 9y ago
    Cannot upvote this enough. I think this is the only way to really grok backpropagation. The hours of staring at the update formula till your eyes glaze over the subscripts and superscripts and the summations would not give you as good an understanding as implementing a toy neural net with just a single hidden layer. Its actually a whole lot easier than parsing those low-level notation. It can be done better with high level notation but then you would need familiarity with the relevant mathematical abstractions.
    - jclos 9y ago
      Yes, once you've dealt with implementing it, the notation just makes sense as the most compact way of formalizing what you just did.
    - annnnd 9y ago
      I respectfully disagree. Of course you need to really understand backpropagation for any advanced stuff, but it is easier to ignore it at the beginning. Take Keras container, copy some MNIST example from somewhere and tweak it. Then, when you have a general feel of how training NNs works, gradually learn about each concept - BP should of course be one of the first. By the time you get into math stuff it will probably make much more sense because you will understand how it applies to your case. But I guess the approach depends on how you best learn, so there is no wrong answer. Just jump in!
  - harel 9y ago
    Good idea! I'll give it a go
    - RandyRanderson 9y ago
candiodari 9y ago
Given that the difference between any 2 ML implementations is going to be 2 things, everything else is going to be eliminated by a hyperparameter search. Neither of these require more than basic mathematical knowledge. Certainly nothing that wouldn't be covered in an undergraduate statistics course. And of course, it's loads and loads of work. What makes the difference in success for machine learning projects (this is assuming you have some process to avoid screwups in place): 1) quality of how the data is input into the network. You can almost never "usefully" put raw data in front of a neural network. Call it "data represantation" (for instance, for comparing stock prices do a neural net DNN predictor on the prices. Now do the same on the deltas (today - yesterday). Works 1000x better (still not good enough, but very clear difference)) It may not compare to the feature engineering of SVMs, but it's very present (it kinda does imho, but people tend to get very defensive when suggesting that) 2) your cost/loss function. There are tricks like GANs which are simple in concept and avoid some of the issues, but even there you have your image comparator you're still going to need. You can massively improve image comparisons (e.g. mean square difference of the image scaled 4x4, times 1e12, plus same for 16x16 times 1e6, plus mean square difference at real res beats the crap out of just taking mean square difference. Very good results have also been obtained for MNIST by comparing sorted lists of black pixels, instead of actual images) Many things depend on your cost/loss function, and you especially have the eternal problem : I want it to improve in 10 different ways. How do I balance those well into a single number ? Robot should grab the teddy bear, and shouldn't hit anything else. Those are easy to balance, but how would you balance grabbing the teddy bear at all versus not damaging it ? It will make a huge difference in whether your neural net converges at all.

by srean 9y ago

harel 9y ago

I get it, and I've actually used the same metaphor just a couple days ago. What I mean though, is that the maths are OK, and although the heavy notation goes over my head (as I'm not academic), I understand the ideas. But I'm lacking some practical examples of how this is put to use. I understand how neural networks work for example, but I do not understand the maths behind how they work. There was (is?) a site called ai-junkie which used (does?) have very practical and layman docs about A.I. before ML became the buzz word du jour.

srean 9y ago
Oh I do understand you. The manual for driving a car has to be different from the manual for designing a car. I believe you are looking for a manual to drive the car where you don't really need a lot of visibility into the inner workings. A problem is that ML is not quite as mature as a car yet, so the driving manuals will be a bit on the thinner / shallower side. > I understand how neural networks work Quickly write that down please, that would do the world a favor. Researchers are still grappling with the question 'why the hell does this freaking thing work as well as it does, when it does'.
- harel 9y ago
  Haha, I meant, the practical idea behind their usage. The example of OCR via a NN was very good in drilling that concept into my head (many inputs leading down to output). The HOW (in capitals) they work bit - I'm not going there. This thread here got me searching and I'm reading through the basic TensorFlow docs. That's, so far, sinking in.
jclos 9y ago
Honestly, the best thing you can do is try to implement your own shitty neural net with only Python + Numpy, from scratch, with only a basic understanding of the math. It will make most of the math very concrete very fast.
- srean 9y ago
  Cannot upvote this enough. I think this is the only way to really grok backpropagation. The hours of staring at the update formula till your eyes glaze over the subscripts and superscripts and the summations would not give you as good an understanding as implementing a toy neural net with just a single hidden layer. Its actually a whole lot easier than parsing those low-level notation. It can be done better with high level notation but then you would need familiarity with the relevant mathematical abstractions.
  - jclos 9y ago
    Yes, once you've dealt with implementing it, the notation just makes sense as the most compact way of formalizing what you just did.
  - annnnd 9y ago
    I respectfully disagree. Of course you need to really understand backpropagation for any advanced stuff, but it is easier to ignore it at the beginning. Take Keras container, copy some MNIST example from somewhere and tweak it. Then, when you have a general feel of how training NNs works, gradually learn about each concept - BP should of course be one of the first. By the time you get into math stuff it will probably make much more sense because you will understand how it applies to your case. But I guess the approach depends on how you best learn, so there is no wrong answer. Just jump in!
- harel 9y ago
  Good idea! I'll give it a go
  - RandyRanderson 9y ago

candiodari 9y ago

Given that the difference between any 2 ML implementations is going to be 2 things, everything else is going to be eliminated by a hyperparameter search. Neither of these require more than basic mathematical knowledge. Certainly nothing that wouldn't be covered in an undergraduate statistics course. And of course, it's loads and loads of work. What makes the difference in success for machine learning projects (this is assuming you have some process to avoid screwups in place): 1) quality of how the data is input into the network. You can almost never "usefully" put raw data in front of a neural network. Call it "data represantation" (for instance, for comparing stock prices do a neural net DNN predictor on the prices. Now do the same on the deltas (today - yesterday). Works 1000x better (still not good enough, but very clear difference)) It may not compare to the feature engineering of SVMs, but it's very present (it kinda does imho, but people tend to get very defensive when suggesting that) 2) your cost/loss function. There are tricks like GANs which are simple in concept and avoid some of the issues, but even there you have your image comparator you're still going to need. You can massively improve image comparisons (e.g. mean square difference of the image scaled 4x4, times 1e12, plus same for 16x16 times 1e6, plus mean square difference at real res beats the crap out of just taking mean square difference. Very good results have also been obtained for MNIST by comparing sorted lists of black pixels, instead of actual images) Many things depend on your cost/loss function, and you especially have the eternal problem : I want it to improve in 10 different ways. How do I balance those well into a single number ? Robot should grab the teddy bear, and shouldn't hit anything else. Those are easy to balance, but how would you balance grabbing the teddy bear at all versus not damaging it ? It will make a huge difference in whether your neural net converges at all.