3 ms·

> Real neural net code looks nothing like this. This negates a lot of promise of the article. If one understands principles, but needs to know how to apply the

by avmich 8y ago

> Real neural net code looks nothing like this. This negates a lot of promise of the article. If one understands principles, but needs to know how to apply them, this statement makes the article practically useless. Usually after reading like this goes a jump in explanations, which directs to apply TensorFlow et. al. and even, if you're lucky, explains how to do that - but doesn't explain what it is that TensorFlow does qualitatively different, which would justify putting such a phrase as a warning.

sgillen 8y ago
I still think it's a useful exercise for people like me who learn best by implementing a thing. Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area.
- avmich 8y ago
  > I still think it's a useful exercise for people like me who learn best by implementing a thing. Yes, but if you implement one thing, and then TensorFlow which you plan to use does something very different, what's the point? Ok, different enough to justify the warning. > Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area. Exactly. It's that magic which is interesting, after you're shown how in principle - but only in principle - the thing could work.
  - sgillen 8y ago
    I guess the point here is that in this article we implement some of the magic that the higher level frameworks use, which I think will be useful later when writing and debugging our high level code. But I see your point which is that it would be nice if the article was telling us which part of the high level library we're implementing and what that would look like. I'm still happy with the article though and would not call it almost useless. But hey if you'd rather jump right into a tensorkerastorch tutorial that's an option too!
    - avmich 8y ago
      > I'm still happy with the article though and would not call it almost useless. I really like the article. My problem is the next step, which I look for, rather unsuccessfully. > But hey if you'd rather jump right into a tensorkerastorch tutorial that's an option too! Do you know good tutorials for that? Preferably on the level of this article, only regarding those libraries?
      - sgillen 8y ago
        Pytorch's official one is very good: https://pytorch.org/tutorials/beginner/pytorch_with_examples.html https://pytorch.org/tutorials/beginner/pytorch_with_examples... They even start with NumPy, as the article does.
- cr0sh 8y ago
  If you want a great introduction, check out Andrew Ng's Coursera course: https://www.coursera.org/learn/machine-learning https://www.coursera.org/learn/machine-learning This is the same course that was originally called "ML Class" in 2011 when it was offered via Stanford (it, plus the other course called "AI Class" - helped to launch Coursera and Udacity, respectively). It uses Octave as it's development language to teach the concepts. Octave has as it's "primitives" the concept of the Vector and Matrix; that is, it is a language designed around the concepts of linear algebra. So, you can easily set a variable to the values that make up a vector, set another to say the identity matrix, multiply them together, and it will return a matrix. But the course is initially taught showing how to do all of this "by hand" (much like this tutorial) - using for-next loops and such; essentially rolling your own matrix calcs. Then, once you know that, you are introduced into how to use the built-in calcs provided by octave, etc. It is really a great course, and I encourage you to try it. It opened my eyes and mind to a lot of concepts I had been trying to grasp for a while that just didn't click until I took that course in 2011. Of course, once it did, I was like "that's it? why couldn't I see that?" - such is the nature of learning I guess! His method of building up from the basics all the way to a neural network using Octave will be a good foundation for further learning or playing with other tools like TensorFlow; I ended up taking a couple of other later MOOCs offered by Udacity (focused mainly on self-driving vehicle concepts), and ML Class (and what I took of AI Class) really helped to prepare me. It also showed me things I was deficient in (something I need to correct some day). But that's a different story. Suffice to say, that course will "unveil the magic"; it was really an amazing course that I am glad I took. Oh - and to show you what you can do with what you learn in that course, a fellow student during that time ended up making this (and the course was not even finished when he did it): https://blog.davidsingleton.org/nnrccar/ https://blog.davidsingleton.org/nnrccar/ Throughout the course, the ALVINN autonomous vehicle was referenced quite often, and it inspired him to recreate it successfully in miniature. It also serves to show just how far computer technology has come - what used to take a van-load of equipment can now be played around with using a device that fits in your pocket!
  - avmich 8y ago
    This is a great course, I took it once a few years ago and sometime think if I should take it again. Wonder how it changed over time. It was eye-opening training a small NN by hand on MNIST dataset and getting it to recognize digits surprisingly well. However even understanding all that doesn't allow one using modern frameworks by itself, they seem to carry too much additional assumptions, terminology which aren't quite explained with similar level of patience.
cr0sh 8y ago
What is being meant is that the example is one of "unrolled loops" - that is, to an extent, some of the matrix math has been unrolled in order to show the discrete steps involved in the calculations that would be lost (more opaque) for someone learning. Real systems like TensorFlow (and Keras, which is a wrapper that simplifies TensorFlow - which is already pretty simple; if you come to love TensorFlow, you will really love Keras - but you can't really appreciate TensorFlow until you understand the basics of what it does - which the posted tutorial tries to show) hide all of these steps behind a variety of API that tries to mimic the conceptual ideas behind neural networks (ie - "layers" of "neurons" with "connections") while hiding the lower-level "complex mathematics". That's a good thing on one level, because it opens up ANNs for more users. But it's a bad thing at another level, for those who want to know what is going on in those "black boxes", as well as for those who sometimes will stumble into certain issues with their networks, and struggle to understand certain things if they don't understand those underlying systems. No one implements neural network code the way it is shown in this tutorial for much the same reason most people don't code in assembler: Because there are higher level means to work with the same operations and do it quicker, more efficiently, and more correctly (also, the lower level operations used in such libraries have been rigorously tested to be correct - it's similar to the reasoning on how you don't roll your own crypto, because you will likely get it wrong - even if you are an expert in the domain).
Shorel 8y ago
> If one understands principles In that case, the hard part is already done. Understanding the principles is what takes the most work, not writing or adapting the code. In fact, understanding the principles is IMO a prerequisite for being able to successfully adapt the code. Now, using something like TensorFlow in another later article is of course super useful. It also will clarify why TensorFlow is more confusing and why this article was written. I find the article enlightening, and specific libraries like TensorFlow are just "implementation details".

by avmich 8y ago

sgillen 8y ago

I still think it's a useful exercise for people like me who learn best by implementing a thing. Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area.

avmich 8y ago
> I still think it's a useful exercise for people like me who learn best by implementing a thing. Yes, but if you implement one thing, and then TensorFlow which you plan to use does something very different, what's the point? Ok, different enough to justify the warning. > Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area. Exactly. It's that magic which is interesting, after you're shown how in principle - but only in principle - the thing could work.
- sgillen 8y ago
  I guess the point here is that in this article we implement some of the magic that the higher level frameworks use, which I think will be useful later when writing and debugging our high level code. But I see your point which is that it would be nice if the article was telling us which part of the high level library we're implementing and what that would look like. I'm still happy with the article though and would not call it almost useless. But hey if you'd rather jump right into a tensorkerastorch tutorial that's an option too!
  - avmich 8y ago
    > I'm still happy with the article though and would not call it almost useless. I really like the article. My problem is the next step, which I look for, rather unsuccessfully. > But hey if you'd rather jump right into a tensorkerastorch tutorial that's an option too! Do you know good tutorials for that? Preferably on the level of this article, only regarding those libraries?
    - sgillen 8y ago
      Pytorch's official one is very good: https://pytorch.org/tutorials/beginner/pytorch_with_examples.html https://pytorch.org/tutorials/beginner/pytorch_with_examples... They even start with NumPy, as the article does.
cr0sh 8y ago
If you want a great introduction, check out Andrew Ng's Coursera course: https://www.coursera.org/learn/machine-learning https://www.coursera.org/learn/machine-learning This is the same course that was originally called "ML Class" in 2011 when it was offered via Stanford (it, plus the other course called "AI Class" - helped to launch Coursera and Udacity, respectively). It uses Octave as it's development language to teach the concepts. Octave has as it's "primitives" the concept of the Vector and Matrix; that is, it is a language designed around the concepts of linear algebra. So, you can easily set a variable to the values that make up a vector, set another to say the identity matrix, multiply them together, and it will return a matrix. But the course is initially taught showing how to do all of this "by hand" (much like this tutorial) - using for-next loops and such; essentially rolling your own matrix calcs. Then, once you know that, you are introduced into how to use the built-in calcs provided by octave, etc. It is really a great course, and I encourage you to try it. It opened my eyes and mind to a lot of concepts I had been trying to grasp for a while that just didn't click until I took that course in 2011. Of course, once it did, I was like "that's it? why couldn't I see that?" - such is the nature of learning I guess! His method of building up from the basics all the way to a neural network using Octave will be a good foundation for further learning or playing with other tools like TensorFlow; I ended up taking a couple of other later MOOCs offered by Udacity (focused mainly on self-driving vehicle concepts), and ML Class (and what I took of AI Class) really helped to prepare me. It also showed me things I was deficient in (something I need to correct some day). But that's a different story. Suffice to say, that course will "unveil the magic"; it was really an amazing course that I am glad I took. Oh - and to show you what you can do with what you learn in that course, a fellow student during that time ended up making this (and the course was not even finished when he did it): https://blog.davidsingleton.org/nnrccar/ https://blog.davidsingleton.org/nnrccar/ Throughout the course, the ALVINN autonomous vehicle was referenced quite often, and it inspired him to recreate it successfully in miniature. It also serves to show just how far computer technology has come - what used to take a van-load of equipment can now be played around with using a device that fits in your pocket!
- avmich 8y ago
  This is a great course, I took it once a few years ago and sometime think if I should take it again. Wonder how it changed over time. It was eye-opening training a small NN by hand on MNIST dataset and getting it to recognize digits surprisingly well. However even understanding all that doesn't allow one using modern frameworks by itself, they seem to carry too much additional assumptions, terminology which aren't quite explained with similar level of patience.

cr0sh 8y ago

What is being meant is that the example is one of "unrolled loops" - that is, to an extent, some of the matrix math has been unrolled in order to show the discrete steps involved in the calculations that would be lost (more opaque) for someone learning. Real systems like TensorFlow (and Keras, which is a wrapper that simplifies TensorFlow - which is already pretty simple; if you come to love TensorFlow, you will really love Keras - but you can't really appreciate TensorFlow until you understand the basics of what it does - which the posted tutorial tries to show) hide all of these steps behind a variety of API that tries to mimic the conceptual ideas behind neural networks (ie - "layers" of "neurons" with "connections") while hiding the lower-level "complex mathematics". That's a good thing on one level, because it opens up ANNs for more users. But it's a bad thing at another level, for those who want to know what is going on in those "black boxes", as well as for those who sometimes will stumble into certain issues with their networks, and struggle to understand certain things if they don't understand those underlying systems. No one implements neural network code the way it is shown in this tutorial for much the same reason most people don't code in assembler: Because there are higher level means to work with the same operations and do it quicker, more efficiently, and more correctly (also, the lower level operations used in such libraries have been rigorously tested to be correct - it's similar to the reasoning on how you don't roll your own crypto, because you will likely get it wrong - even if you are an expert in the domain).

Shorel 8y ago

> If one understands principles In that case, the hard part is already done. Understanding the principles is what takes the most work, not writing or adapting the code. In fact, understanding the principles is IMO a prerequisite for being able to successfully adapt the code. Now, using something like TensorFlow in another later article is of course super useful. It also will clarify why TensorFlow is more confusing and why this article was written. I find the article enlightening, and specific libraries like TensorFlow are just "implementation details".