7 ms·

Build a Neural Network

174 points by shamdasani 7y ago

fartcannon 7y ago
As someone who has read a lot of implementing neural networks from articles, the massive problem with all of them is that they import numpy. You may think that it is silly to reimplement the matrix math but with out that part of the code, you can't easily port it to other languages/microcontrollers/microwaves/badgers. It's a legitimately valid part of machine learning, and its not easy to do for novices. And I need help putting it on my badger damn it!
- simias 7y ago
  If you don't care much about performance (and if you are reimplementing a neural network from scratch you're probably doing it more as a learning project than anything else) implementing matrix operations isn't very difficult. If you're not used to work with matrices simply reading the Wikipedia article might tell you enough to implement them yourself.
  - peterhj 7y ago
    If you have an assembler or C compiler you can implement matrix multiplication (GEMM) which usually does most of the heavy lifting in your neural net. Now you correctly alluded that it may not be simple to efficiently implement GEMM but if you have a simple architecture without a complex memory hierarchy then using whatever SIMD facilities are available and some standard tricks will get you in ballpark of peak FLOP/s. Or, just download a fast BLAS from your hardware vendor...
- whatshisface 7y ago
  The complicated parts of Numpy are themselves a wrapper for the seminal LAPACK: http://www.netlib.org/lapack/ http://www.netlib.org/lapack/. It has C language APIs, so that might help you with what you need to do.
  - fartcannon 7y ago
    That's very interesting, thank you.
- Anon84 7y ago
  As someone who does teach tutorials as a side gig, I would argue that implementing matrix operations in a tutorial on neural networks is overkill. No matter what the level of the tutorial you always need to draw a line and assume a certain amount of background knowledge and knowing how to use standard tools isn't too much to ask. (yes, I know numpy isn't part of python's standard library, but it comes with pretty much any Python distribution as many other libraries depend on it.) If we're talking about a longer format, such as a book, then we might consider digging deeper and implementing as much as possible using the barest of Python requirements. Indeed, Joel Grus does implement everything from scratch in his great (although a bit dated) book https://www.amazon.com/Data-Science-Scratch-Principles-Python/dp/149190142X https://www.amazon.com/Data-Science-Scratch-Principles-Pytho.... EDIT: This is still a work in progress (and relies on numpy and matplotlib), but here is my version: https://github.com/DataForScience/DeepLearning https://github.com/DataForScience/DeepLearning These notebooks are meant as support for a webinar so they might not be the clearest as standalone, but you also have the slides there.
  - HuShifang 7y ago
    A new edition of Grus comes out next week actually... https://www.amazon.com/Data-Science-Scratch-Principles-Python/dp/1492041130 https://www.amazon.com/Data-Science-Scratch-Principles-Pytho...
    - Anon84 7y ago
      Nice! He mentioned he was working on it when I met him at Strata last year, but I didn't know it was coming out already.
    - quantumOctopus 7y ago
      Ugh I just bought the old one a week ago.
  - bigred100 7y ago
    I’d agree... Outside of very rare circumstances (specialist in numerical linear algebra implementations), my opinion is that implementing matrix operations is something you do once (twice) in your numerical courses to get an intuition for the algorithm, and then never again. But maybe it’s educational to do once if you never have before.
- peterhj 7y ago
  I'm curious if you have a particular language/microcontroller/microwave/badger you have in mind? Depending on which, YMMV.
  - fartcannon 7y ago
    No, not off hand I don't. It's just something I've noticed in all these make a neural network posts. Feels a little like they're just drawing the rest of the owl, if you know what I mean. But thank you.
- gnulinux 7y ago
  Matrix math is easy peasy. Freshman level programming. Just lookup algorithms on Wikipedia and you're all set. The problem is it's extremely hard to make it efficient. Dozens of men-years are spent trying to optimize linear algebra libraries. There are handful linalg libraries that have competitive performance. It was my college project to make a fast linalg library, and boy it is fast. There are some things like matrix multiplication that if you implement in C with the trivial algorithm, takes >2 mins but with some tricks you can make it as fast as <second (vectorization, OpenMP, handwritten assembly, automatically optimized code, various optimizations, better algorithm.....). So, if you want to implement linalg in some language and compile it, go ahead, more power to you. But it's basically impossible to do it efficiently. My opinion is: this is fine and we should do this. There should be linalg libraries written in pure python (and are 1000x slower than lapack) but just understand that it's impossible to satisfy all use cases of numpy this way (at least currently).
  - asdfman123 7y ago
    Can anyone simply explain the gist of how matrix multiplication is optimized? I know a lot of is farmed out to the GPU (if you've got a good GPU), but what's the essence of it? Caching? Some kind of clever mathematical tricks? All of the above?
    - johndough 7y ago
      The answer is: No one really knows because cuBLAS is closed source. But to get within the same order of magnitude, tiling the workload for better cache utilization is usually the most important step. This article [1] explains it quite well and also lists a few other tricks. In addition, there's also the fast Fourier transform for large filter kernels and Winograd convolutions [2] for small filter kernels. [1] https://cnugteren.github.io/tutorial/pages/page1.html https://cnugteren.github.io/tutorial/pages/page1.html [2] https://arxiv.org/pdf/1509.09308.pdf https://arxiv.org/pdf/1509.09308.pdf
      - tntn 7y ago
        Not entirely true - Scott Gray knows: https://github.com/NervanaSystems/maxas/wiki/SGEMM https://github.com/NervanaSystems/maxas/wiki/SGEMM IIRC his kernels shipped in cuBLAS at some point.
- asdfman123 7y ago
  It is easy to do unless you don't code at all, or are completely confused by the math. I'm a C# developer and I'm sure it would take me all of about 30 seconds to install a matrix multiplication package through nuget. I'm sure it would be immediately obvious how to add items to matrices or do a dot product.
  - animal531 7y ago
    I'm a C# developer who wrote my own implementation (with help from random tutorials etc). It was dead easy to get code examples as needed.
- pilooch 7y ago
  Exactly the reason why my colleagues and myself do all deep learning in C++, performance and portability, from cloud to RPie. We've even modified caffe2 so we could build the training graph from pure C++. We know this is not the current doxa :) It's also all open sourced just in case others might need it...
  - fartcannon 7y ago
    Link? I love C++ and would love to see it.
- felipellrocha 7y ago
  Yep. I would like to see an article that implements everything without using matrices first, then creates the matrices library with you, and refactors everything over. So much learning that we're missing by not going through this step.
  - cr0sh 7y ago
    A good course that comes close to this would be the Coursera Machine Learning course (what used to be known as "ML Class" by Andrew Ng). It uses Octave - but you first do everything (in the section on NN) "by hand" - building and looping for the matrix operations. Only after you've gone that far, does he (Ng) introduce the fact that Octave has vector/matrix primitives... I took the original ML Class in the Fall of 2011; it was a great class, and opened my eyes a great deal on the topic of machine learning and neural networks, which I had struggled with understanding in the past (mainly on what and how backprop worked).
  - fartcannon 7y ago
    To me it feels a bit like that joke about drawing instructions. "1. Draw some circles. 2. Now draw the rest of the owl."
- cscheid 7y ago
  Hm. I just finished teaching an ML course where all of the assignments were pure Python (on purpose, so students would actually have the chance to see all of the code). One of the assignments included implementing reverse-mode autodiff and a NN classifier on top. It can be done in ~600 lines of clear python, serious!
  - fartcannon 7y ago
    Which course?
    - cscheid 7y ago
      https://cscheid.net/courses/spr19/csc665/ https://cscheid.net/courses/spr19/csc665/ The assignments are not directly available, but my email is easy to find.
- danlugo92 7y ago
  http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/
melling 7y ago
Here's another Neural Network from scratch that I found useful: https://victorzhou.com/blog/intro-to-neural-networks/ https://victorzhou.com/blog/intro-to-neural-networks/
- samsonradu 7y ago
  Thanks a lot for this, it is indeed very clear and easy to follow! Good walkthrough on the partial derivatives calculations which imo are the hardest part.
rrggrr 7y ago
Would be great if this included real world data or application to understand context.
cwt137 7y ago
If you think this blog article is lacking, get "Make Your Own Neural Network" by Tariq Rashid[1]. It is way more comprehensive, but still easy to comprehend. It also uses Python to create NN from scratch. 1. https://www.amazon.com/Make-Your-Own-Neural-Network/dp/1530826608 https://www.amazon.com/Make-Your-Own-Neural-Network/dp/15308...
- asdfman123 7y ago
  Also, Andrew Ng's course on Coursera is free if you want to really learn it and have a few weeks to throw at it.
  - cr0sh 7y ago
    I second this suggestion; I took that course when it was called "ML Class" during the Fall of 2011 (yep, I was one of the guinea pigs for what became one of the first courses of Coursera). It was an excellent course. Here's an example of what one student of the ML Class built, after being inspired by what he was learning and videos that played during the course: https://blog.davidsingleton.org/nnrccar/ https://blog.davidsingleton.org/nnrccar/ It kinda shocked me at the time, because I knew quite a bit about ALVINN from books and articles I had read as a teenager in the 80s and 90s. This guy had created the same thing using a cell phone and a cheap RC vehicle! Ok, there was also an Arduino and computer involved - but it really hit home the fact that technology around neural networks had advanced quite a bit! I also took the other course, "AI Class", but due to personal issues I had to drop out about halfway through. The next year, after Udacity started, they introduced a course similar to AI Class called "How to Build Your Own Self-Driving Vehicle" (it's called something else today - something like "Robotics and Artificial Intelligence 302" or something like that). That class was done in Python, and taught me even more about AI/ML - with a focus towards self-driving vehicles of course. Things I learned about that I struggled with or had no real concepts of before: 1. SLAM (Simultaneous Localization and Mapping) 2. Path Finding algorithms (A* and the like) 3. Kalman Filtering (what it is for, how it works) 4. PID Algorithm (how to implement and tune it) 5. More neural network stuff... ...and many other things. Another very excellent and free course to take if you're interested in learning this stuff.
_jsdw 7y ago
In case it helps, I also had a go at an introductory neural net tutorial which I probably never shared anywhere: https://jsdw.me/posts/neural-nets/ https://jsdw.me/posts/neural-nets/ I found that I had to read a bunch of these things to really grasp them myself.
markbnj 7y ago
Seems like a good intro and I plan to work through it later. I've been learning a lot from Michael Nielsen's book, available at http://neuralnetworksanddeeplearning.com/index.html http://neuralnetworksanddeeplearning.com/index.html. He doesn't shy away from the underlying math, and his appreciation for it comes through in the writing. Even without a strong math background I was able to punch through the notation and figure things out.
jorgeleo 7y ago
This tutorial explained to me at the exact level of detail: https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ https://mattmazur.com/2015/03/17/a-step-by-step-backpropagat... It was detailed enough for me to do all the calculations in an excel workbook, 1 complete cycle (forward, backward, and forward with the learned weights) https://1drv.ms/x/s!Ar06sKFtc9d7goR5WQLo-RkB0XvWAA https://1drv.ms/x/s!Ar06sKFtc9d7goR5WQLo-RkB0XvWAA Which allowed me to play with the name and factors to understand better how they impact the network as a whole.
- inertiatic 7y ago
  Having spent a lot of time hunting for the best way to figure out backprop, that is the best resource I've found and the one that finally made everything I've read click.
codesternews 7y ago
Why no biases?

174 points by shamdasani 7y ago

As someone who has read a lot of implementing neural networks from articles, the massive problem with all of them is that they import numpy. You may think that it is silly to reimplement the matrix math but with out that part of the code, you can't easily port it to other languages/microcontrollers/microwaves/badgers. It's a legitimately valid part of machine learning, and its not easy to do for novices. And I need help putting it on my badger damn it!

simias 7y ago
If you don't care much about performance (and if you are reimplementing a neural network from scratch you're probably doing it more as a learning project than anything else) implementing matrix operations isn't very difficult. If you're not used to work with matrices simply reading the Wikipedia article might tell you enough to implement them yourself.
- peterhj 7y ago
  If you have an assembler or C compiler you can implement matrix multiplication (GEMM) which usually does most of the heavy lifting in your neural net. Now you correctly alluded that it may not be simple to efficiently implement GEMM but if you have a simple architecture without a complex memory hierarchy then using whatever SIMD facilities are available and some standard tricks will get you in ballpark of peak FLOP/s. Or, just download a fast BLAS from your hardware vendor...
whatshisface 7y ago
The complicated parts of Numpy are themselves a wrapper for the seminal LAPACK: http://www.netlib.org/lapack/ http://www.netlib.org/lapack/. It has C language APIs, so that might help you with what you need to do.
- fartcannon 7y ago
  That's very interesting, thank you.
Anon84 7y ago
As someone who does teach tutorials as a side gig, I would argue that implementing matrix operations in a tutorial on neural networks is overkill. No matter what the level of the tutorial you always need to draw a line and assume a certain amount of background knowledge and knowing how to use standard tools isn't too much to ask. (yes, I know numpy isn't part of python's standard library, but it comes with pretty much any Python distribution as many other libraries depend on it.) If we're talking about a longer format, such as a book, then we might consider digging deeper and implementing as much as possible using the barest of Python requirements. Indeed, Joel Grus does implement everything from scratch in his great (although a bit dated) book https://www.amazon.com/Data-Science-Scratch-Principles-Python/dp/149190142X https://www.amazon.com/Data-Science-Scratch-Principles-Pytho.... EDIT: This is still a work in progress (and relies on numpy and matplotlib), but here is my version: https://github.com/DataForScience/DeepLearning https://github.com/DataForScience/DeepLearning These notebooks are meant as support for a webinar so they might not be the clearest as standalone, but you also have the slides there.
- HuShifang 7y ago
  A new edition of Grus comes out next week actually... https://www.amazon.com/Data-Science-Scratch-Principles-Python/dp/1492041130 https://www.amazon.com/Data-Science-Scratch-Principles-Pytho...
  - Anon84 7y ago
    Nice! He mentioned he was working on it when I met him at Strata last year, but I didn't know it was coming out already.
  - quantumOctopus 7y ago
    Ugh I just bought the old one a week ago.
- bigred100 7y ago
  I’d agree... Outside of very rare circumstances (specialist in numerical linear algebra implementations), my opinion is that implementing matrix operations is something you do once (twice) in your numerical courses to get an intuition for the algorithm, and then never again. But maybe it’s educational to do once if you never have before.
peterhj 7y ago
I'm curious if you have a particular language/microcontroller/microwave/badger you have in mind? Depending on which, YMMV.
- fartcannon 7y ago
  No, not off hand I don't. It's just something I've noticed in all these make a neural network posts. Feels a little like they're just drawing the rest of the owl, if you know what I mean. But thank you.
gnulinux 7y ago
Matrix math is easy peasy. Freshman level programming. Just lookup algorithms on Wikipedia and you're all set. The problem is it's extremely hard to make it efficient. Dozens of men-years are spent trying to optimize linear algebra libraries. There are handful linalg libraries that have competitive performance. It was my college project to make a fast linalg library, and boy it is fast. There are some things like matrix multiplication that if you implement in C with the trivial algorithm, takes >2 mins but with some tricks you can make it as fast as <second (vectorization, OpenMP, handwritten assembly, automatically optimized code, various optimizations, better algorithm.....). So, if you want to implement linalg in some language and compile it, go ahead, more power to you. But it's basically impossible to do it efficiently. My opinion is: this is fine and we should do this. There should be linalg libraries written in pure python (and are 1000x slower than lapack) but just understand that it's impossible to satisfy all use cases of numpy this way (at least currently).
- asdfman123 7y ago
  Can anyone simply explain the gist of how matrix multiplication is optimized? I know a lot of is farmed out to the GPU (if you've got a good GPU), but what's the essence of it? Caching? Some kind of clever mathematical tricks? All of the above?
  - johndough 7y ago
    The answer is: No one really knows because cuBLAS is closed source. But to get within the same order of magnitude, tiling the workload for better cache utilization is usually the most important step. This article [1] explains it quite well and also lists a few other tricks. In addition, there's also the fast Fourier transform for large filter kernels and Winograd convolutions [2] for small filter kernels. [1] https://cnugteren.github.io/tutorial/pages/page1.html https://cnugteren.github.io/tutorial/pages/page1.html [2] https://arxiv.org/pdf/1509.09308.pdf https://arxiv.org/pdf/1509.09308.pdf
    - tntn 7y ago
      Not entirely true - Scott Gray knows: https://github.com/NervanaSystems/maxas/wiki/SGEMM https://github.com/NervanaSystems/maxas/wiki/SGEMM IIRC his kernels shipped in cuBLAS at some point.
asdfman123 7y ago
It is easy to do unless you don't code at all, or are completely confused by the math. I'm a C# developer and I'm sure it would take me all of about 30 seconds to install a matrix multiplication package through nuget. I'm sure it would be immediately obvious how to add items to matrices or do a dot product.
- animal531 7y ago
  I'm a C# developer who wrote my own implementation (with help from random tutorials etc). It was dead easy to get code examples as needed.
pilooch 7y ago
Exactly the reason why my colleagues and myself do all deep learning in C++, performance and portability, from cloud to RPie. We've even modified caffe2 so we could build the training graph from pure C++. We know this is not the current doxa :) It's also all open sourced just in case others might need it...
- fartcannon 7y ago
  Link? I love C++ and would love to see it.
felipellrocha 7y ago
Yep. I would like to see an article that implements everything without using matrices first, then creates the matrices library with you, and refactors everything over. So much learning that we're missing by not going through this step.
- cr0sh 7y ago
  A good course that comes close to this would be the Coursera Machine Learning course (what used to be known as "ML Class" by Andrew Ng). It uses Octave - but you first do everything (in the section on NN) "by hand" - building and looping for the matrix operations. Only after you've gone that far, does he (Ng) introduce the fact that Octave has vector/matrix primitives... I took the original ML Class in the Fall of 2011; it was a great class, and opened my eyes a great deal on the topic of machine learning and neural networks, which I had struggled with understanding in the past (mainly on what and how backprop worked).
- fartcannon 7y ago
  To me it feels a bit like that joke about drawing instructions. "1. Draw some circles. 2. Now draw the rest of the owl."
cscheid 7y ago
Hm. I just finished teaching an ML course where all of the assignments were pure Python (on purpose, so students would actually have the chance to see all of the code). One of the assignments included implementing reverse-mode autodiff and a NN classifier on top. It can be done in ~600 lines of clear python, serious!
- fartcannon 7y ago
  Which course?
  - cscheid 7y ago
    https://cscheid.net/courses/spr19/csc665/ https://cscheid.net/courses/spr19/csc665/ The assignments are not directly available, but my email is easy to find.
danlugo92 7y ago
http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/

melling 7y ago

Here's another Neural Network from scratch that I found useful: https://victorzhou.com/blog/intro-to-neural-networks/ https://victorzhou.com/blog/intro-to-neural-networks/

samsonradu 7y ago
Thanks a lot for this, it is indeed very clear and easy to follow! Good walkthrough on the partial derivatives calculations which imo are the hardest part.

rrggrr 7y ago

Would be great if this included real world data or application to understand context.

cwt137 7y ago

If you think this blog article is lacking, get "Make Your Own Neural Network" by Tariq Rashid[1]. It is way more comprehensive, but still easy to comprehend. It also uses Python to create NN from scratch. 1. https://www.amazon.com/Make-Your-Own-Neural-Network/dp/1530826608 https://www.amazon.com/Make-Your-Own-Neural-Network/dp/15308...

asdfman123 7y ago
Also, Andrew Ng's course on Coursera is free if you want to really learn it and have a few weeks to throw at it.
- cr0sh 7y ago
  I second this suggestion; I took that course when it was called "ML Class" during the Fall of 2011 (yep, I was one of the guinea pigs for what became one of the first courses of Coursera). It was an excellent course. Here's an example of what one student of the ML Class built, after being inspired by what he was learning and videos that played during the course: https://blog.davidsingleton.org/nnrccar/ https://blog.davidsingleton.org/nnrccar/ It kinda shocked me at the time, because I knew quite a bit about ALVINN from books and articles I had read as a teenager in the 80s and 90s. This guy had created the same thing using a cell phone and a cheap RC vehicle! Ok, there was also an Arduino and computer involved - but it really hit home the fact that technology around neural networks had advanced quite a bit! I also took the other course, "AI Class", but due to personal issues I had to drop out about halfway through. The next year, after Udacity started, they introduced a course similar to AI Class called "How to Build Your Own Self-Driving Vehicle" (it's called something else today - something like "Robotics and Artificial Intelligence 302" or something like that). That class was done in Python, and taught me even more about AI/ML - with a focus towards self-driving vehicles of course. Things I learned about that I struggled with or had no real concepts of before: 1. SLAM (Simultaneous Localization and Mapping) 2. Path Finding algorithms (A* and the like) 3. Kalman Filtering (what it is for, how it works) 4. PID Algorithm (how to implement and tune it) 5. More neural network stuff... ...and many other things. Another very excellent and free course to take if you're interested in learning this stuff.

_jsdw 7y ago

In case it helps, I also had a go at an introductory neural net tutorial which I probably never shared anywhere: https://jsdw.me/posts/neural-nets/ https://jsdw.me/posts/neural-nets/ I found that I had to read a bunch of these things to really grasp them myself.

markbnj 7y ago

Seems like a good intro and I plan to work through it later. I've been learning a lot from Michael Nielsen's book, available at http://neuralnetworksanddeeplearning.com/index.html http://neuralnetworksanddeeplearning.com/index.html. He doesn't shy away from the underlying math, and his appreciation for it comes through in the writing. Even without a strong math background I was able to punch through the notation and figure things out.

jorgeleo 7y ago

This tutorial explained to me at the exact level of detail: https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ https://mattmazur.com/2015/03/17/a-step-by-step-backpropagat... It was detailed enough for me to do all the calculations in an excel workbook, 1 complete cycle (forward, backward, and forward with the learned weights) https://1drv.ms/x/s!Ar06sKFtc9d7goR5WQLo-RkB0XvWAA https://1drv.ms/x/s!Ar06sKFtc9d7goR5WQLo-RkB0XvWAA Which allowed me to play with the name and factors to understand better how they impact the network as a whole.

inertiatic 7y ago
Having spent a lot of time hunting for the best way to figure out backprop, that is the best resource I've found and the one that finally made everything I've read click.

codesternews 7y ago

Why no biases?