22 ms·

Implementing a Neural Network from Scratch in Python

616 points by vzhou842 8y ago

fredley 8y ago
My first reading of this headline was "Implementing a Neural Network in Scratch". Now that I would like to see. https://scratch.mit.edu/ https://scratch.mit.edu/
- dbcurtis 8y ago
  That would be a challenge. Back in the heyday of expert systems and original Mac’s, I did an expert system in Hypercard. Not really production worthy. But interesting from perspective of exploring UI.
  - fit2rule 8y ago
    Still got it somewhere? I've got a project to demonstrate a bunch of Hypercard stacks in a retro-computing context, and it'd be definitely interesting to see, in the case that you'd consider finding it and sharing it ...
    - dbcurtis 8y ago
      Oh gosh.... that was so many years ago. I suspect I could never find the floppy, assuming I haven't chucked it. All I can remember about it is that it was structured to ask the user one query per card, and there was some kind of global function that kept the state of the consultation and navigated to the next card. The expert system that I implemented was a simple toy, because the point was to explore the UI aspects. For instance, you could easily embed images and ask "Click the image that most closely resembles your specimen." Stuff like that, intermixed with traditional text queries.
      - fit2rule 8y ago
        Well, it was worth the try .. sounds interesting though! There's a resurgent interest in Hypercard these days .. if you ever get the druthers to go look for your old stack, I'm sure it'll be of interest to some of us. :)
- emilwallner 8y ago
  Ha, that's an excellent idea. Someone's already started by creating a program for matrix multiplication: https://scratch.mit.edu/projects/18640517/ https://scratch.mit.edu/projects/18640517/
- raimondious 8y ago
  Here's one! https://scratch.mit.edu/projects/26682825/ https://scratch.mit.edu/projects/26682825/
inputcoffee 8y ago
There are so many little details to remember when you implement a Neural Network from "scratch". Or, I suppose, even if you do not. You know what would be a great contribution? An extensive set of unit tests, or even just problems with solutions. That way people can write their own implementations and test them. And even if a person were to implement the net in Pytorch of Tensorflow, they could test the work. So there would be a matrix of weights, and a vector of input nodes, and the "answer" would be the output vector. Then there would be another "answer" which is the output with a particular activation function, and so on. This library would just be there so people who are doing their own implementation can test their work. As I said, a unit test would work too but then it would have to be language specific. Just the matrix and answer would be language agnostic. For people who think: can't you just make up an example yourself using a sheet of paper or in excel. Yes, for most purposes this is fine, but if you forget one little implementation detail of a three layer network with a ReLu, you really want an external way to check that.
- muttled 8y ago
  Built-in testing would be fantastic. Being able to tell if you designed a model wrong or just made an error setting up the code would reduce frustration a ton!
  - yorwba 8y ago
    If you're implementing your own derivatives, you should probably use numerical methods to check that the gradient is computed correctly, and pytorch comes with tools to help with that https://pytorch.org/docs/stable/autograd.html#numerical-gradient-checking https://pytorch.org/docs/stable/autograd.html#numerical-grad... If you're reimplementing something for learning purposes, you can just compare the output you get with the existing implementation. But as soon as you're trying to do something novel, no automatic test can tell you whether the model architecture or your implementation of it is at fault when you don't get the results you'd like, because there's nothing you could use for comparison.
- sgillen 8y ago
  That would be a great contribution but unfortunately I think not an easy one to provide. A few issues 1. Most BLAS libraries aren't deterministic. Floating point numbers aren't in general associative, (a + b) + c =/= a + (b + c). However most BLAS libraries will happily shuffle the order of operations around if it saves some CPU cycles, I've seen this issue cause huge differences in behavior between different CPUs before, although I think that was a pathological case. So if you want to do a test like this you really need to have everyone on the same page an installing a deterministic BLAS. 2. Training neural networks is stochastic I know you are talking about just the forward pass but I think it would be interesting to have these tests for training your network as well. Naively this sounds like it might be easy to fix (just provide a seed!), but gets very tricky when you want something to output the same code over multiple libraries, operating systems, hardware etc. Multithreading code, putting stuff on the GPU or even the cloud add even more difficulty.
lowpro 8y ago
This is a great write up! I'm in the process of writing a similar article, although I think for many non-programmers, neural networks can be useful through using algorithms that are already made to solve problems where people have the data but don't understand the math. This article is good if you've been around some math and have a little coding experience, I think Neural Nets are even more accessible than the author is making them out to be however. For example, using something like YOLO for object detection can be done even if you've never taken algebra or a programming class, and really just requires some configuration and training on a dataset you have.
rrggrr 8y ago
Great, but the problem with these summaries is no real-world data/use-case. An actual business case example with code is easier to comprehend for less math-inclined folks.
- jackallis 8y ago
  i agree, especially for beginners.
- deleted 8y ago
  [deleted]
- cr0sh 8y ago
  The problem with that is these kinds of simplified "how it works" neural net examples don't scale for real world problems - especially when using an interpreted language like Python. Even a simple but useful scenario like the MNIST number recognition system would probably run "dog-slow" on such a neural network, given the number of input nodes, plus the size of the hidden layer - the combinatorial "explosion" of edges in the graph, all the processing needed during backprop and forwardprop... It might be doable with C/C++ - but it still won't run as well as it could. These examples are really just meant as teaching tools; they are the bare-bones-basics of neural networks, to get you to understand the underlying mechanisms at work (actually, the XOR network is the real bare-bones NN example, because it requires so few nodes, that it can be worked out by pencil and paper methods). Real-world implementations are best done using more advanced libraries and tools, like Tensorflow and Keras, or similar (and moving to C/C++ and/or CUDA if even more processing power is needed).
  - ldng 8y ago
    Maybe you missed the disclaimer ? "The code below is intended to be simple and educational, NOT optimal."
- Shorel 8y ago
  With neural networks, real-world data and real-world use-cases have expensive hardware requirements. You can, however, try this service for free during the first two months: https://aws.amazon.com/blogs/aws/sagemaker/ https://aws.amazon.com/blogs/aws/sagemaker/ (I am not affiliated with AWS and I have not used SageMaker) You can also read the DeepMind blog to find out the latest state-of-the-art real-world usage of Neural Networks.
jackallis 8y ago
how come every information like this, intended for beginners, don't say why you even need NN or when would you even use it.
- cr0sh 8y ago
  That is a good question - I will attempt to answer it for you (probably poorly): First off, for most problems you don't need a neural network - many, many problems can be solved using simple statistical regression techniques - what could be called "classical machine learning". It's when your data starts to involve more than a few variables (usually more than 2 or 3) that you need an answer, and/or when you don't know exactly what the best "classical" machine learning technique to use, is when you may find a neural network to be a potential solution. Ultimately, what a regular neural network does (I'm not going to get into anything like GANs or CNNs or anything special like that - though arguably they all work in a similar manner) is come up with a solution that encapsulates and encodes, through "learning", the proper algorithm(s) needed to provide a solution for the problem - whether that is classification or something else. It's basically the classic argument that all a neural network is, is a complicated form of linear regression. And to an extent, that is true - but it's done in a "black box" manner, which may or may not be important to you for your problem (the issue becomes "how does it do what it does?" - and that's where things can become tricky to answer). Neural networks basically take information - various attributes of a problem set - and then, once trained, can output an answer for an example (that it hasn't seen before!) that fits that problem set criteria. It does this by having been "trained" by seeing a lot of different examples, with each example matched up to a "labeled" output. Depending on whether it is right or wrong for the labeled output, it will then propagate the error backwards through the network to correct things by a small amount, and try again (note: your training data needs to be extremely varied, and should include both positive examples and negative examples, and shouldn't be biased toward any particular set of examples). Anyhow - your data must ultimately be represented by numbers in some manner, usually continuous, and can be anything from a single input to multiple inputs. For instance, you might have the single input of "temperature" to control the output of "turn on/off the furnace" - ie, modeling a thermostat (note that you would never do this actually, outside of learning how a simple perceptron works I guess). Usually you don't use a single continuous input - you would use multiple inputs - maybe a date, coupled with gps coordinates (like say for housing prices), with an output of "price". Or maybe 10,000 grayscale values (0.00-1.00), representing a 100x100 pixel image, with a classification output of the numbers 0-9 and letters A-Z (36 symbols). In other words, you are trying to figure out the answer to a problem (is this a picture of a letter or number?) that would be difficult or impossible to code a set of rules by hand to answer with any statistical certainty. Note that last part: Neural networks do not give you an absolute result; they output a continuous value or a set of values that represent the most probable likelihood of a correct answer based upon what the model has learned in the past. In the case of a single output, that could represent a "yes/no" or "true/false" answer (a value between 1 and 0 respectively); it could also be "left/right", "up/down" or something similar if controlling direction/pressure/flow rate, etc based on inputs. For classification, it will be a set of continuous values, indicating for each "class" which is the higher probability (0-1) of being correct. For instance, if you had three categories like: Monkey: 0.1 Banana: 0.8 Coconut: 0.2 Then it is likely that the picture was of a banana. But if you saw this: Monkey: 0.7 Banana: 0.9 Coconut: 0.1 Maybe the picture contained a monkey holding a banana? It might be possible - even if the network was never trained on that particular imagery! Note that it is also possible to fool such a network, as numerous studies have shown. It's also possible to have the network output an image (ie - by having it output a very large array of node values that represents the pixel values of the image); this is how neural networks generate or alter images. Sound data can also be done in a similar manner. Alright - I think I am rambling a bit, so I'm going to leave it here; I hope this answers at least something of your questions (and I hope it didn't confuse you - if so, I apologize and that wasn't my intent).
11thEarlOfMar 8y ago
"We’ll use an optimization algorithm called stochastic gradient descent (SGD) that tells us how to change our weights and biases to minimize loss. It’s basically just this update equation: w1 ← w1 − η ∂w|∂L η is a constant called the learning rate that controls how fast we train. All we’re doing is subtracting η ∂w|∂L from w1: - If ∂L | ∂w1 is positive, w1 will decrease, which makes L decrease - If ∂L | ∂w1 is positive, w1 will increase, which makes L decrease" Simple indeed.
- rerx 8y ago
  ?
- joshklein 8y ago
  I think this comment is facetious (apologies if I’m reading it incorrectly), so I want to offer that this is “simple”, but opaque to someone who is unfamiliar with the syntax or background context. Andrew Ng’s course on machine learning from Stanford (on Coursera, which you should be able to audit for free) gives great intuitive explanations of these topics, if you’re curious enough to invest a couple solid weekends to get the foundations. The foundations alone will get you very far indeed.
  - cr0sh 8y ago
    Seconding Ng's course here. I took it in 2011 (ML Class) before Coursera existed, and it finally opened my eyes on not only what and how backprop worked, but also how everything in a NN could be represented and calculated using vectors and matrices (linear algebra), and how that process was "parallelizable". The course uses Octave as it's programming environment, which is essentially an open-source and (mostly) compatible implementation of Matlab. My first thought was "Finally! A use case for a home Beowulf cluster that is somewhat practical!" It really opened my eyes and mind to a number concepts that I had looked into before, but couldn't quite wrap my brain around completely.
- profunctor 8y ago
  I think it's supposed to be dL/dw1. As in the loss function derived wrt w1.
kowdermeister 8y ago
I started playing around with NN-s recently and this looked like a good tutorial until the partial derivatives :) It would have been so much better to turn a complex math concept like that to code in smaller chunks because it's hard to extract from the full sample which part refers to which equation. Also, the bias is not explained at all. - Why is it there? - What does it do? - What value should it have?
- dragandj 8y ago
  I currently publish a series of detailed articles that cover exactly that. https://dragan.rocks/articles/19/Deep-Learning-in-Clojure-From-Scratch-to-GPU-0-Why-Bother https://dragan.rocks/articles/19/Deep-Learning-in-Clojure-Fr...
  - victor106 8y ago
    dragon.rocks, rocks. Your articles are some of the best i have seen on this topic. Thank you. I hope you write a book on this topic.
    - dragandj 8y ago
      Thanks! We'll see. BTW, its dragAn, not dragOn :)
- xeRTRex 8y ago
  Agreed, it is a little confusing. What makes it even more confusing is bias can be introduced as a regularization technique (to address overfitting) and these are not the same thing. Bias in this context can be simply interpreted as analogous to the intercept in y = mx + b. Kind of like the baseline when no additional information is given.
- applecrazy 8y ago
  Partial derivatives are basically how much a multivariate function changes based on a tiny change in a single variable. The actual math is irrelevant.
alexlchen2019 8y ago
amazing read, good content
- noperks 8y ago
  probably the most easy going read-through i've seen
eam 8y ago
I didn't have time to read all of it, but half way through. So far it has bee a pretty good write up, however, just one question. Can the author or anyone explain how were the amount for values shifted chosen? I.e. Weight (minus 135) and Height (minus 66). Why was -135 and -66 chosen? An explanation would be helpful. Thanks!
- jventura 8y ago
  I think they were chosen such that the weight and height values become negative for Females and positive for Males.
- xeRTRex 8y ago
  I haven't read the article thoroughly but I suspect part of the data preprocessing step included centering and standardizing. Those must be the mean weight and height values.
- LodeOfCode 8y ago
  "I arbitrarily chose the shift amounts (135 and 66) to make the numbers look nice. Normally, you’d shift by the mean."
  - eam 8y ago
    I am pretty sure that note wasn't written there before. The author must have updated the article after I posted this question. Though "nice" is still somewhat vague to me. Nice in what way? Smaller numbers are nicer? Negative and positives make it nice? The second part of the answer is more useful information, "Normally, you’d shift by the mean."
- emgee_1 8y ago
  I think this a form of normalizing a normal distribution into a standard normal distribution ( a normal distribution has two parameters \mu and \sigma which in standard form gives respectively 0 and 1. Any normal distribution is a special case of the standard normal distribution N(0,1)
avmich 8y ago
> Real neural net code looks nothing like this. This negates a lot of promise of the article. If one understands principles, but needs to know how to apply them, this statement makes the article practically useless. Usually after reading like this goes a jump in explanations, which directs to apply TensorFlow et. al. and even, if you're lucky, explains how to do that - but doesn't explain what it is that TensorFlow does qualitatively different, which would justify putting such a phrase as a warning.
- sgillen 8y ago
  I still think it's a useful exercise for people like me who learn best by implementing a thing. Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area.
  - avmich 8y ago
    > I still think it's a useful exercise for people like me who learn best by implementing a thing. Yes, but if you implement one thing, and then TensorFlow which you plan to use does something very different, what's the point? Ok, different enough to justify the warning. > Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area. Exactly. It's that magic which is interesting, after you're shown how in principle - but only in principle - the thing could work.
    - sgillen 8y ago
      I guess the point here is that in this article we implement some of the magic that the higher level frameworks use, which I think will be useful later when writing and debugging our high level code. But I see your point which is that it would be nice if the article was telling us which part of the high level library we're implementing and what that would look like. I'm still happy with the article though and would not call it almost useless. But hey if you'd rather jump right into a tensorkerastorch tutorial that's an option too!
      - 8y ago
gambler 8y ago
Why are deep learning researchers allergic to meaningful variable names?
- chrisfosterelli 8y ago
  These are meaningful variable names in the context of a code implementation of a math algorithm. The variables directly map to the algorithmic notation.
  - jacobolus 8y ago
    And when trying to do algebraic manipulation on paper, having an 8 or 10 letter variable name is incredibly cumbersome. Frankly in the middle of a numerical algorithm it is typically also cumbersome in code to have descriptive variable names for everything. However, mathematical code (especially when written by scientists, etc.) often takes this too far, introducing many 1- or few-letter variable names without enough context or description to figure out what they stand for, and for more of the variables than necessary. This blog post was just fine though. h = hidden, o = output, y_pred = predicted value of y, etc. are quite clear
    - gambler 8y ago
      >h = hidden, o = output, y_pred = predicted value of y, etc. are quite clear ∀ abbr + to the cogntv load prsn wht mntn in thr hd. ∀ -> every + -> adds wht -> who has to h -> head Was that easier to read than the following? Every abbreviation adds to the cognitive load the person has to maintain in their head. This isn't code, but the principles of making code (un)readable are exactly the same. If you see that full-word variable names create too much clutter, it means your code structure is wrong. So you fix it. You go from this: self.w1 = np.random.normal() self.w2 = np.random.normal() self.w3 = np.random.normal() self.w4 = np.random.normal() self.w5 = np.random.normal() self.w6 = np.random.normal() To this: set_to_random_numbers(self.weights) After 9 years of maintaining several dozen legacy codebases (which sometimes involved financial math) I am fully convinced that the only people who like terse code are people who write it and never read it after.
fartcannon 8y ago
It's fun, but it's cheating a bit if you use numpy. Can someone implement one with all the matrix math from scratch too?
- siekmanj 8y ago
  shameless self-promotion: https://github.com/siekmanj/sieknet https://github.com/siekmanj/sieknet
  - fartcannon 8y ago
    Looks great. Even got yourself a genetic algorithm in there, eh? How well does it work?
    - siekmanj 8y ago
      The genetic algorithm currently isn't in a working state unfortunately. I did a big architectural rewrite recently and haven't gotten around to GA stuff - but I'm looking forward to applying some more modern results in neuroevolution from the last two or three years.
- archgoon 8y ago
  I don't see why using matrix math shouldn't be viewed as a basic primitive. Implementing a matrix library would obscure the point that is being made.
posix_compliant 8y ago
Friendly comment: at the beginning of section 4 there's this sum: 1 ∑(y_true - y_pred)^2 i=1 There's no 'i' in the sum, should probably be y_true_i - y_pred_i.
- posix_compliant 8y ago
  I do get that in the example given there's only one element in each of the true/predicted vectors, but it threw me off for a sec.
amasad 8y ago
Runnable code from the article: https://repl.it/@vzhou842/An-Introduction-to-Neural-Networks https://repl.it/@vzhou842/An-Introduction-to-Neural-Networks I see so many implementations of NeuralNets from Scratch on Repl.it (I'm a co-founder) but this is one of the simplest and clearest one. One that blew my mind recently is one by a 12 year old programmer: https://repl.it/@pyelias/netlib https://repl.it/@pyelias/netlib
syntaxing 8y ago
If you're interested in learning something like this, I highly recommend Udacity's deep learning nanodegree. They have a section that teaches you how to build your own neural network with the the help of numpy. They even have a section where you write your own sentimental analysis neural network from scratch. Easily the best explanation and notebook to follow I have seen so far. I've been taking their revamped one that focuses on PyTorch rather than Tensorflow. Disclaimer: I am taking the deep learning nanodegree degree for free but its sponsored by Facebook rather than Udacity directly.
- applecrazy 8y ago
  The notebooks are available on GitHub for free.
laneb 8y ago
I appreciate the author showing some of the math that drives the optimization. As a reader, I'd suggest introducing SGD ahead of the PD calculations, because it would give a clearer motivation for the PDs.
arendtio 8y ago
Recently, I watched a Scrimba tutorial [1] about brain.js. That one brought me quite a step further to understand how different kinds of neural networks are capable of (NN vs. RNN vs. CNN). [1]: https://scrimba.com/g/gneuralnetworks https://scrimba.com/g/gneuralnetworks
ddebernardy 8y ago
One of the nice parts of NNs is that it takes some maths education to understand what's going on. My initial thought when reading the title was: "uh oh, are AI tutorials going to be the next PHP/MySQL tutorials, with god awful code all over the internet?" They haven't yet insofar as I'm aware. I hope the maths involved will prevent them from becoming that.
- deytempo 8y ago
  Luckily math isn’t actually required to implement them in a series of arrays so that little relief will be short lived.
yreaderynotread 8y ago
The derive of sigmoid is exp(-x) / (1+exp(-x))² and not exp(x) / (1+exp(-x))²
ujuj 8y ago
See this code posted on HN a month ago too: https://gist.github.com/macournoyer/620a8ba4a2ecd6d6feaf https://gist.github.com/macournoyer/620a8ba4a2ecd6d6feaf I found the code very clear and understandable.
gillesjacobs 8y ago
This is very similar to Andrew Ng's Deep Learning Specialization course [1]. If you found this blog post enjoyable be sure to check it out. It is a great course and the intuitions behind NNs are explained very clearly. 1. https://www.coursera.org/specializations/deep-learning https://www.coursera.org/specializations/deep-learning
iamaziz 8y ago
The visualization is cool, thanks! Here's another neural network example from scratch in 13 lines of Python: https://iamtrask.github.io/2015/07/27/python-network-part2/ https://iamtrask.github.io/2015/07/27/python-network-part2/
buildbot 8y ago
Since this is based on numpy, you could quickly have it GPU accelerated by using cupy: https://cupy.chainer.org/ https://cupy.chainer.org/
starpilot 8y ago
So many examples of this: https://hn.algolia.com/?query=python%20neural%20scratch&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=python%20neural%20scratch&sort... Is this example better than previous ones?
techbio 8y ago
Very cool. I want to give a shout out to the simple classic http://arctrix.com/nas/python/bpnn.py http://arctrix.com/nas/python/bpnn.py
cgopalan 8y ago
I took the Pytorch scholarship challenge this past December with Udacity. I couldn't make it to the nanodegree scholarship but the challenge course was really good in exposing me to the various facets of deep learning. The notebooks are free, but the videos really help explain some concepts well. For people who took the course, the videos are available for another year, so that helps. I wanted to get a good grasp of DL but was floundering around before with so many sources and tutorials - and frameworks like Keras, Pytorch, TensorFlow etc. This course helped me make some decisions - like if I needed to get a developer's handle on DL, I would have to know Perceptrons, CNNs, RNNs, Style Transfer, Sentiment Prediction and GANs (even though GANs were not part of the course). And Pytorch would be my tool of choice.
deleted 8y ago
[deleted]
theblackcat1002 8y ago
I think a better and elegant way to implement is the pytorch way. By defining a forward and backward function for easy chaining, you can easily chain multiple layers into a single model. Placing the training loop inside the layer will be hard to write a multi layer version. Just wrote a similar from scratch code few months back: https://github.com/theblackcat102/DL_2018/blob/master/HW2/layers.py https://github.com/theblackcat102/DL_2018/blob/master/HW2/la...
ecmascript 8y ago
This is very cool and informative. Thank you for this article!

616 points by vzhou842 8y ago

My first reading of this headline was "Implementing a Neural Network in Scratch". Now that I would like to see. https://scratch.mit.edu/ https://scratch.mit.edu/

dbcurtis 8y ago
That would be a challenge. Back in the heyday of expert systems and original Mac’s, I did an expert system in Hypercard. Not really production worthy. But interesting from perspective of exploring UI.
- fit2rule 8y ago
  Still got it somewhere? I've got a project to demonstrate a bunch of Hypercard stacks in a retro-computing context, and it'd be definitely interesting to see, in the case that you'd consider finding it and sharing it ...
  - dbcurtis 8y ago
    Oh gosh.... that was so many years ago. I suspect I could never find the floppy, assuming I haven't chucked it. All I can remember about it is that it was structured to ask the user one query per card, and there was some kind of global function that kept the state of the consultation and navigated to the next card. The expert system that I implemented was a simple toy, because the point was to explore the UI aspects. For instance, you could easily embed images and ask "Click the image that most closely resembles your specimen." Stuff like that, intermixed with traditional text queries.
    - fit2rule 8y ago
      Well, it was worth the try .. sounds interesting though! There's a resurgent interest in Hypercard these days .. if you ever get the druthers to go look for your old stack, I'm sure it'll be of interest to some of us. :)
emilwallner 8y ago
Ha, that's an excellent idea. Someone's already started by creating a program for matrix multiplication: https://scratch.mit.edu/projects/18640517/ https://scratch.mit.edu/projects/18640517/
raimondious 8y ago
Here's one! https://scratch.mit.edu/projects/26682825/ https://scratch.mit.edu/projects/26682825/

inputcoffee 8y ago

There are so many little details to remember when you implement a Neural Network from "scratch". Or, I suppose, even if you do not. You know what would be a great contribution? An extensive set of unit tests, or even just problems with solutions. That way people can write their own implementations and test them. And even if a person were to implement the net in Pytorch of Tensorflow, they could test the work. So there would be a matrix of weights, and a vector of input nodes, and the "answer" would be the output vector. Then there would be another "answer" which is the output with a particular activation function, and so on. This library would just be there so people who are doing their own implementation can test their work. As I said, a unit test would work too but then it would have to be language specific. Just the matrix and answer would be language agnostic. For people who think: can't you just make up an example yourself using a sheet of paper or in excel. Yes, for most purposes this is fine, but if you forget one little implementation detail of a three layer network with a ReLu, you really want an external way to check that.

muttled 8y ago
Built-in testing would be fantastic. Being able to tell if you designed a model wrong or just made an error setting up the code would reduce frustration a ton!
- yorwba 8y ago
  If you're implementing your own derivatives, you should probably use numerical methods to check that the gradient is computed correctly, and pytorch comes with tools to help with that https://pytorch.org/docs/stable/autograd.html#numerical-gradient-checking https://pytorch.org/docs/stable/autograd.html#numerical-grad... If you're reimplementing something for learning purposes, you can just compare the output you get with the existing implementation. But as soon as you're trying to do something novel, no automatic test can tell you whether the model architecture or your implementation of it is at fault when you don't get the results you'd like, because there's nothing you could use for comparison.
sgillen 8y ago
That would be a great contribution but unfortunately I think not an easy one to provide. A few issues 1. Most BLAS libraries aren't deterministic. Floating point numbers aren't in general associative, (a + b) + c =/= a + (b + c). However most BLAS libraries will happily shuffle the order of operations around if it saves some CPU cycles, I've seen this issue cause huge differences in behavior between different CPUs before, although I think that was a pathological case. So if you want to do a test like this you really need to have everyone on the same page an installing a deterministic BLAS. 2. Training neural networks is stochastic I know you are talking about just the forward pass but I think it would be interesting to have these tests for training your network as well. Naively this sounds like it might be easy to fix (just provide a seed!), but gets very tricky when you want something to output the same code over multiple libraries, operating systems, hardware etc. Multithreading code, putting stuff on the GPU or even the cloud add even more difficulty.

lowpro 8y ago

This is a great write up! I'm in the process of writing a similar article, although I think for many non-programmers, neural networks can be useful through using algorithms that are already made to solve problems where people have the data but don't understand the math. This article is good if you've been around some math and have a little coding experience, I think Neural Nets are even more accessible than the author is making them out to be however. For example, using something like YOLO for object detection can be done even if you've never taken algebra or a programming class, and really just requires some configuration and training on a dataset you have.

rrggrr 8y ago

Great, but the problem with these summaries is no real-world data/use-case. An actual business case example with code is easier to comprehend for less math-inclined folks.

jackallis 8y ago
i agree, especially for beginners.
deleted 8y ago
[deleted]
cr0sh 8y ago
The problem with that is these kinds of simplified "how it works" neural net examples don't scale for real world problems - especially when using an interpreted language like Python. Even a simple but useful scenario like the MNIST number recognition system would probably run "dog-slow" on such a neural network, given the number of input nodes, plus the size of the hidden layer - the combinatorial "explosion" of edges in the graph, all the processing needed during backprop and forwardprop... It might be doable with C/C++ - but it still won't run as well as it could. These examples are really just meant as teaching tools; they are the bare-bones-basics of neural networks, to get you to understand the underlying mechanisms at work (actually, the XOR network is the real bare-bones NN example, because it requires so few nodes, that it can be worked out by pencil and paper methods). Real-world implementations are best done using more advanced libraries and tools, like Tensorflow and Keras, or similar (and moving to C/C++ and/or CUDA if even more processing power is needed).
- ldng 8y ago
  Maybe you missed the disclaimer ? "The code below is intended to be simple and educational, NOT optimal."
Shorel 8y ago
With neural networks, real-world data and real-world use-cases have expensive hardware requirements. You can, however, try this service for free during the first two months: https://aws.amazon.com/blogs/aws/sagemaker/ https://aws.amazon.com/blogs/aws/sagemaker/ (I am not affiliated with AWS and I have not used SageMaker) You can also read the DeepMind blog to find out the latest state-of-the-art real-world usage of Neural Networks.

jackallis 8y ago

how come every information like this, intended for beginners, don't say why you even need NN or when would you even use it.

cr0sh 8y ago
That is a good question - I will attempt to answer it for you (probably poorly): First off, for most problems you don't need a neural network - many, many problems can be solved using simple statistical regression techniques - what could be called "classical machine learning". It's when your data starts to involve more than a few variables (usually more than 2 or 3) that you need an answer, and/or when you don't know exactly what the best "classical" machine learning technique to use, is when you may find a neural network to be a potential solution. Ultimately, what a regular neural network does (I'm not going to get into anything like GANs or CNNs or anything special like that - though arguably they all work in a similar manner) is come up with a solution that encapsulates and encodes, through "learning", the proper algorithm(s) needed to provide a solution for the problem - whether that is classification or something else. It's basically the classic argument that all a neural network is, is a complicated form of linear regression. And to an extent, that is true - but it's done in a "black box" manner, which may or may not be important to you for your problem (the issue becomes "how does it do what it does?" - and that's where things can become tricky to answer). Neural networks basically take information - various attributes of a problem set - and then, once trained, can output an answer for an example (that it hasn't seen before!) that fits that problem set criteria. It does this by having been "trained" by seeing a lot of different examples, with each example matched up to a "labeled" output. Depending on whether it is right or wrong for the labeled output, it will then propagate the error backwards through the network to correct things by a small amount, and try again (note: your training data needs to be extremely varied, and should include both positive examples and negative examples, and shouldn't be biased toward any particular set of examples). Anyhow - your data must ultimately be represented by numbers in some manner, usually continuous, and can be anything from a single input to multiple inputs. For instance, you might have the single input of "temperature" to control the output of "turn on/off the furnace" - ie, modeling a thermostat (note that you would never do this actually, outside of learning how a simple perceptron works I guess). Usually you don't use a single continuous input - you would use multiple inputs - maybe a date, coupled with gps coordinates (like say for housing prices), with an output of "price". Or maybe 10,000 grayscale values (0.00-1.00), representing a 100x100 pixel image, with a classification output of the numbers 0-9 and letters A-Z (36 symbols). In other words, you are trying to figure out the answer to a problem (is this a picture of a letter or number?) that would be difficult or impossible to code a set of rules by hand to answer with any statistical certainty. Note that last part: Neural networks do not give you an absolute result; they output a continuous value or a set of values that represent the most probable likelihood of a correct answer based upon what the model has learned in the past. In the case of a single output, that could represent a "yes/no" or "true/false" answer (a value between 1 and 0 respectively); it could also be "left/right", "up/down" or something similar if controlling direction/pressure/flow rate, etc based on inputs. For classification, it will be a set of continuous values, indicating for each "class" which is the higher probability (0-1) of being correct. For instance, if you had three categories like: Monkey: 0.1 Banana: 0.8 Coconut: 0.2 Then it is likely that the picture was of a banana. But if you saw this: Monkey: 0.7 Banana: 0.9 Coconut: 0.1 Maybe the picture contained a monkey holding a banana? It might be possible - even if the network was never trained on that particular imagery! Note that it is also possible to fool such a network, as numerous studies have shown. It's also possible to have the network output an image (ie - by having it output a very large array of node values that represents the pixel values of the image); this is how neural networks generate or alter images. Sound data can also be done in a similar manner. Alright - I think I am rambling a bit, so I'm going to leave it here; I hope this answers at least something of your questions (and I hope it didn't confuse you - if so, I apologize and that wasn't my intent).

11thEarlOfMar 8y ago

"We’ll use an optimization algorithm called stochastic gradient descent (SGD) that tells us how to change our weights and biases to minimize loss. It’s basically just this update equation: w1 ← w1 − η ∂w|∂L η is a constant called the learning rate that controls how fast we train. All we’re doing is subtracting η ∂w|∂L from w1: - If ∂L | ∂w1 is positive, w1 will decrease, which makes L decrease - If ∂L | ∂w1 is positive, w1 will increase, which makes L decrease" Simple indeed.

rerx 8y ago
?
joshklein 8y ago
I think this comment is facetious (apologies if I’m reading it incorrectly), so I want to offer that this is “simple”, but opaque to someone who is unfamiliar with the syntax or background context. Andrew Ng’s course on machine learning from Stanford (on Coursera, which you should be able to audit for free) gives great intuitive explanations of these topics, if you’re curious enough to invest a couple solid weekends to get the foundations. The foundations alone will get you very far indeed.
- cr0sh 8y ago
  Seconding Ng's course here. I took it in 2011 (ML Class) before Coursera existed, and it finally opened my eyes on not only what and how backprop worked, but also how everything in a NN could be represented and calculated using vectors and matrices (linear algebra), and how that process was "parallelizable". The course uses Octave as it's programming environment, which is essentially an open-source and (mostly) compatible implementation of Matlab. My first thought was "Finally! A use case for a home Beowulf cluster that is somewhat practical!" It really opened my eyes and mind to a number concepts that I had looked into before, but couldn't quite wrap my brain around completely.
profunctor 8y ago
I think it's supposed to be dL/dw1. As in the loss function derived wrt w1.

kowdermeister 8y ago

I started playing around with NN-s recently and this looked like a good tutorial until the partial derivatives :) It would have been so much better to turn a complex math concept like that to code in smaller chunks because it's hard to extract from the full sample which part refers to which equation. Also, the bias is not explained at all. - Why is it there? - What does it do? - What value should it have?

dragandj 8y ago
I currently publish a series of detailed articles that cover exactly that. https://dragan.rocks/articles/19/Deep-Learning-in-Clojure-From-Scratch-to-GPU-0-Why-Bother https://dragan.rocks/articles/19/Deep-Learning-in-Clojure-Fr...
- victor106 8y ago
  dragon.rocks, rocks. Your articles are some of the best i have seen on this topic. Thank you. I hope you write a book on this topic.
  - dragandj 8y ago
    Thanks! We'll see. BTW, its dragAn, not dragOn :)
xeRTRex 8y ago
Agreed, it is a little confusing. What makes it even more confusing is bias can be introduced as a regularization technique (to address overfitting) and these are not the same thing. Bias in this context can be simply interpreted as analogous to the intercept in y = mx + b. Kind of like the baseline when no additional information is given.
applecrazy 8y ago
Partial derivatives are basically how much a multivariate function changes based on a tiny change in a single variable. The actual math is irrelevant.

alexlchen2019 8y ago

amazing read, good content

noperks 8y ago
probably the most easy going read-through i've seen

eam 8y ago

I didn't have time to read all of it, but half way through. So far it has bee a pretty good write up, however, just one question. Can the author or anyone explain how were the amount for values shifted chosen? I.e. Weight (minus 135) and Height (minus 66). Why was -135 and -66 chosen? An explanation would be helpful. Thanks!

jventura 8y ago
I think they were chosen such that the weight and height values become negative for Females and positive for Males.
xeRTRex 8y ago
I haven't read the article thoroughly but I suspect part of the data preprocessing step included centering and standardizing. Those must be the mean weight and height values.
LodeOfCode 8y ago
"I arbitrarily chose the shift amounts (135 and 66) to make the numbers look nice. Normally, you’d shift by the mean."
- eam 8y ago
  I am pretty sure that note wasn't written there before. The author must have updated the article after I posted this question. Though "nice" is still somewhat vague to me. Nice in what way? Smaller numbers are nicer? Negative and positives make it nice? The second part of the answer is more useful information, "Normally, you’d shift by the mean."
emgee_1 8y ago
I think this a form of normalizing a normal distribution into a standard normal distribution ( a normal distribution has two parameters \mu and \sigma which in standard form gives respectively 0 and 1. Any normal distribution is a special case of the standard normal distribution N(0,1)

avmich 8y ago

> Real neural net code looks nothing like this. This negates a lot of promise of the article. If one understands principles, but needs to know how to apply them, this statement makes the article practically useless. Usually after reading like this goes a jump in explanations, which directs to apply TensorFlow et. al. and even, if you're lucky, explains how to do that - but doesn't explain what it is that TensorFlow does qualitatively different, which would justify putting such a phrase as a warning.

sgillen 8y ago
I still think it's a useful exercise for people like me who learn best by implementing a thing. Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area.
- avmich 8y ago
  > I still think it's a useful exercise for people like me who learn best by implementing a thing. Yes, but if you implement one thing, and then TensorFlow which you plan to use does something very different, what's the point? Ok, different enough to justify the warning. > Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area. Exactly. It's that magic which is interesting, after you're shown how in principle - but only in principle - the thing could work.
  - sgillen 8y ago
    I guess the point here is that in this article we implement some of the magic that the higher level frameworks use, which I think will be useful later when writing and debugging our high level code. But I see your point which is that it would be nice if the article was telling us which part of the high level library we're implementing and what that would look like. I'm still happy with the article though and would not call it almost useless. But hey if you'd rather jump right into a tensorkerastorch tutorial that's an option too!
    - 8y ago

gambler 8y ago

Why are deep learning researchers allergic to meaningful variable names?

chrisfosterelli 8y ago
These are meaningful variable names in the context of a code implementation of a math algorithm. The variables directly map to the algorithmic notation.
- jacobolus 8y ago
  And when trying to do algebraic manipulation on paper, having an 8 or 10 letter variable name is incredibly cumbersome. Frankly in the middle of a numerical algorithm it is typically also cumbersome in code to have descriptive variable names for everything. However, mathematical code (especially when written by scientists, etc.) often takes this too far, introducing many 1- or few-letter variable names without enough context or description to figure out what they stand for, and for more of the variables than necessary. This blog post was just fine though. h = hidden, o = output, y_pred = predicted value of y, etc. are quite clear
  - gambler 8y ago
    >h = hidden, o = output, y_pred = predicted value of y, etc. are quite clear ∀ abbr + to the cogntv load prsn wht mntn in thr hd. ∀ -> every + -> adds wht -> who has to h -> head Was that easier to read than the following? Every abbreviation adds to the cognitive load the person has to maintain in their head. This isn't code, but the principles of making code (un)readable are exactly the same. If you see that full-word variable names create too much clutter, it means your code structure is wrong. So you fix it. You go from this: self.w1 = np.random.normal() self.w2 = np.random.normal() self.w3 = np.random.normal() self.w4 = np.random.normal() self.w5 = np.random.normal() self.w6 = np.random.normal() To this: set_to_random_numbers(self.weights) After 9 years of maintaining several dozen legacy codebases (which sometimes involved financial math) I am fully convinced that the only people who like terse code are people who write it and never read it after.

fartcannon 8y ago

It's fun, but it's cheating a bit if you use numpy. Can someone implement one with all the matrix math from scratch too?

siekmanj 8y ago
shameless self-promotion: https://github.com/siekmanj/sieknet https://github.com/siekmanj/sieknet
- fartcannon 8y ago
  Looks great. Even got yourself a genetic algorithm in there, eh? How well does it work?
  - siekmanj 8y ago
    The genetic algorithm currently isn't in a working state unfortunately. I did a big architectural rewrite recently and haven't gotten around to GA stuff - but I'm looking forward to applying some more modern results in neuroevolution from the last two or three years.
archgoon 8y ago
I don't see why using matrix math shouldn't be viewed as a basic primitive. Implementing a matrix library would obscure the point that is being made.

posix_compliant 8y ago

Friendly comment: at the beginning of section 4 there's this sum: 1 ∑(y_true - y_pred)^2 i=1 There's no 'i' in the sum, should probably be y_true_i - y_pred_i.

posix_compliant 8y ago
I do get that in the example given there's only one element in each of the true/predicted vectors, but it threw me off for a sec.

amasad 8y ago

Runnable code from the article: https://repl.it/@vzhou842/An-Introduction-to-Neural-Networks https://repl.it/@vzhou842/An-Introduction-to-Neural-Networks I see so many implementations of NeuralNets from Scratch on Repl.it (I'm a co-founder) but this is one of the simplest and clearest one. One that blew my mind recently is one by a 12 year old programmer: https://repl.it/@pyelias/netlib https://repl.it/@pyelias/netlib

syntaxing 8y ago

If you're interested in learning something like this, I highly recommend Udacity's deep learning nanodegree. They have a section that teaches you how to build your own neural network with the the help of numpy. They even have a section where you write your own sentimental analysis neural network from scratch. Easily the best explanation and notebook to follow I have seen so far. I've been taking their revamped one that focuses on PyTorch rather than Tensorflow. Disclaimer: I am taking the deep learning nanodegree degree for free but its sponsored by Facebook rather than Udacity directly.

applecrazy 8y ago
The notebooks are available on GitHub for free.

laneb 8y ago

I appreciate the author showing some of the math that drives the optimization. As a reader, I'd suggest introducing SGD ahead of the PD calculations, because it would give a clearer motivation for the PDs.

arendtio 8y ago

Recently, I watched a Scrimba tutorial [1] about brain.js. That one brought me quite a step further to understand how different kinds of neural networks are capable of (NN vs. RNN vs. CNN). [1]: https://scrimba.com/g/gneuralnetworks https://scrimba.com/g/gneuralnetworks

ddebernardy 8y ago

One of the nice parts of NNs is that it takes some maths education to understand what's going on. My initial thought when reading the title was: "uh oh, are AI tutorials going to be the next PHP/MySQL tutorials, with god awful code all over the internet?" They haven't yet insofar as I'm aware. I hope the maths involved will prevent them from becoming that.

deytempo 8y ago
Luckily math isn’t actually required to implement them in a series of arrays so that little relief will be short lived.

yreaderynotread 8y ago

The derive of sigmoid is exp(-x) / (1+exp(-x))² and not exp(x) / (1+exp(-x))²

ujuj 8y ago

See this code posted on HN a month ago too: https://gist.github.com/macournoyer/620a8ba4a2ecd6d6feaf https://gist.github.com/macournoyer/620a8ba4a2ecd6d6feaf I found the code very clear and understandable.

gillesjacobs 8y ago

This is very similar to Andrew Ng's Deep Learning Specialization course [1]. If you found this blog post enjoyable be sure to check it out. It is a great course and the intuitions behind NNs are explained very clearly. 1. https://www.coursera.org/specializations/deep-learning https://www.coursera.org/specializations/deep-learning

iamaziz 8y ago

The visualization is cool, thanks! Here's another neural network example from scratch in 13 lines of Python: https://iamtrask.github.io/2015/07/27/python-network-part2/ https://iamtrask.github.io/2015/07/27/python-network-part2/

buildbot 8y ago

Since this is based on numpy, you could quickly have it GPU accelerated by using cupy: https://cupy.chainer.org/ https://cupy.chainer.org/

starpilot 8y ago

So many examples of this: https://hn.algolia.com/?query=python%20neural%20scratch&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=python%20neural%20scratch&sort... Is this example better than previous ones?

techbio 8y ago

Very cool. I want to give a shout out to the simple classic http://arctrix.com/nas/python/bpnn.py http://arctrix.com/nas/python/bpnn.py

cgopalan 8y ago

I took the Pytorch scholarship challenge this past December with Udacity. I couldn't make it to the nanodegree scholarship but the challenge course was really good in exposing me to the various facets of deep learning. The notebooks are free, but the videos really help explain some concepts well. For people who took the course, the videos are available for another year, so that helps. I wanted to get a good grasp of DL but was floundering around before with so many sources and tutorials - and frameworks like Keras, Pytorch, TensorFlow etc. This course helped me make some decisions - like if I needed to get a developer's handle on DL, I would have to know Perceptrons, CNNs, RNNs, Style Transfer, Sentiment Prediction and GANs (even though GANs were not part of the course). And Pytorch would be my tool of choice.

deleted 8y ago

[deleted]

theblackcat1002 8y ago

I think a better and elegant way to implement is the pytorch way. By defining a forward and backward function for easy chaining, you can easily chain multiple layers into a single model. Placing the training loop inside the layer will be hard to write a multi layer version. Just wrote a similar from scratch code few months back: https://github.com/theblackcat102/DL_2018/blob/master/HW2/layers.py https://github.com/theblackcat102/DL_2018/blob/master/HW2/la...

ecmascript 8y ago

This is very cool and informative. Thank you for this article!