10 ms·

Backpropagation is a leaky abstraction

365 points by nafizh 10y ago

aibottle 10y ago
People complaining about having to implement backprop in a ML class? Nice one. Had to hand in this exact assignment last Friday. A valuable lesson, that's for sure. Generally I love how a proper lesson on Machine Learning really contradicts the "universal power weapon" narrative media put on machine learning in the last few years. Everything is not so magic anymore when it boils down to proper algorithm use.
- jlg23 10y ago
  Hear, hear. Indeed, "welcome" to everybody who takes the time to understand what (s)he is talking about.
- vertex-four 10y ago
  > People complaining about having to implement backprop in a ML class? Wouldn't that suggest that other classes the students take have not been academic enough - i.e. that they focus too much on "things you might use day-to-day" vs "this is how/why things work"?
  - tnecniv 10y ago
    This attitude is not uncommon in programming / CS. It's similar to those who pine about learning a half dozen search algorithms when they rarely need to implement them in practice.
- esalman 10y ago
  Agreed. The most important lesson I took away from my ML class is that I have to learn more of optimization theory.
tnecniv 10y ago
The phrase "raw numpy" strikes me as funny. I would figure that's about as abstract as you could get while still working with the math (discarding symbolic engines).
- bmh100 10y ago
  The phrase seems appropriate to me. The students are still working with matrices and linear algebra, focusing on the major algorithms. Going into the element-by-element linear algebra algorithms, for example in matrix multiplication, would be more appropriate for a high-performance, numerical computing class. The right level of detail is given when he talks about considering the behavior of individual gradient elements.
- p1esk 10y ago
  Yes, after implementing a simple neural network in C (with AVX, and pthreads), "raw numpy" does sound funny! On the other hand, try implementing a convnet in numpy, especially the backprop, and you might start feeling some of its "rawness" :-)
  - chestervonwinch 10y ago
    I know what you mean. To take partial derivatives with respect to the filter parameters from a correlation (or convolution), it's simpler to go down to the component level. However, it's hard to get back up to the matrix/vector level after doing so (to write the operations in NumPy). I'm developing a model (not exactly a convnet) that uses a correlation step. Because of the above problem and its resulting pure-python loops, I may have to cythonize or use the NumPy C API for the gradient evaluation. Do you know of any examples I could check out that implement partial derivatives w.r.t. a correlation (or convolution) in "raw numpy"?
  - TPCrow 10y ago
    As someone who also hand-coded a neural network implementation, forward and back prop as well as an RNN in C, yea "raw numpy" is a joke. Something I've always hated about people that say "why do I have to write a backprop when TF does it for me?" Here's why: you go to a company, they want you to incorporate machine learning into their c++ engine. Have fun using numpy, you said you knew machine learning right? implement backprop for me, you can do that right?
tsomctl 10y ago
> Why do we have to write the backward pass when frameworks in the real world, such as TensorFlow, compute them for you automatically? Why do you have to learn to calculate integrals and derivatives in school, or how compilers work internally? Same answer. But seriously, the CS231 class is excellent, and Andrej is an excellent teacher. You can follow along at home (which is what I am doing.) The syllabus (at http://cs231n.stanford.edu/syllabus.html http://cs231n.stanford.edu/syllabus.html) has the course notes and the assignments. The assignments are self grading, you know when you have it programmed correctly. The lectures are here: https://www.youtube.com/playlist?list=PLlJy-eBtNFt6EuMxFYRiNRS07MCWN5UIA https://www.youtube.com/playlist?list=PLlJy-eBtNFt6EuMxFYRiN...
- gime_tree_fiddy 10y ago
  Sort of off topic. I have been planning to do the same course(CS 231n), and was wondering, is it okay if I do the Andrew Ng Machine Learning (Coursera) in parallel, or is it more of a prerequisite ?
  - tsomctl 10y ago
    I only did the first several videos of Andrew Ng's class, and didn't like it. The only prerequisites to CS231n are linear algebra, calculus, and Python.
  - jongomez 10y ago
    As long as you know a lil bit of math and coding, you're good to go. If you're watching the cs231n videos and don't understand something, you can always check out their amazing class notes. The cs231n course covers a lot of material, including some (or most) of the material from the Andrew Ng's Coursera course. You can totally do them in parallel though.
fizixer 10y ago
Backpropagation is a leaky abstraction in the sense that every algorithm/physics-principle/mathematical-theorem is a leaky abstraction. Take 'sort.' If you use a sort API for large N in a performance-critical section of your code without knowing if the implementation of that sort is an insertion-sort or a quicksort, "you would be nervous." Hence you are dealing with a leaky abstraction, per this article. I would much rather respond to the students who ask about why they need to know the details, by saying "because you registered for this class." Essentially the question "if you’re never going to write backward passes once the class is over, why practice writing them?" is another way of asking "why reinvent the wheel?". And the best answer is "because learning is all about reinventing the wheel". The "code reuse as much as possible" mantra applies to when you're "using" a technique to do something else, not when you're "learning" the technique itself. They might as well register for Calculus and ask "why learn integrals and derivatives when mathematica can do them for you", or take an Aerodynamics class and ask "why learn fluid mechanics and dynamics, heck newton's laws, when an airplane can run on autopilot." I doubt "because calculus is a leaky abstraction" or "because fluid dynamics is a leaky abstraction" is a good answer to that.
- thecity2 10y ago
  "because fluid dynamics is a leaky abstraction" No pun intended!
- skybrian 10y ago
  I think you're sweeping an important distinction under the rug. If every major language provides an O(n log n) sort function, is it still a leaky abstraction? I'd say no. You can use it without worrying much about the details. But it sounds like the situation with back-propagation is different, since the internal details of the algorithm affect whether you get a usable answer at all. A borderline case might be something like a SQL database, where in theory, creating an index shouldn't change query results, but in practice, the performance can change so much that it effectively does. Unlike with back-propagation, you can tune database performance without worrying about queries returning different results (assuming they don't time out). So it's still a useful guarantee.
  - 10y ago
thecity2 10y ago
Backprop is just automatic differentiation. The end.
- deleted 10y ago
  [deleted]
- ww520 10y ago
  That's a succinct way of putting it.
- deleted 10y ago
  [deleted]
cr0sh 10y ago
I do think the complaint on having to write the backward pass seems especially shallow; finding out they were working with numpy makes it even more so (since numpy takes the pain out of the matrix operations). IIRC, when I took the ML Class in 2011, we used Octave, but Ng had us first write stuff "the hard way" - so we'd understand what was going on later when we used Octave's methods. Something about this article as a whole, though, does raise a question that I've been wondering about, and I want to present it here for a bit of discussion (maybe it needs its own thread?): Does anyone else here think that the current approach to neural networks has some fundamental flaws? Now - I'm not an expert; call me an interested student right now, with only the barest of experience (beyond the ML Class, I also took Udacity's CS373 course, and I am also currently enrolled in their Self-Driving Car Engineer nanodegree program). I understand that what we currently have and know does work. What I mean by that is the basic idea of an artificial neural network using forward and back-prop, multiple layers, etc (and all the derivatives - RNN, CNN, deep learning, etc). I understand the need and reasoning behind using activation functions based around calculus and derivatives and the chain-rule, etc (though I admit I need further education in these items). But something nags at me. All of this, despite the fact that it works and works well (provided all your tuning and such is right, etc), just seems like it is over-complicated. Real neurons don't use calculus and activation functions, nor back-propagation, etc in order to learn. All of those things in an ANN are just abstractions and models around what occurs in nature. Maybe (probably?) I am wrong - but it seems like what nature does is simpler. Much less power is used, for instance, and the package is much more compact. I just have this feeling that in some manner we may have gone down a path that while it has produced a working representation, that representation is overly complex, and had we took another approach (whatever that might be?), our ANNs would look and work much differently - perhaps even more efficiently. About the only alternatives I have heard about otherwise have been things like spike-train neural networks, and some of the other "closer to nature" simulation (of ion pumps and real synapses, etc). Still, even those, while seemingly closer, also have what appears to be too much complexity. I'm probably just talking out of my nether regions as a general n00b to the field. I do wonder, though, if there might be another solution, seemingly out in "left-field" that might push things forward, if someone was willing to look and experiment. It is something I plan to look into myself, as I find time and such between lessons and other work for my current learning experience.
geuis 10y ago
Edit: hmm not sure why this comment is getting downvoted. Backprop isn't just a leaky abstraction. There's not really any evidence yet that biological neural networks use anything like backprop. So its important that students be taught the low-level aspects of the current state of the art so that better architectures can be invented in the future. (Note: at the very end of this comment, I am leaving a link to one hypothesis about how something like backprop may happen at the dendritic level.) I've been doing a deep dive on ML and bio neurons in the last week or so to try and get up to speed. The feedforward networks with backprop that are doing such awesome work right now share almost nothing in common with actual neurons. Just some differences: (neuron == bio, node == artificial) Terms: Neuron resting potential is about -70mV. Action potential (threshold) is about -55mV. 1) Neurons are spikey and don't have weights. A neuron sums all of its inputs to reach a threshold. When that threshold is reached, an electric current (action potential) is spread equally to all synaptic terminals. A node has a weight value for each input node that it multiplies by the value received from the input node. All of these values are then summed and passed to the connected nodes in the next layer. 2) There are multiple types of neurons, but for simplification there are 2 types that called "excitatory" and "inhibatory". A neuron can mostly only emit "positive" or "negative" signals via the synapse. There are different neurotransmitters that either add or subtract to the voltage potential for each dendritic input. The total sums work together to cross the threshold voltage. I say mostly because there is some evidence for a minority of neurons to be able to release neurotransmitters from both groups. 3) Artificial networks are generally summation based with no inclusion of time. Bio networks use summation and summation over time together to determine if they should fire or not. If I recall correctly, a neuron can repeatedly fire about 300 times per second. A dendritic input can last for up to 1.5 milliseconds. So if a neuron gets enough positive inputs at the same time, or collects enough over time, it will fire. I haven't found any hypotheses or experiments that try to explain how reinforcement learning takes place in neurons yet in the absence of back propagation. Pretty sure that information is out there but I haven't run across it. Overall I think that there will be multiple engineering approaches to AI, just like there are to construction and flight. We understand how birds and bees fly, but we don't build planes the same way. Its important to remember than cognition is based in physics just like any other physical system. When the principles are well understood, there are multiple avenues to using them. Here is a short collection of links that I've been finding helpful. https://en.wikipedia.org/wiki/Neurotransmission https://en.wikipedia.org/wiki/Neurotransmission http://neuroscience.uth.tmc.edu/s1/introduction.html http://neuroscience.uth.tmc.edu/s1/introduction.html https://www.quora.com/Can-a-synapse-change-from-being-inhibitory-to-excitatory-What-is-the-mechanism-behind-it https://www.quora.com/Can-a-synapse-change-from-being-inhibi... https://en.wikipedia.org/wiki/Dale's_principle https://en.wikipedia.org/wiki/Dale's_principle https://en.wikipedia.org/wiki/Neural_backpropagation https://en.wikipedia.org/wiki/Neural_backpropagation
rahrahrah 10y ago
> “Why do we have to write the backward pass when frameworks in the real world, such as TensorFlow, compute them for you automatically?” How many more times do you need to see the same phenomenon under different guises before you stop asking stupid questions? "Hey teach, why do I need to learn how to multiply if I can just use a calculator?"
- posterboy 10y ago
  Isn't it somewhat reasonable question, given the relatively recent advent of TensorFlow, compared to ML curricula? The stress is on, why don't we learn TensorFlow / Caffe / etc.
  - tnecniv 10y ago
    Because frameworks come and go. The important thing are the abstract concepts At that level, they assume you are pretty smart and capable of figuring out something like an API on your own time as needed. They'd rather you know what all these funky things in these APIs are doing at a core level so that you can employ them in an effective manner.
    - posterboy 10y ago
      I could use that same abstract argument, just that with a framework you wanna see how far you can get, not how low. From the schools perspective it's all the same, just some test on your mental powers. With a framework it's hard to think outside the frame. With a low level core it's hard to do anything really. It's a matter of compromise. Noone starts writing asm to begin with, although it is interesting, e.g. nand tetris being a famous example.
    - HelloNurse 10y ago
      Letting the computer do the menial work of constructing formulas and generating code from user specifications is a fairly important framework feature and "abstract concept".
  - rahrahrah 10y ago
alxmdev 10y ago
Questions like "Why do we have to write X, when framework Y does it for you?" are why I dislike the reinventing the wheel analogy, especially when it finds its way in education. There's no substitute for the deep understanding you get by solving a complex problem yourself from beginning to end. Students complaining about implementing a foundational algorithm instead of using a framework is depressing. Not to mention that computer science and software engineering are such young fields that it seems unhealthy to take readily available abstractions as absolute givens. Everything stands to improve, even products and concepts that have been around for decades and that everyone uses.
- posterboy 10y ago
  > There's no substitute for the deep understanding you get by solving a complex problem yourself from beginning to end Yes there is, watch someone else do it. In fact there are three ways to learn, as the saying goes: trial and error, copying, and insight. I'd be hard pressed to explain the difference of trial and error vs insight, but I wouldn't confuse them either, because only one of them is painful.
  - sn9 10y ago
    Watching someone else do it is still less effective than bashing your head against the problem yourself before watching them do it. Whether or not you succeed in solving it on your own, it will emotionally invest you in the problem and its solution while showing you what didn't work and having a better handle on the shape of the problem. This lets you get more out of seeing someone else work out the solution.
  - alxmdev 10y ago
    I think there are intangible things that get lost when you watch someone else do something as opposed to doing it yourself. The exercise is about going through the mental motions of understanding the problem, designing a solution, and iterating on it until it's correct. The last part is all about learning from your own mistakes, seeing what specific things trip you up, so you know to improve on them. That's not something you can get by watching others. I agree that there isn't enough time in a life to learn everything you'd want to first hand or from a low level of abstraction, but school should be a place to do as much of it as possible. Just my 2c.
bmh100 10y ago
After reading the article, my summary of the message is not "backpropagation is a leaky abstraction" but instead "if you don't understand how the derivatives are being calculated, it will come back to bite you". The author talks about issues that arise with s-curve activation functions having saturated outputs and minimized gradients (part of the historical reason for moving away from these functions), the vanishing/exploding gradient problem in RNNs caused by repeated multiplication, the issue with clipping gradients as a solution, and the issue with zero-valued gradients in ReLUs. This would be equivalent to being a DBA without understanding indexes, or being a web developer without understanding the DOM. Yes, you can get by for a while, coasting on your tools. But when the stakes are high and you need to get it right, that ignorance will hamstring you. You don't want to be put in the situation of building a NN for someone and having no good idea about why it isn't working yet.
deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
vazamb 10y ago
Backprop is such a fundamental part of Neural Networks, I am very surprised how anyone can complain about having to know how it works. It is true that once you grocked the principle implementing it for more than 2 layers is pretty tedious. I would propose, however, that without having done the tedious work at least once you cannot truly understand it.
godmodus 10y ago
https://m.youtube.com/watch?v=1SrHzSGn-I8 https://m.youtube.com/watch?v=1SrHzSGn-I8 Feynmann pretty much settles it.
conjectures 10y ago
Much discussion of backprop could be avoided by recalling that the bit that does the work is the chain rule from calculus. Error terms represent a sum and product of derivatives. The product of a bunch of terms will tend to get really big or really small. The rest is detail: are the terms in some interval? Which? how many are we multiplying? how many are we summing over? do we doctor the sum after we get it?
annnnd 10y ago
For anyone trying to learn backpropagation but having trouble with math, I can't recommend Matt Mazur's "Step by Step" guide [0] enough. What is great is that he is using real numbers so that one can check implementation for correctness. [0] https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ https://mattmazur.com/2015/03/17/a-step-by-step-backpropagat...
- cjf101 10y ago
  I used this recently when I was relearning NN theory (last time I looked at it before 2015 was the late 90's,) and I agree. It's a thorough walkthrough of the math that turned the lightbulbs back on enough for me to write a simple swift MLP NN without consulting other implementations.
thatsadude 10y ago
Isn't backprop a clever application of chain rule in multivariate calculus?

365 points by nafizh 10y ago

People complaining about having to implement backprop in a ML class? Nice one. Had to hand in this exact assignment last Friday. A valuable lesson, that's for sure. Generally I love how a proper lesson on Machine Learning really contradicts the "universal power weapon" narrative media put on machine learning in the last few years. Everything is not so magic anymore when it boils down to proper algorithm use.

jlg23 10y ago
Hear, hear. Indeed, "welcome" to everybody who takes the time to understand what (s)he is talking about.
vertex-four 10y ago
> People complaining about having to implement backprop in a ML class? Wouldn't that suggest that other classes the students take have not been academic enough - i.e. that they focus too much on "things you might use day-to-day" vs "this is how/why things work"?
- tnecniv 10y ago
  This attitude is not uncommon in programming / CS. It's similar to those who pine about learning a half dozen search algorithms when they rarely need to implement them in practice.
esalman 10y ago
Agreed. The most important lesson I took away from my ML class is that I have to learn more of optimization theory.

tnecniv 10y ago

The phrase "raw numpy" strikes me as funny. I would figure that's about as abstract as you could get while still working with the math (discarding symbolic engines).

bmh100 10y ago
The phrase seems appropriate to me. The students are still working with matrices and linear algebra, focusing on the major algorithms. Going into the element-by-element linear algebra algorithms, for example in matrix multiplication, would be more appropriate for a high-performance, numerical computing class. The right level of detail is given when he talks about considering the behavior of individual gradient elements.
p1esk 10y ago
Yes, after implementing a simple neural network in C (with AVX, and pthreads), "raw numpy" does sound funny! On the other hand, try implementing a convnet in numpy, especially the backprop, and you might start feeling some of its "rawness" :-)
- chestervonwinch 10y ago
  I know what you mean. To take partial derivatives with respect to the filter parameters from a correlation (or convolution), it's simpler to go down to the component level. However, it's hard to get back up to the matrix/vector level after doing so (to write the operations in NumPy). I'm developing a model (not exactly a convnet) that uses a correlation step. Because of the above problem and its resulting pure-python loops, I may have to cythonize or use the NumPy C API for the gradient evaluation. Do you know of any examples I could check out that implement partial derivatives w.r.t. a correlation (or convolution) in "raw numpy"?
- TPCrow 10y ago
  As someone who also hand-coded a neural network implementation, forward and back prop as well as an RNN in C, yea "raw numpy" is a joke. Something I've always hated about people that say "why do I have to write a backprop when TF does it for me?" Here's why: you go to a company, they want you to incorporate machine learning into their c++ engine. Have fun using numpy, you said you knew machine learning right? implement backprop for me, you can do that right?

tsomctl 10y ago

> Why do we have to write the backward pass when frameworks in the real world, such as TensorFlow, compute them for you automatically? Why do you have to learn to calculate integrals and derivatives in school, or how compilers work internally? Same answer. But seriously, the CS231 class is excellent, and Andrej is an excellent teacher. You can follow along at home (which is what I am doing.) The syllabus (at http://cs231n.stanford.edu/syllabus.html http://cs231n.stanford.edu/syllabus.html) has the course notes and the assignments. The assignments are self grading, you know when you have it programmed correctly. The lectures are here: https://www.youtube.com/playlist?list=PLlJy-eBtNFt6EuMxFYRiNRS07MCWN5UIA https://www.youtube.com/playlist?list=PLlJy-eBtNFt6EuMxFYRiN...

gime_tree_fiddy 10y ago
Sort of off topic. I have been planning to do the same course(CS 231n), and was wondering, is it okay if I do the Andrew Ng Machine Learning (Coursera) in parallel, or is it more of a prerequisite ?
- tsomctl 10y ago
  I only did the first several videos of Andrew Ng's class, and didn't like it. The only prerequisites to CS231n are linear algebra, calculus, and Python.
- jongomez 10y ago
  As long as you know a lil bit of math and coding, you're good to go. If you're watching the cs231n videos and don't understand something, you can always check out their amazing class notes. The cs231n course covers a lot of material, including some (or most) of the material from the Andrew Ng's Coursera course. You can totally do them in parallel though.

fizixer 10y ago

Backpropagation is a leaky abstraction in the sense that every algorithm/physics-principle/mathematical-theorem is a leaky abstraction. Take 'sort.' If you use a sort API for large N in a performance-critical section of your code without knowing if the implementation of that sort is an insertion-sort or a quicksort, "you would be nervous." Hence you are dealing with a leaky abstraction, per this article. I would much rather respond to the students who ask about why they need to know the details, by saying "because you registered for this class." Essentially the question "if you’re never going to write backward passes once the class is over, why practice writing them?" is another way of asking "why reinvent the wheel?". And the best answer is "because learning is all about reinventing the wheel". The "code reuse as much as possible" mantra applies to when you're "using" a technique to do something else, not when you're "learning" the technique itself. They might as well register for Calculus and ask "why learn integrals and derivatives when mathematica can do them for you", or take an Aerodynamics class and ask "why learn fluid mechanics and dynamics, heck newton's laws, when an airplane can run on autopilot." I doubt "because calculus is a leaky abstraction" or "because fluid dynamics is a leaky abstraction" is a good answer to that.

thecity2 10y ago
"because fluid dynamics is a leaky abstraction" No pun intended!
skybrian 10y ago
I think you're sweeping an important distinction under the rug. If every major language provides an O(n log n) sort function, is it still a leaky abstraction? I'd say no. You can use it without worrying much about the details. But it sounds like the situation with back-propagation is different, since the internal details of the algorithm affect whether you get a usable answer at all. A borderline case might be something like a SQL database, where in theory, creating an index shouldn't change query results, but in practice, the performance can change so much that it effectively does. Unlike with back-propagation, you can tune database performance without worrying about queries returning different results (assuming they don't time out). So it's still a useful guarantee.
- 10y ago

thecity2 10y ago

Backprop is just automatic differentiation. The end.

deleted 10y ago
[deleted]
ww520 10y ago
That's a succinct way of putting it.
deleted 10y ago
[deleted]

cr0sh 10y ago

I do think the complaint on having to write the backward pass seems especially shallow; finding out they were working with numpy makes it even more so (since numpy takes the pain out of the matrix operations). IIRC, when I took the ML Class in 2011, we used Octave, but Ng had us first write stuff "the hard way" - so we'd understand what was going on later when we used Octave's methods. Something about this article as a whole, though, does raise a question that I've been wondering about, and I want to present it here for a bit of discussion (maybe it needs its own thread?): Does anyone else here think that the current approach to neural networks has some fundamental flaws? Now - I'm not an expert; call me an interested student right now, with only the barest of experience (beyond the ML Class, I also took Udacity's CS373 course, and I am also currently enrolled in their Self-Driving Car Engineer nanodegree program). I understand that what we currently have and know does work. What I mean by that is the basic idea of an artificial neural network using forward and back-prop, multiple layers, etc (and all the derivatives - RNN, CNN, deep learning, etc). I understand the need and reasoning behind using activation functions based around calculus and derivatives and the chain-rule, etc (though I admit I need further education in these items). But something nags at me. All of this, despite the fact that it works and works well (provided all your tuning and such is right, etc), just seems like it is over-complicated. Real neurons don't use calculus and activation functions, nor back-propagation, etc in order to learn. All of those things in an ANN are just abstractions and models around what occurs in nature. Maybe (probably?) I am wrong - but it seems like what nature does is simpler. Much less power is used, for instance, and the package is much more compact. I just have this feeling that in some manner we may have gone down a path that while it has produced a working representation, that representation is overly complex, and had we took another approach (whatever that might be?), our ANNs would look and work much differently - perhaps even more efficiently. About the only alternatives I have heard about otherwise have been things like spike-train neural networks, and some of the other "closer to nature" simulation (of ion pumps and real synapses, etc). Still, even those, while seemingly closer, also have what appears to be too much complexity. I'm probably just talking out of my nether regions as a general n00b to the field. I do wonder, though, if there might be another solution, seemingly out in "left-field" that might push things forward, if someone was willing to look and experiment. It is something I plan to look into myself, as I find time and such between lessons and other work for my current learning experience.

geuis 10y ago

Edit: hmm not sure why this comment is getting downvoted. Backprop isn't just a leaky abstraction. There's not really any evidence yet that biological neural networks use anything like backprop. So its important that students be taught the low-level aspects of the current state of the art so that better architectures can be invented in the future. (Note: at the very end of this comment, I am leaving a link to one hypothesis about how something like backprop may happen at the dendritic level.) I've been doing a deep dive on ML and bio neurons in the last week or so to try and get up to speed. The feedforward networks with backprop that are doing such awesome work right now share almost nothing in common with actual neurons. Just some differences: (neuron == bio, node == artificial) Terms: Neuron resting potential is about -70mV. Action potential (threshold) is about -55mV. 1) Neurons are spikey and don't have weights. A neuron sums all of its inputs to reach a threshold. When that threshold is reached, an electric current (action potential) is spread equally to all synaptic terminals. A node has a weight value for each input node that it multiplies by the value received from the input node. All of these values are then summed and passed to the connected nodes in the next layer. 2) There are multiple types of neurons, but for simplification there are 2 types that called "excitatory" and "inhibatory". A neuron can mostly only emit "positive" or "negative" signals via the synapse. There are different neurotransmitters that either add or subtract to the voltage potential for each dendritic input. The total sums work together to cross the threshold voltage. I say mostly because there is some evidence for a minority of neurons to be able to release neurotransmitters from both groups. 3) Artificial networks are generally summation based with no inclusion of time. Bio networks use summation and summation over time together to determine if they should fire or not. If I recall correctly, a neuron can repeatedly fire about 300 times per second. A dendritic input can last for up to 1.5 milliseconds. So if a neuron gets enough positive inputs at the same time, or collects enough over time, it will fire. I haven't found any hypotheses or experiments that try to explain how reinforcement learning takes place in neurons yet in the absence of back propagation. Pretty sure that information is out there but I haven't run across it. Overall I think that there will be multiple engineering approaches to AI, just like there are to construction and flight. We understand how birds and bees fly, but we don't build planes the same way. Its important to remember than cognition is based in physics just like any other physical system. When the principles are well understood, there are multiple avenues to using them. Here is a short collection of links that I've been finding helpful. https://en.wikipedia.org/wiki/Neurotransmission https://en.wikipedia.org/wiki/Neurotransmission http://neuroscience.uth.tmc.edu/s1/introduction.html http://neuroscience.uth.tmc.edu/s1/introduction.html https://www.quora.com/Can-a-synapse-change-from-being-inhibitory-to-excitatory-What-is-the-mechanism-behind-it https://www.quora.com/Can-a-synapse-change-from-being-inhibi... https://en.wikipedia.org/wiki/Dale's_principle https://en.wikipedia.org/wiki/Dale's_principle https://en.wikipedia.org/wiki/Neural_backpropagation https://en.wikipedia.org/wiki/Neural_backpropagation

rahrahrah 10y ago

> “Why do we have to write the backward pass when frameworks in the real world, such as TensorFlow, compute them for you automatically?” How many more times do you need to see the same phenomenon under different guises before you stop asking stupid questions? "Hey teach, why do I need to learn how to multiply if I can just use a calculator?"

posterboy 10y ago
Isn't it somewhat reasonable question, given the relatively recent advent of TensorFlow, compared to ML curricula? The stress is on, why don't we learn TensorFlow / Caffe / etc.
- tnecniv 10y ago
  Because frameworks come and go. The important thing are the abstract concepts At that level, they assume you are pretty smart and capable of figuring out something like an API on your own time as needed. They'd rather you know what all these funky things in these APIs are doing at a core level so that you can employ them in an effective manner.
  - posterboy 10y ago
    I could use that same abstract argument, just that with a framework you wanna see how far you can get, not how low. From the schools perspective it's all the same, just some test on your mental powers. With a framework it's hard to think outside the frame. With a low level core it's hard to do anything really. It's a matter of compromise. Noone starts writing asm to begin with, although it is interesting, e.g. nand tetris being a famous example.
  - HelloNurse 10y ago
    Letting the computer do the menial work of constructing formulas and generating code from user specifications is a fairly important framework feature and "abstract concept".
- rahrahrah 10y ago

alxmdev 10y ago

Questions like "Why do we have to write X, when framework Y does it for you?" are why I dislike the reinventing the wheel analogy, especially when it finds its way in education. There's no substitute for the deep understanding you get by solving a complex problem yourself from beginning to end. Students complaining about implementing a foundational algorithm instead of using a framework is depressing. Not to mention that computer science and software engineering are such young fields that it seems unhealthy to take readily available abstractions as absolute givens. Everything stands to improve, even products and concepts that have been around for decades and that everyone uses.

posterboy 10y ago
> There's no substitute for the deep understanding you get by solving a complex problem yourself from beginning to end Yes there is, watch someone else do it. In fact there are three ways to learn, as the saying goes: trial and error, copying, and insight. I'd be hard pressed to explain the difference of trial and error vs insight, but I wouldn't confuse them either, because only one of them is painful.
- sn9 10y ago
  Watching someone else do it is still less effective than bashing your head against the problem yourself before watching them do it. Whether or not you succeed in solving it on your own, it will emotionally invest you in the problem and its solution while showing you what didn't work and having a better handle on the shape of the problem. This lets you get more out of seeing someone else work out the solution.
- alxmdev 10y ago
  I think there are intangible things that get lost when you watch someone else do something as opposed to doing it yourself. The exercise is about going through the mental motions of understanding the problem, designing a solution, and iterating on it until it's correct. The last part is all about learning from your own mistakes, seeing what specific things trip you up, so you know to improve on them. That's not something you can get by watching others. I agree that there isn't enough time in a life to learn everything you'd want to first hand or from a low level of abstraction, but school should be a place to do as much of it as possible. Just my 2c.

bmh100 10y ago

After reading the article, my summary of the message is not "backpropagation is a leaky abstraction" but instead "if you don't understand how the derivatives are being calculated, it will come back to bite you". The author talks about issues that arise with s-curve activation functions having saturated outputs and minimized gradients (part of the historical reason for moving away from these functions), the vanishing/exploding gradient problem in RNNs caused by repeated multiplication, the issue with clipping gradients as a solution, and the issue with zero-valued gradients in ReLUs. This would be equivalent to being a DBA without understanding indexes, or being a web developer without understanding the DOM. Yes, you can get by for a while, coasting on your tools. But when the stakes are high and you need to get it right, that ignorance will hamstring you. You don't want to be put in the situation of building a NN for someone and having no good idea about why it isn't working yet.

deleted 10y ago

[deleted]

deleted 10y ago

[deleted]

vazamb 10y ago

Backprop is such a fundamental part of Neural Networks, I am very surprised how anyone can complain about having to know how it works. It is true that once you grocked the principle implementing it for more than 2 layers is pretty tedious. I would propose, however, that without having done the tedious work at least once you cannot truly understand it.

godmodus 10y ago

https://m.youtube.com/watch?v=1SrHzSGn-I8 https://m.youtube.com/watch?v=1SrHzSGn-I8 Feynmann pretty much settles it.

conjectures 10y ago

Much discussion of backprop could be avoided by recalling that the bit that does the work is the chain rule from calculus. Error terms represent a sum and product of derivatives. The product of a bunch of terms will tend to get really big or really small. The rest is detail: are the terms in some interval? Which? how many are we multiplying? how many are we summing over? do we doctor the sum after we get it?

annnnd 10y ago

For anyone trying to learn backpropagation but having trouble with math, I can't recommend Matt Mazur's "Step by Step" guide [0] enough. What is great is that he is using real numbers so that one can check implementation for correctness. [0] https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ https://mattmazur.com/2015/03/17/a-step-by-step-backpropagat...

cjf101 10y ago
I used this recently when I was relearning NN theory (last time I looked at it before 2015 was the late 90's,) and I agree. It's a thorough walkthrough of the math that turned the lightbulbs back on enough for me to write a simple swift MLP NN without consulting other implementations.

thatsadude 10y ago

Isn't backprop a clever application of chain rule in multivariate calculus?