4 ms·

Backpropagation 101 (2020)

77 points by polm23 6y ago

teruakohatu 6y ago
This is a very, very convoluted way of understanding backprop. If you want to understand it I highly recommend writing a neural network starting from a basic perceptron (no hidden layer) or even implementing basic linear regression with stochastic gradient descent. Then work your way up to a network with one hidden layer. Try to focus on the maths and avoid looking at code examples.
- selimthegrim 6y ago
  Is scholastic gradient descent anything like grad student descent? ;)
- reitzensteinm 6y ago
  This is a great book that breaks down the process step by step. I had my own neural nets working before I touched a framework, and it demystified what would have otherwise seemed like black magic. https://www.amazon.com.au/Make-Your-Own-Neural-Network-ebook/dp/B01EER4Z4G https://www.amazon.com.au/Make-Your-Own-Neural-Network-ebook...
  - 29athrowaway 6y ago
    The book I used to learn the basics many years ago was "An Introduction to Neural Networks", by Kevin Gurney. It's a bit outdated now, but I still consider it valuable.
- cultyyy123 6y ago
  This is a very, very convoluted way of gate keeping. > Try to focus on the maths and avoid looking at code examples. Straight out dictatorship. > If you want to understand it I highly recommend writing a neural network starting from a basic perceptron (no hidden layer) or even implementing basic linear regression with stochastic gradient descent. Then work your way up to a network with one hidden layer. Like it's only way to understand backpropagation. This quote is still valid to date. “Today's scientists have substituted mathematics for experiments, and they wander off through equation after equation, and eventually build a structure which has no relation to reality. ” - Nikola Tesla Trust me, you can't fool people for too long. Abstractions always wins the game.
  - Vhano 6y ago
    I agree. Why talk in precise terms when you can do abstract mythology instead.
- datameta 6y ago
  I absolutely second the suggestion of working through the math of a shallow neural network to start. I would also suggest that once one thinks they have a sufficient grasp of the steps to try to implement them at a low level without frameworks, just math libraries. Personally, I found Octave (highly similar to MATLAB) to be a nice granularity tier for learning the process programmatically.
kristianov 6y ago
Why are people so afraid of math?
- ganafagol 6y ago
  Because "I've always been bad at math" is a hip thing to say in today's society and gets you laughter and approving nods.
  - cultyyy123 6y ago
    In reality most people are bad/afraid at Numbers not at Maths. The book Maths without numbers highlight what needs to be done to Education system.
    - tgv 6y ago
      Sure. It's much easier to understand the Dedekind cut, rings, and function spaces than to subtract 35% from 140.
      - laichzeit0 6y ago
        Most people afraid of “math” only have exposure to the second kind. I figure it has more to do with finding those type of mechanical operations boring (they are), and hence all of maths must be equally boring. If you’re a software developer, enjoy abstractions, programming language theory, and solving problems there is no way you would find math of the first sort boring. Alas, in high school pretty much all the math is of the boring mechanical kind. At least mine was, anyway. Even the first exposure to non-rigorous calculus is of the shitty plug-and-chug variety.
        tgv 6y ago
        It's hard to go through life without developing a modest understanding of arithmetic and reading. You wouldn't even be able to check your receipts if school started with Peano's axioms. Consequently, numbers come first, as they have for millennia now. Which makes sense, because maths developed out of intuitive arithmetic. As a matter of fact, my parents complain they never understood maths, because they were taught algebraic rules by rote. a * b = b * a. They can recite that, they just don't have a clue what to do with it. Teachers just aren't good, and certainly not at maths. It's a struggle to teach billions the basic skills. There is no solution. I also don't know any mathematician or physicist bad at arithmetic. Quite a few of them seem to delight in numbers and arithmetic puzzles. So I really don't see how to avoid numbers and boring school work, nor how the former could have a positive effect.
sxp 6y ago
This has a lot of words, but not enough content. I recommend 3Blue1Brown's neural net videos instead: https://www.youtube.com/watch?v=Ilg3gGewQ5U https://www.youtube.com/watch?v=Ilg3gGewQ5U
- adamfaliq 6y ago
  This. And https://ml-cheatsheet.readthedocs.io/en/latest/backpropagation.html https://ml-cheatsheet.readthedocs.io/en/latest/backpropagati... . I am currently doing MIT Micromasters in Data Science, specifically the Machine Learning & Deep learning course. Last week's assignment was to build a neural network from scratch and these two resources have been a lifesaver.
phnofive 6y ago
Try reading the last two paragraphs of the article first; it won’t make much more sense unless (I suspect) you already understood the point it was intended to convey, but you’ll save some time being confused: > By now it’s probably worth dropping the allegory: the “workers” in our story are models, which could be individual layers of a neural network, or even whole models. And the process we’ve been discussing is of course the backpropagation of gradients, which are used to iteratively update the weights of a model. > The allegory also introduced Thinc’s particular implementation strategy for backpropagation, which uses function composition. This approach lets you express neural network operations as higher-order functions. On the one hand, there are sometimes where managing the backward pass explicitly is tricky, and it’s another place your code can go wrong. But the trade-off is that there’s much less API surface to work with, and you can spend more time thinking about the computations that should be executed, instead of the framework that’s executing them. For more about how Thinc is put together, read on to its Concept and Design.
dragandj 6y ago
Let me chip in with some self-promotion. This book explains and executes every single line of code interactively, from low level operations to high-level networks that do everything automatically. The code is built on the state of the art performance operations of oneDNN (Intel, CPU) and cuDNN (CUDA, GPU). Very concise readable and understandable by humans. https://aiprobook.com/deep-learning-for-programmers/ https://aiprobook.com/deep-learning-for-programmers/ Here's the open source library built throughout the book: https://github.com/uncomplicate/deep-diamond https://github.com/uncomplicate/deep-diamond Some chapters from the beginning of the book are available on my blog, as a tutorial series: https://dragan.rocks https://dragan.rocks
29athrowaway 6y ago
Backpropagation is interesting, but there is no evidence that animal brains are using it.
ad404b8a372f2b9 6y ago
This is all you need to understand backpropagation, I give this link to all my students: https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ https://mattmazur.com/2015/03/17/a-step-by-step-backpropagat...

77 points by polm23 6y ago

This is a very, very convoluted way of understanding backprop. If you want to understand it I highly recommend writing a neural network starting from a basic perceptron (no hidden layer) or even implementing basic linear regression with stochastic gradient descent. Then work your way up to a network with one hidden layer. Try to focus on the maths and avoid looking at code examples.

selimthegrim 6y ago
Is scholastic gradient descent anything like grad student descent? ;)
reitzensteinm 6y ago
This is a great book that breaks down the process step by step. I had my own neural nets working before I touched a framework, and it demystified what would have otherwise seemed like black magic. https://www.amazon.com.au/Make-Your-Own-Neural-Network-ebook/dp/B01EER4Z4G https://www.amazon.com.au/Make-Your-Own-Neural-Network-ebook...
- 29athrowaway 6y ago
  The book I used to learn the basics many years ago was "An Introduction to Neural Networks", by Kevin Gurney. It's a bit outdated now, but I still consider it valuable.
cultyyy123 6y ago
This is a very, very convoluted way of gate keeping. > Try to focus on the maths and avoid looking at code examples. Straight out dictatorship. > If you want to understand it I highly recommend writing a neural network starting from a basic perceptron (no hidden layer) or even implementing basic linear regression with stochastic gradient descent. Then work your way up to a network with one hidden layer. Like it's only way to understand backpropagation. This quote is still valid to date. “Today's scientists have substituted mathematics for experiments, and they wander off through equation after equation, and eventually build a structure which has no relation to reality. ” - Nikola Tesla Trust me, you can't fool people for too long. Abstractions always wins the game.
- Vhano 6y ago
  I agree. Why talk in precise terms when you can do abstract mythology instead.
datameta 6y ago
I absolutely second the suggestion of working through the math of a shallow neural network to start. I would also suggest that once one thinks they have a sufficient grasp of the steps to try to implement them at a low level without frameworks, just math libraries. Personally, I found Octave (highly similar to MATLAB) to be a nice granularity tier for learning the process programmatically.

kristianov 6y ago

Why are people so afraid of math?

ganafagol 6y ago
Because "I've always been bad at math" is a hip thing to say in today's society and gets you laughter and approving nods.
- cultyyy123 6y ago
  In reality most people are bad/afraid at Numbers not at Maths. The book Maths without numbers highlight what needs to be done to Education system.
  - tgv 6y ago
    Sure. It's much easier to understand the Dedekind cut, rings, and function spaces than to subtract 35% from 140.
    - laichzeit0 6y ago
      Most people afraid of “math” only have exposure to the second kind. I figure it has more to do with finding those type of mechanical operations boring (they are), and hence all of maths must be equally boring. If you’re a software developer, enjoy abstractions, programming language theory, and solving problems there is no way you would find math of the first sort boring. Alas, in high school pretty much all the math is of the boring mechanical kind. At least mine was, anyway. Even the first exposure to non-rigorous calculus is of the shitty plug-and-chug variety.
      - tgv 6y ago
        It's hard to go through life without developing a modest understanding of arithmetic and reading. You wouldn't even be able to check your receipts if school started with Peano's axioms. Consequently, numbers come first, as they have for millennia now. Which makes sense, because maths developed out of intuitive arithmetic. As a matter of fact, my parents complain they never understood maths, because they were taught algebraic rules by rote. a * b = b * a. They can recite that, they just don't have a clue what to do with it. Teachers just aren't good, and certainly not at maths. It's a struggle to teach billions the basic skills. There is no solution. I also don't know any mathematician or physicist bad at arithmetic. Quite a few of them seem to delight in numbers and arithmetic puzzles. So I really don't see how to avoid numbers and boring school work, nor how the former could have a positive effect.

sxp 6y ago

This has a lot of words, but not enough content. I recommend 3Blue1Brown's neural net videos instead: https://www.youtube.com/watch?v=Ilg3gGewQ5U https://www.youtube.com/watch?v=Ilg3gGewQ5U

adamfaliq 6y ago
This. And https://ml-cheatsheet.readthedocs.io/en/latest/backpropagation.html https://ml-cheatsheet.readthedocs.io/en/latest/backpropagati... . I am currently doing MIT Micromasters in Data Science, specifically the Machine Learning & Deep learning course. Last week's assignment was to build a neural network from scratch and these two resources have been a lifesaver.

phnofive 6y ago

Try reading the last two paragraphs of the article first; it won’t make much more sense unless (I suspect) you already understood the point it was intended to convey, but you’ll save some time being confused: > By now it’s probably worth dropping the allegory: the “workers” in our story are models, which could be individual layers of a neural network, or even whole models. And the process we’ve been discussing is of course the backpropagation of gradients, which are used to iteratively update the weights of a model. > The allegory also introduced Thinc’s particular implementation strategy for backpropagation, which uses function composition. This approach lets you express neural network operations as higher-order functions. On the one hand, there are sometimes where managing the backward pass explicitly is tricky, and it’s another place your code can go wrong. But the trade-off is that there’s much less API surface to work with, and you can spend more time thinking about the computations that should be executed, instead of the framework that’s executing them. For more about how Thinc is put together, read on to its Concept and Design.

dragandj 6y ago

Let me chip in with some self-promotion. This book explains and executes every single line of code interactively, from low level operations to high-level networks that do everything automatically. The code is built on the state of the art performance operations of oneDNN (Intel, CPU) and cuDNN (CUDA, GPU). Very concise readable and understandable by humans. https://aiprobook.com/deep-learning-for-programmers/ https://aiprobook.com/deep-learning-for-programmers/ Here's the open source library built throughout the book: https://github.com/uncomplicate/deep-diamond https://github.com/uncomplicate/deep-diamond Some chapters from the beginning of the book are available on my blog, as a tutorial series: https://dragan.rocks https://dragan.rocks

29athrowaway 6y ago

Backpropagation is interesting, but there is no evidence that animal brains are using it.

ad404b8a372f2b9 6y ago

This is all you need to understand backpropagation, I give this link to all my students: https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ https://mattmazur.com/2015/03/17/a-step-by-step-backpropagat...