8 ms·
Backpropagation algorithm visual explanation
- Chilinot 8y agoNice demonstration, but misses to explain the bias value in the forward propagation step. While quite important, this value is often left out when demonstrating the propagation function. So having it in warrants a short description in my opinion. It also skips over the bias value in the back propagation step.
- ironSkillet 8y agoThe bias is just another input node whose value happens to be constant (granted with full connectivity), right? So the motivating idea/derivation of backpropogation doesn't change.
- Chilinot 8y agoThe bias is often updated/corrected in the back propagation step as well. The purpose of the value is to shift the activation function along the x-axis basically. While the weights define the slope of the activation function.
- londons_explore 8y agoI find explanations like these are great for people who understand mathematical notation, sigmoid functions, derivitives, etc. These people however typically understand whats going on from a simple text description of the process. For those without a math background, the notation is very opaque. A far better explanation is to explain it numerically with simple examples. For example, have two bits of training data: input -> output 1 -> 0 0 -> 1 And a simple network with zero hidden nodes and train it. By hand... Then add another bit of training data: 0.5 -> 1.5 Notice that it is now impossible to fit the training data exactly, however many training iterations we do. Now add a hidden layer with one or two nodes. Now we can perfectly fit the data, but show that depending on initialization weights we might never get there through gradient descent. Nows the time to mention different types of optimizers, momentum, etc.
- newen 8y agoKind of tired of people in the programming community proudly complaining about not knowing simple math notation. Educate yourself. Everyone else in the engineering world knows these basic math notations.
- londons_explore 8y agoThats the point I'm making... If you've mastered this "simple" notation, then you've probably already mastered simple neural networks like this, so you are not the target for this tutorial.
- alanbernstein 8y agoSummation and derivatives are used quite broadly outside of neural networks.
- telchar 8y agoMath notation is simple because it's heavily overloaded. For example, it does me no good to know about exponentials when a superscript is used in a different context. Reading any nontrivial math requires knowing what the notation means in the particular context of the work. IME mathematicians typically assume the reader is familiar already and usually don't explain. Anyone targeting their work at non-experts should explain even what seems like trivial notation to them since they can't know what other meanings the reader may think the notation holds.
- dmartinez 8y agoI agree. The superscript example is a good one. In most contexts it refers to the exponent, but in the context of a cost function that minimizes a linear regression (for example), it indicates the index of the set. Computer languages benefit from the fact that poorly designed syntax can be deprecated (not in all cases, e.g. C++) by introducing new features to the language. Notation in math never advances in the same way for some reason.
- 8y ago
- sandGorgon 8y agoDoes anyone know if there is a tool that helps to create these kind of presentations?
- pc86 8y agoThis just uses http://scrollerjs.com http://scrollerjs.com and https://d3js.org https://d3js.org. Any tool that combines these things is by necessity going to limit your creativity with them. Using the tools themselves is your best bet.
- mathisonian 8y agohttps://idyll-lang.org/ https://idyll-lang.org/ is built for this. You'll still have to write code for your custom graphics but it will help you get things up and running quickly. Check out https://mathisonian.github.io/idyll/scaffolding-interactives/ https://mathisonian.github.io/idyll/scaffolding-interactives... (scroll example is towards the bottom)
- stared 8y agoIt is only me who gets frustrated by networks drawn upside-down (i.e. data flowing from down to up? IMHO it is a poor convetion, mindlessly repeated. In English we read from top to bottom. Data flows (be it equations or flow charts) typically follow the same convention, so we can read articles in in a coherent way. Even trees (both data structures and decision trees), grow from their roots downwards (so, against their original biological metaphor). At least most of researchers write neural networks from left to right, consistent with English. More of this point: https://www.reddit.com/r/MachineLearning/comments/6j28t9/d_why_do_people_draw_neural_networks_upside_down/ https://www.reddit.com/r/MachineLearning/comments/6j28t9/d_w...
- zerostar07 8y agoforward and back usually refer to right- and leftwards in 2D so it should probably be drawn left-to-right
- stared 8y agoIt's not clear what you mean. - top->bottom is not compatible with left->right - "back" propagation is "back" for a reason, so it should go against the normal (forward) direction
- zerostar07 8y agoE.g. in Feynman diagrams time flows forwards from left to right.
- deleted 8y ago[deleted]
- munificent 8y agoOnly in cultures whose writing system reads left-to-right. :)
- zerostar07 8y agomath is generally written left-to-right
- nsthorat 8y agoUnfortunately there is no attribution, but this tool was created by Daniel Smilkov, who also built TensorFlow Playground and who is a cocreator of TensorFlow.js. https://twitter.com/dsmilkov https://twitter.com/dsmilkov
- ehsankia 8y agoAlso, going back one on the URL leads you to the Google ML crash course, which I guess this is part of: https://google-developers.appspot.com/machine-learning/crash-course/ https://google-developers.appspot.com/machine-learning/crash... Definitely worth checking out.
- michaelmior 8y agoAm I the only one who starts seeing some LaTeX creep in halfway through instead of the rendered formulas?
- inputcoffee 8y agoGreat Viz. Although my favorite is still the video of Andrej Karpathy's Stanford lecture explaining it. He sort of goes through some other implications which, from an intuition massaging pov, is great. https://www.youtube.com/watch?v=i94OvYb6noo https://www.youtube.com/watch?v=i94OvYb6noo For those who are impressed by such things -- as I am -- he is now head of AI or ML or something at Tesla.
- jorgeleo 8y agoLove the animation and the math explanation. Only one, I hope constructive, criticism: Too many formulas without numbers. It will help the explanation if you include numbers and how the results are calculated. Not everybody is comfortable with the chain rule to distribute the error across the individual weights
- H1Supreme 8y agoAgree. Although I've been putting effort into learning more Calculus (former art student!), Linear Algebra, and the like, a version of this with actual numbers would go a long way.
- apexalpha 8y agoWhat a nice website. No weird flashy banners, no external scripts, zoomable, no ads, no tracking. Just text accompanied by great visualisations. Kudos!
- pastelsky 8y agoWhile we're at this, I found the explainations by 3Brown1Blue to be very intuitive when it comes to neural networks, especially for folks who're new and don't necessarily grasp concepts when explained primarily through math. What is backpropagation really doing? https://youtu.be/Ilg3gGewQ5U https://youtu.be/Ilg3gGewQ5U His other videos on this topic are just as good.
- 2bitencryption 8y agoquestion about this part: > f(x) has to be a non-linear function, otherwise the neural network will only be able to learn linear models. I thought one of the most common functions was relu, which is linear (but cuts off to 0 for x values below 0) https://en.wikipedia.org/wiki/Rectifier_(neural_networks) https://en.wikipedia.org/wiki/Rectifier_(neural_networks)
- brchr 8y agoIt’s exactly the cutoff at zero that puts the "kink" into the function which makes ReLU nonlinear. :)
- mehh 8y agounderwhelming
- pvg 8y agoYou don't need to sign your comments on HN.
- teekert 8y agoSorry, this is not "visual explanation", this is "math heavy explanation with some drawings next to it".
- jakemor 8y agothis is so freaking cool.
- keithnz 8y agoI found in firefox scrolling down left a lot of thing greyed out, I had to highlight and unhighlight things to get them to show properly.