17 ms·
What is backpropagation and what is it doing? [video]
- nouveaux 9y agoThis was very timely for me and for anyone else learning, here are the first few videos of the series: https://www.youtube.com/watch?v=aircAruvnKk https://www.youtube.com/watch?v=aircAruvnKk https://www.youtube.com/watch?v=IHZwWFHWa-w https://www.youtube.com/watch?v=IHZwWFHWa-w https://www.youtube.com/watch?v=Ilg3gGewQ5U https://www.youtube.com/watch?v=Ilg3gGewQ5U (Original video) https://www.youtube.com/watch?v=tIeHLnjs5U8 https://www.youtube.com/watch?v=tIeHLnjs5U8
- samueloph 9y agooh my god, as soon as i saw this video was from 3Blue1Brown i immediately thought "this gonna be good!". I didn't realize he was posting a Deep Learning series.
- perfmode 9y agoEach one of these videos consists of thousands of lines of code. The attention to detail is impressive.
- posterboy 9y agohave you looked at visualizations done in mathematica? LoC is not a good measure here. http://community.wolfram.com/content?curTag=graphics%20and%20visualization http://community.wolfram.com/content?curTag=graphics%20and%2...
- kharms 9y agoIn my opinion this author produces the best math videos on youtube. If you can afford it and enjoyed this video, consider supporting him on Patreon. https://www.patreon.com/3blue1brown https://www.patreon.com/3blue1brown
- SonOfLilit 9y agoIf you liked him you will love acko.net. Try https://acko.net/blog/how-to-fold-a-julia-fractal/ https://acko.net/blog/how-to-fold-a-julia-fractal/
- smortaz 9y agoIf you enjoy his videos (and other creators'), please consider signing up on Patreon and supporting them.
- aidos 9y agoSomeone on here (I think) recommended his videos on linear algebra a while back and I've since watched them all, several times. A couple of hours of watching time built an intuition and understanding of linear algebra and the broader maths around it that 4 years of university training didn't give me. That's a little unfair, because I obviously learnt a lot on the courses that make these videos easier to understand, but man, they're so well done.
- NuclearFishin 9y agoTotally agree. Had exactly the same experience. Became a patron as a way to offer my thanks!
- adamnemecek 9y agoThere needs to be some sort of organized push for visualization tools. I know, I might be bringing the proverbial owls to the proverbial Athens with saying that here, but I really do feel that if done right this could impact the course of the world like nothing else. This could be as important as idk, invention of book press or smth. Make computer "the visualization machine". I think that one of the fundamental problems is that to be a visualization machine, you need to have easy access of the GPU and OpenGL is provides anything but. I think that shadertoy (shadertoy.com) is the thing that comes the closest but the learning curve is kinda steep. I know that people like Alan Kay, Bret Victor or Michael Nielsen (his post was on the fp the other day https://news.ycombinator.com/item?id=15616637 https://news.ycombinator.com/item?id=15616637) share these sentiments but this is a task bigger than a single people. Idk what I really mean by "organized push". I'm not sure if the problem is well defined too
- alfla 9y agoI agree. Visualization is often key to understanding and identifying non-trivial issues. Here's a tool a colleague of mine made for inline "visual debugging" for e.g. computer vision, written in c++: https://github.com/lightbits/vdb https://github.com/lightbits/vdb. I haven't used it myself, but when he presented it I think it made a lot of sense to have these sorts of tools for processing data in real time.
- minimaxir 9y agoIn deep learning, TensorBoard (https://www.tensorflow.org/get_started/summaries_and_tensorboard https://www.tensorflow.org/get_started/summaries_and_tensorb...) works with TensorFlow and Keras to show what the model is doing. However, it ends up being more complicated/unintuitive than a YouTube video, so it's not as useful.
- adamnemecek 9y agoThe problem is that this is an ad hoc solution. What I'm talking about would be some formalization of visualization (I guess kinda like grammar of graphics without the statistical aspect) so you can visualize just about anything.
- SonOfLilit 9y agoWe need more people teaching math through visual intuition. Life a friend of mine said, "if you want to do computation fast, phrase it as a problem for your GPU, er, visual cortex". Here is a tool you can play with to visualize this: http://playground.tensorflow.org http://playground.tensorflow.org If you liked this video, try this different visual intuition of what a neural network does that I find even better: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ Also, remember that back propagation is a very general algorithm, it works not only on linear transformation weights but on any direct acyclic computation graph that is differentiable in its weights.
- phreeza 9y agoThis can also go wrong, for example visualising probability distributions in low dimensions leads to very wrong intuitions about the behavior of high-dimensional dimensional distributions.
- vbuwivbiu 9y agoplease elaborate - are you thinking of the curse of dimensionality ?
- phreeza 9y agoThere are many examples, one I came across recently is that the large majority of the probability mass of a high-dimensional gaussian distribution is in a shell at a distance from the mean, the mass at the center is actually quite low. Also anything related to topology, which is important when you are looking at decision boundaries, becomes counterintuitive in high dimensions, because so many things can be adjacent at the same time.
- shas3 9y agoTheoretically, yes. Can you give a more concrete example? Many hard high dimensional and general topology problems can be visualized through their 2D special cases.
- phkahler 9y agoWhat tools does a person use to make a video like this? I've been wanting to do the same on my topic of expertise for a while now.
- henrikeh 9y agoHe uses custom, self-developed tools http://www.3blue1brown.com/about/ http://www.3blue1brown.com/about/ https://github.com/3b1b/manim https://github.com/3b1b/manim
- lelandbatey 9y agoHe creates each animation using a set of Python tools and libraries he wrote. You can find them published here: https://github.com/3b1b/manim https://github.com/3b1b/manim
- edanm 9y agoHe uses custom tools. However, he actually recommends against using his tools. He suggests a better option is to use traditional animation tools. I'm actually not sure what one would use for more traditional animations of his style though. I mean, theoretically you can use blender/etc for most 3d things, but how easy would it be to make something math-based there? Hopefully someone with some real animation experience can chime in.
- mcintyre1994 9y agoOn the Manim Github he has some suggestions: "For 9/10 math animation needs, you'd probably be better off using a more well-maintained tool, like matplotlib, mathematica or even going a non-programatic route with something like After Effects. I also happen to think the program "Grapher" built into osx is really great, and surprisingly versatile for many needs."
- raverbashing 9y ago> I also happen to think the program "Grapher" built into osx is really great I didn't know about this, and it's a nice find
- quotemstr 9y agoThe entire YouTube channel is fantastic. 3Blue1Brown's series on linear algebra is the best I've seen anywhere.
- wybiral 9y agoAgreed. Pretty much every video on that channel is just as good as this one.
- mlamat 9y agoThank you very much. I must code a neural network with backpropagation for my AI class. Can anyone recommend a book?
- fnbr 9y agoIf you're looking to understand the underlying theory behind deep learning, the Deep Learning book by Goodfellow et al. is awesome. If you're interested in general machine learning, the Elements of Statistical Learning, by Tibsihirani et. al is great; a more applied book is Applied Statistical Learning by the same author. For a more applied view, I'd check out Tensorflow or PyTorch tutorials; there's no good book, as far as I'm aware, because the tech changes so quickly. I've done a series of videos on how to do deep learning that might be useful; if you're interested, there's a link in my profile.
- Yreval 9y agoThis book is a good practical introduction that walks you through the basic ideas as you develop some basic functionality. http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/ I'm often pretty skeptical of e-books and self publications, but the above link is pretty good (and the video series linked here references it as well.) The Goodfellow book that another commenter mentioned is a high-quality survey of the field and a nice, high-level overview of different research directions in deep learning, but isn't as pragmatic as an introduction.
- bmc7505 9y agoCOMP 551?
- alexasmyths 9y agoThis video series is amazing and I wish it existed long ago.
- i25959341 9y agoadsf
- cmatt01 9y agoThere needs to be some sort of intense advocation for visualization tools.
- oliv__ 9y agoI dove into this not knowing anything about neural networks, but the feeling I came out of it with was incredible: I love it when something blurry and obscure slowly morphs into a sharper picture in your mind, it's so empowering.
- afarrell 9y agoA bit of a side-note, but I think it is an interesting piece of marketing that Amplify Partners decided to sponsor[1] the previous video in this series. I wonder (and hope) we'll see more VCs sponsoring open educational content relevant to their focus. [1] https://www.youtube.com/watch?v=IHZwWFHWa-w&t=1205 https://www.youtube.com/watch?v=IHZwWFHWa-w&t=1205
- timonoko 9y agoIn 1988 Teuvo Kohonen had an "animation" with rotating disks how the perceptron learns https://youtu.be/Qy3h7kT3P5I?t=42m24s https://youtu.be/Qy3h7kT3P5I?t=42m24s. Did not help comprehension much.
- crusso 9y agoI played around with ML a few years back. I took the Andrew Ng course on Coursera and spent some time with some python notebooks - but I never did anything with it beyond just proving that I could follow the examples and implement my own ML solutions for some simple training sets. Now I have some problems I'd like to solve with ML. So assuming that I understand the basic concepts, what's the HN recommendation for a good library/system to get started with on doing some practical ML with neural nets? Would TensorFlow be the best way to get into it?
- ranman 9y agoCheckout MXNet as well
- ranman 9y agohttps://mxnet.incubator.apache.org/tutorials/ https://mxnet.incubator.apache.org/tutorials/
- andreyk 9y agoKeras is probably the best library to get started - Ternsorflow is mainly needed if you need to mess with lower-level details, which for many things is not really necessary.
- crusso 9y agoWould you still recommend Keras for non-image work?