4 ms·

Author here. I'd love if you could give me pointers to your favourite explanations of how neural networks function (even better, if you can say what you partic

Author here. I'd love if you could give me pointers to your favourite explanations of how neural networks function (even better, if you can say what you particularly liked about them). (I really enjoyed reading the remainder of your comment.)

zackmorris 13y ago
Wow cool, sorry I didn't mean to sound critical, I should have used (perhaps) instead of (arguably). I had to search my bookmarks but honestly can't remember if one of these was the explanation I remember (it could have been in a book when Borders bookstore was still around): http://natureofcode.com/book/chapter-10-neural-networks/ http://natureofcode.com/book/chapter-10-neural-networks/ http://zerkpage.tripod.com/index.htm http://zerkpage.tripod.com/index.htm http://www.ai-junkie.com/ann/evolved/nnt1.html http://www.ai-junkie.com/ann/evolved/nnt1.html Then I used to skim this compilation before it went away: https://web.archive.org/web/20030601074826/http://www.emsl.pnl.gov:2080/proj/neuron/neural/systems/shareware.html https://web.archive.org/web/20030601074826/http://www.emsl.p... Mostly what I remember about the explanation that stood out was that it was concise. So the way it described backpropagation just "clicked" in a way that the previous articles that used tons of summation/matrix math/probability and calculus had not. I’ve read some positively atrocious papers where it was like the authors were showing their notation prowess rather than conveying information. Your chapters have a good balance, although for someone who’s been out of college for 15 years like me, it might be nice to have a tiny refresher about derivatives before you dive in, especially over several variables/dimensions. I mean like a paragraph, to explain the notation, because the knowledge is in my head but foggy. Everything else was very good, for example when you explained how bias comes from the threshold factor on the other side but makes notation easier. I don’t remember seeing it explained quite that way before. Oh and the sigmoid function always seemed arbitrary too (and frankly turned me off from neural nets because it seemed too analog), but explaining how it simplifies derivatives makes perfect sense now. Unfortunately with other books, I was also never able to find off-the-shelf code that was approachable or up to date, so I ended up forgetting everything I had learned and had to start over each time until I got tired of trying. I very much like your 72 line code example, where you provide a backpropagation function without explaining it yet. That’s ok, as programmers we encounter that all of the time, and it’s actually kind of fun to decipher the algorithm before reading how it works later. I believe it even allows for multiple hidden layers which is just fantastic. As for the other stuff, well, I just look at it like this: Ask a developer to write an iOS app that say, interacts with the twitter or facebook API and presents the user’s N closest friends in a list view and then submit that to the app store, and they can do it from start to finish with no mystery (other than a little help from stack overflow). Each of those steps replies on some pretty heavy lifting like REST APIs, possibly SQL, message passing in objective-c, even a rudimentary understanding of security and encryption for networking and app submission, yet those things have become mainstream. But ask a developer to do even the simplest machine learning task, like a little data mining or spam filtering, heaven forbid weak AI like image recognition, and it’s a whole different can of worms. Why hasn’t anyone standardized this stuff in the languages that developers use every day, as opposed to MATLAB or prolog or whatnot? Why can’t I literally read a text file and pipe it through a shell script that does a Bayes classifier? Why doesn’t iOS have an AI.framework to go along with its Social.framework? To me these aren’t all that much to ask. I really feel that we’re still working at the assembly language or hand rolled affine texture mapping level from the 80s and 90s with respect to AI today. We don’t have something like the web metaphor yet to catapult it into the mainstream. That said, I’m extremely optimistic that concurrent (or at least parallelized) languages like Octave/NumPy/Go, and the more approachable functional languages like Haskell/Scala/Clojure might help deliver APIs that we can interact with from more mainstream languages. To me, most weak AI problems have been solved now, and it’s time that programmers have an intuitive understanding of how the algorithms work so that they can combine them in novel ways and get to strong AI in, well, my lifetime hah. Thanks for your efforts and keep it up!
- michael_nielsen 13y ago
  Belated thanks, that's extremely helpful. I'll take a look at those links which are unfamiliar to me (I've already read the Nature of Code link, it's a great book.) Tidbits: + Yes, the code presented "works" for multiple hidden layers. But it converges too slowly to be useful, except for very small networks. Later chapters introduce new ideas which help backprop converge faster, and which start to make multiple hidden layers quite feasible. + On the broader machine learning front, I think scikit learn is a pretty nice general library: it's approachable, easy to use, and has reasonable docs. It'll be interesting to see how it develops. + For neural nets in particular, elsewhere in this thread Dave Sullivan has done a nice job distilling out a list of good libraries. I think it'll be a while before there's a real out-of-the-box solution for neural nets, though, since setting parameters for neural nets is something of an art.

zackmorris 13y ago

Wow cool, sorry I didn't mean to sound critical, I should have used (perhaps) instead of (arguably). I had to search my bookmarks but honestly can't remember if one of these was the explanation I remember (it could have been in a book when Borders bookstore was still around): http://natureofcode.com/book/chapter-10-neural-networks/ http://natureofcode.com/book/chapter-10-neural-networks/ http://zerkpage.tripod.com/index.htm http://zerkpage.tripod.com/index.htm http://www.ai-junkie.com/ann/evolved/nnt1.html http://www.ai-junkie.com/ann/evolved/nnt1.html Then I used to skim this compilation before it went away: https://web.archive.org/web/20030601074826/http://www.emsl.pnl.gov:2080/proj/neuron/neural/systems/shareware.html https://web.archive.org/web/20030601074826/http://www.emsl.p... Mostly what I remember about the explanation that stood out was that it was concise. So the way it described backpropagation just "clicked" in a way that the previous articles that used tons of summation/matrix math/probability and calculus had not. I’ve read some positively atrocious papers where it was like the authors were showing their notation prowess rather than conveying information. Your chapters have a good balance, although for someone who’s been out of college for 15 years like me, it might be nice to have a tiny refresher about derivatives before you dive in, especially over several variables/dimensions. I mean like a paragraph, to explain the notation, because the knowledge is in my head but foggy. Everything else was very good, for example when you explained how bias comes from the threshold factor on the other side but makes notation easier. I don’t remember seeing it explained quite that way before. Oh and the sigmoid function always seemed arbitrary too (and frankly turned me off from neural nets because it seemed too analog), but explaining how it simplifies derivatives makes perfect sense now. Unfortunately with other books, I was also never able to find off-the-shelf code that was approachable or up to date, so I ended up forgetting everything I had learned and had to start over each time until I got tired of trying. I very much like your 72 line code example, where you provide a backpropagation function without explaining it yet. That’s ok, as programmers we encounter that all of the time, and it’s actually kind of fun to decipher the algorithm before reading how it works later. I believe it even allows for multiple hidden layers which is just fantastic. As for the other stuff, well, I just look at it like this: Ask a developer to write an iOS app that say, interacts with the twitter or facebook API and presents the user’s N closest friends in a list view and then submit that to the app store, and they can do it from start to finish with no mystery (other than a little help from stack overflow). Each of those steps replies on some pretty heavy lifting like REST APIs, possibly SQL, message passing in objective-c, even a rudimentary understanding of security and encryption for networking and app submission, yet those things have become mainstream. But ask a developer to do even the simplest machine learning task, like a little data mining or spam filtering, heaven forbid weak AI like image recognition, and it’s a whole different can of worms. Why hasn’t anyone standardized this stuff in the languages that developers use every day, as opposed to MATLAB or prolog or whatnot? Why can’t I literally read a text file and pipe it through a shell script that does a Bayes classifier? Why doesn’t iOS have an AI.framework to go along with its Social.framework? To me these aren’t all that much to ask. I really feel that we’re still working at the assembly language or hand rolled affine texture mapping level from the 80s and 90s with respect to AI today. We don’t have something like the web metaphor yet to catapult it into the mainstream. That said, I’m extremely optimistic that concurrent (or at least parallelized) languages like Octave/NumPy/Go, and the more approachable functional languages like Haskell/Scala/Clojure might help deliver APIs that we can interact with from more mainstream languages. To me, most weak AI problems have been solved now, and it’s time that programmers have an intuitive understanding of how the algorithms work so that they can combine them in novel ways and get to strong AI in, well, my lifetime hah. Thanks for your efforts and keep it up!

michael_nielsen 13y ago
Belated thanks, that's extremely helpful. I'll take a look at those links which are unfamiliar to me (I've already read the Nature of Code link, it's a great book.) Tidbits: + Yes, the code presented "works" for multiple hidden layers. But it converges too slowly to be useful, except for very small networks. Later chapters introduce new ideas which help backprop converge faster, and which start to make multiple hidden layers quite feasible. + On the broader machine learning front, I think scikit learn is a pretty nice general library: it's approachable, easy to use, and has reasonable docs. It'll be interesting to see how it develops. + For neural nets in particular, elsewhere in this thread Dave Sullivan has done a nice job distilling out a list of good libraries. I think it'll be a while before there's a real out-of-the-box solution for neural nets, though, since setting parameters for neural nets is something of an art.