6 ms·

Basics of Neural Networks with example codes and illustrations

231 points by coderjack 13y ago

pests 13y ago
In regards to the first interactive demo, it seems to be adjusting the line to be parallel to the one drawn on the background. Was this intentional or are the supposed to converge?
- coderjack 13y ago
  The line in the background is the function towards which the neural net is expected to converge.
  - pests 13y ago
    When I first posted my reply every time I watched that demo the neural net line was converging on a line perpendicular to the line in the background. (rereading my original comment I think I described what I was seeing incorrectly but this new reply correctly explains what I was originally seeing)
NKCSS 13y ago
An awesome book; I've now started reading from the beginning of the book :) One thing I've noticed though, is that img 10 of chapter 1 is missing. http://natureofcode.com/book/chapter-1-vectors/imgs/chapter01/ch01_10.png http://natureofcode.com/book/chapter-1-vectors/imgs/chapter0...
- CodeCube 13y ago
  You can submit a pull request ;) https://github.com/shiffman/The-Nature-of-Code https://github.com/shiffman/The-Nature-of-Code I think that's so amazingly awesome, that it can evolve as a living document in this way.
mekarpeles 13y ago
The experience (specifically the careful choice of mediums + examples + presentation though which the concepts are conveyed) is pretty fantastic.
- coderjack 13y ago
  You may also want to read the chapter on fractal programming in this book. Its pretty intuitive too.
gustavodemari 13y ago
nice article
gustavodemari 13y ago
nice article
Lambdanaut 13y ago
This book is everything I've ever wanted in a programming text. I'm sorry that I don't have much of anything substantial to say except praise, but seriously, thank you for writing this.
- coderjack 13y ago
  Thanks for liking this post. But the actual credits must go to the author of this text as I am just another fan of this book like you are now.
JDDunn9 13y ago
Does anyone have any examples of areas where neural networks beat out statistical based methods, other than maybe image recognition? I can't even think of another major area where they dominate. - Search engines use algorithms, not neural nets. - The most popular algorithm on Kaggle (data analysis competitions) is random forests - Google's self-driving car uses statistical-based methods I can't imagine commercial aircraft would use a neural net. What happened if one crashed? They would analyze the data and ask questions like, Q: "What happened?" A: "I don't know" Q: "Can we fix it so it doesn't happen again?" A: "I don't know".
- dave_sullivan 13y ago
  Glad you asked... Definitely image recognition: http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf Speech recognition: http://www.cs.toronto.edu/~hinton/absps/RNN13.pdf http://www.cs.toronto.edu/~hinton/absps/RNN13.pdf Natural language processing: http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutorial http://www.socher.org/index.php/DeepLearningTutorial/DeepLea..., http://aclweb.org/anthology/N/N13/N13-1090.pdf http://aclweb.org/anthology/N/N13/N13-1090.pdf If you're into kaggle competitions: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st-place-interview/ http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it... I don't think there are going to be any further major advances in eg SVMs or random forests (famous last words maybe...) Neural nets, on the other hand, are just scratching the surface of what's possible. So right now they are state of the art in some historically very difficult areas. But these are early days still.
  - lightcatcher 13y ago
    You wrote the answer I was just a little bit too lazy to write... As to the GP: Geoff Hinton (probably the most well-known neural networks researcher) said in his Coursera course that neural networks thrive at problems with a lot of structure that could be encoded, while simpler models like SVMs or Gaussian processes might be better for problems without as much deep structure to discover. Also, a lot of the current research with neural networks involves using neural networks to learn better representations of data. These cleaner representations of data (which can be thought about as a sort of semantic PCA) often make classification far easier, which explains the great results. Learning representations also makes transfer learning (transferring knowledge from one domain to another) much easier/more possible.
catshirt 13y ago
cool. just bought the book on Amazon! i know some small amount about neural networks (i was able to skim the article), but the book as a whole looks stellar.
nrox 13y ago
brain.js library is a NN implementation in JavaScript. It's very easy to use. https://github.com/harthur/brain https://github.com/harthur/brain Here is a test with a model of a robotic arm: https://assemblino.com/show/public20123372.html https://assemblino.com/show/public20123372.html
bcuccioli 13y ago
I wrote a simple neural network about a year ago for doing optical character recognition as a class project. I think looking over the code could be good for learning, as it has a pretty simple OOP structure: https://github.com/bcuccioli/neural-ocr https://github.com/bcuccioli/neural-ocr

231 points by coderjack 13y ago

pests 13y ago

In regards to the first interactive demo, it seems to be adjusting the line to be parallel to the one drawn on the background. Was this intentional or are the supposed to converge?

coderjack 13y ago
The line in the background is the function towards which the neural net is expected to converge.
- pests 13y ago
  When I first posted my reply every time I watched that demo the neural net line was converging on a line perpendicular to the line in the background. (rereading my original comment I think I described what I was seeing incorrectly but this new reply correctly explains what I was originally seeing)

NKCSS 13y ago

An awesome book; I've now started reading from the beginning of the book :) One thing I've noticed though, is that img 10 of chapter 1 is missing. http://natureofcode.com/book/chapter-1-vectors/imgs/chapter01/ch01_10.png http://natureofcode.com/book/chapter-1-vectors/imgs/chapter0...

CodeCube 13y ago
You can submit a pull request ;) https://github.com/shiffman/The-Nature-of-Code https://github.com/shiffman/The-Nature-of-Code I think that's so amazingly awesome, that it can evolve as a living document in this way.

mekarpeles 13y ago

The experience (specifically the careful choice of mediums + examples + presentation though which the concepts are conveyed) is pretty fantastic.

coderjack 13y ago
You may also want to read the chapter on fractal programming in this book. Its pretty intuitive too.

gustavodemari 13y ago

nice article

gustavodemari 13y ago

nice article

Lambdanaut 13y ago

This book is everything I've ever wanted in a programming text. I'm sorry that I don't have much of anything substantial to say except praise, but seriously, thank you for writing this.

coderjack 13y ago
Thanks for liking this post. But the actual credits must go to the author of this text as I am just another fan of this book like you are now.

JDDunn9 13y ago

Does anyone have any examples of areas where neural networks beat out statistical based methods, other than maybe image recognition? I can't even think of another major area where they dominate. - Search engines use algorithms, not neural nets. - The most popular algorithm on Kaggle (data analysis competitions) is random forests - Google's self-driving car uses statistical-based methods I can't imagine commercial aircraft would use a neural net. What happened if one crashed? They would analyze the data and ask questions like, Q: "What happened?" A: "I don't know" Q: "Can we fix it so it doesn't happen again?" A: "I don't know".

dave_sullivan 13y ago
Glad you asked... Definitely image recognition: http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf Speech recognition: http://www.cs.toronto.edu/~hinton/absps/RNN13.pdf http://www.cs.toronto.edu/~hinton/absps/RNN13.pdf Natural language processing: http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutorial http://www.socher.org/index.php/DeepLearningTutorial/DeepLea..., http://aclweb.org/anthology/N/N13/N13-1090.pdf http://aclweb.org/anthology/N/N13/N13-1090.pdf If you're into kaggle competitions: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st-place-interview/ http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it... I don't think there are going to be any further major advances in eg SVMs or random forests (famous last words maybe...) Neural nets, on the other hand, are just scratching the surface of what's possible. So right now they are state of the art in some historically very difficult areas. But these are early days still.
- lightcatcher 13y ago
  You wrote the answer I was just a little bit too lazy to write... As to the GP: Geoff Hinton (probably the most well-known neural networks researcher) said in his Coursera course that neural networks thrive at problems with a lot of structure that could be encoded, while simpler models like SVMs or Gaussian processes might be better for problems without as much deep structure to discover. Also, a lot of the current research with neural networks involves using neural networks to learn better representations of data. These cleaner representations of data (which can be thought about as a sort of semantic PCA) often make classification far easier, which explains the great results. Learning representations also makes transfer learning (transferring knowledge from one domain to another) much easier/more possible.

catshirt 13y ago

cool. just bought the book on Amazon! i know some small amount about neural networks (i was able to skim the article), but the book as a whole looks stellar.

nrox 13y ago

brain.js library is a NN implementation in JavaScript. It's very easy to use. https://github.com/harthur/brain https://github.com/harthur/brain Here is a test with a model of a robotic arm: https://assemblino.com/show/public20123372.html https://assemblino.com/show/public20123372.html

bcuccioli 13y ago

I wrote a simple neural network about a year ago for doing optical character recognition as a class project. I think looking over the code could be good for learning, as it has a pretty simple OOP structure: https://github.com/bcuccioli/neural-ocr https://github.com/bcuccioli/neural-ocr