8 ms·

Introduction to Artificial Neural Networks – Part 1

168 points by bananacurve 13y ago

maaaats 13y ago
Does anyone here have any good guides/articles about the structure of the artificial network? At university I solved simple problems in a course with ANs, but the structure of my network (hidden layers, how they were connected etc.) were basically just try/fail until I achieved ok results.
- howeman 13y ago
  I think that's basically it. If you want to do a lot of work you can do somethings with cross validation (see which parameters perform the best), but that's a form of automatic try/fail
  - pilooch 13y ago
    hyperparameter search is a solution. See http://jmlr.org/papers/v13/bergstra12a.html http://jmlr.org/papers/v13/bergstra12a.html and http://jaberg.github.io/hyperopt/ http://jaberg.github.io/hyperopt/
- dave_sullivan 13y ago
  There aren't any I'm aware of that tie this question together nicely, but here are the general structures in use right now. Google will lead you to more information on each. Standard feed forward deep net: like the ones you used at university but with a few important features. One, you can stack layers on top of each other then train the whole network with back propagation. The nonlinearity you use can be important (rectified nonlinear units--Max(0,x) are popular now). Regularization can be important (with dropout being a popular method). Pre-processing your data can also be important (eg scaling the inputs, subtracting the mean, ...). How you initialize your weights is important. Using tricks like momentum and learning rate decay are important too. Autoencoder: A standard feed forward deep net that tries to output an "uncorrupted" sample of a "corrupted" input. So basically, take an image, add noise, ask it to output the denoised image. Why you ask? Well, in doing this, the hidden units (the layers in between the input and output) tend to discover new features which can then be used elsewhere in machine learning pipelines. The big win here is automated feature engineering. RBM: Restricted boltzman machine, sort of like an autoencoder but not. I'm not convinced one is better than the other, but there are definitely differences. Recurrent neural networks: Like standard feed forward deep nets but extended to time series problems. The hidden units at each timestep feed in as additional input to the next timestep (along with the new input). So basically, each step it gets this as input: [whatever came before described as hidden units] + [input at T]. These are my personal favorite and currently hold state of the art in speech recognition, at least academically. Convolutional neural networks: State of the art in vision, this one is kind of hard to explain quickly, but you can think of it kind of like this: I take some image where I want to recognize objects in it. Then I train a bunch of neural nets that represent different aspects of an image (one net might look for vertical lines, another might look for horizontal. It figures out what it's looking for automatically and distributes what it thinks are important across these little networks. You can think of this as "feature detection") You then take a window on the image (400x400 image, maybe you've got a 20x20 window) which you then slide across the image and check each section for the presence of whatever the net is looking for (vertical/horizontal lines, etc.) There's several layers of this (specifically a convolution operation followed by a pooling operation) before the result is fed to a standard fully connected feed forward net which then outputs a prediction. While lower layers look for low level aspects of an image, each layer progressively looks for higher and higher level phenomena--for instance, when you hear about "the cat neuron" it comes from probing a neuron in the higher levels of a convolutional net and finding that pictures of cats happen to turn this neuron "on" while pictures of anything else don't. What is "high level" exactly? In this case, it really just means composed from "lower level" components... Some also say "higher level = more abstract" but I don't know this is quite the case--abstract means something different to me. An important consideration is that the conv net architecture allows you to cut down on the total number of parameters (good when you've got something high dimensional like image data) and it also takes advantage of the fact that you can make certain assumptions about images and how objects move/appear in space basically. Conv nets are probably the most difficult architecture to grasp and my explanation is extremely high level. Ha, that was a bit longer than I had hoped, but I hope it's helpful to someone...
  - derefr 13y ago
    > Conv nets are probably the most difficult architecture to grasp I might be totally off-base, but is Numenta's HTM (Hierarchical Temporal Memory) model a conv net? If so, there are some really good introductory/TED-talk-like explanations out there for the idea.
    - dave_sullivan 13y ago
      Heh, I still can't figure out what an HTM is exactly, but I think it's sort of similar to a recurrent convolutional network. It handles learning completely differently (allegedly in a more biologically plausible way). Personally, I think there are two big issues with what numenta was doing: they diverged too far from mainline neural net research (which, when they started in ~2005 was just before things started getting interesting--it was also a time when "mainline neural net research" was widely assumed to be at a dead end) and they tried too hard to come up with something that was biologically plausible rather than mathematically expedient. Sort of how airplanes need wings but don't need feathers. As far as I am aware, HTMs really just don't work well in practice (and by that, I mean they are not anywhere near competitive with any of the architectures I listed above).
      - derefr 13y ago
        From what I remember, I think HTM learning is isomorphic to that of a recurrent conv net. You'll get roughly the same change given the same training data, but with a parallel-agent approach to learning, instead of the kind of monolithic computation you can throw a GPU at. (In other words, you flip the "data" and "instruction" streams from SIMD to get Multiple-Instruction-Same-Data processes.) You could, in other words, see a recurrent conv net as an opmitization of an HTM given a von Neumann architecture, or the reverse -- an HTM as an optimization of a recurrent conv net given a biological substrate (where it's much less costly to build tons of crummy processors and link them into an arbitrary graph, than it is to build a single fast processor.) Again, though, I'm not an ML person, so I might be way off.
      - chmike 13y ago
        What do you mean by things getting interresting ? Did I miss something after 2005 ? Do you have some reference on the web ?
- bravura 13y ago
  Quick and dirty guide: - Use the tanh activation function. - Use many hidden nodes, e.g. 100 or 1000 for complex tasks. - Tune the learning rate. - Add a penalty of k*w^2 to each hidden node. Tune the parameter k to minimize the held-out error. - If this isn't working, plot the error on the training set as you train. It should decrease to 0. Read Yann LeCun's efficient backprop work for more tips.
- agibsonccc 13y ago
  I'm just going to suggest this as well. When working with neural networks, you can see the learning curve for them go down massively if you understand linear algebra. Numpy based neural networks are a lot more digestable as far as understanding all of the moving parts due to the easy syntax. Neural nets are usually verbose if you use an object oriented representation. Things like feedforward and backpropagation don't require loops if you know how to think about individual neurons and weight matrices. Also, for working with them, here's something that cleared up a lot for me: when initializing neuron weights, it's randomly initialized. It seemed like magic to me at first and kept tricking me up when getting started with them. For those who do the JVM, I'm working on finishing up a library for myself to use in various bits of work I do. Java based, but you get access to fortran matrix routines, and it's interopable with scala and the like as well. https://github.com/agibsonccc/sda-jblas https://github.com/agibsonccc/sda-jblas For those of you who need hadoop or want to scale out neural networks, I'm going to suggest a friend of mine's iterative reduce project for neural nets on hadoop. https://github.com/jpatanooga/Metronome https://github.com/jpatanooga/Metronome I just finished up a parallel training setup I'm going to release based on this for neural nets as well. When working with them, you'll discover crazy long training times. Multicore tends to make things easy if you know how to balance out the weights though. I'd be happy to answer any other questions as well.
funky_vodka 13y ago
That guy did another nice tutorial on a simple evolutionary algorithm and how to implement it to solve the travelling salesman problem. His explanations are useful for beginners like me.
amazedsaint 13y ago
Cool, here is another one I wrote few years back - http://www.amazedsaint.com/2008/01/neural-networks-part-i-simple.html http://www.amazedsaint.com/2008/01/neural-networks-part-i-si...
justplay 13y ago
Please please do not stop writing about this. I really want to know more. thanks.
- Anon84 13y ago
  You should check out the Coursera ML class videos.
howeman 13y ago
If anyone is interested, I have a working neural net implementation https://github.com/btracey/nnet https://github.com/btracey/nnet . I don't make any stability promises (especially on the training routines, that's a mess at the moment), but sometimes it's nice to see code. The core of the net is https://github.com/btracey/nnet/blob/master/nnet/functions.go https://github.com/btracey/nnet/blob/master/nnet/functions.g... and https://github.com/btracey/nnet/blob/master/nnet/nnet.go https://github.com/btracey/nnet/blob/master/nnet/nnet.go
- howeman 13y ago
  On the stability, the point is that if you'd like to use it you should fork it.
pilooch 13y ago
A very good source of information, and tutorials with source code is http://deeplearning.net/tutorial/contents.html http://deeplearning.net/tutorial/contents.html
- stillsut 13y ago
  Also, Coursera partnered with Geoffrey Hinton (the big name in Deep Learning right now) to put together a pretty good course: https://www.coursera.org/course/neuralnets https://www.coursera.org/course/neuralnets
gamegoblin 13y ago
I have been messing with ANNs since I learned how to program. For years, understanding of ANNs eluded me. Here is the breakthrough I had some time ago that finally let me understand ANNs: Think about them like circuits. It is easy to see that a circuit full of NAND, AND, OR, etc can carry out computations. An AND gate is a gate that requires all of its inputs to be true and then it goes high. An OR gate requires at least 1 of its inputs to be true and it goes high. Think of a neuron as being somewhere in between that (I understand that in the article he shows how to simulate gates, but doesn't quite describe neurons as being fancy gates (which is where my breakthrough happening)). It requires N of its inputs to be true and it goes high. Then imagine that the "wires" in our neural network don't carry information in the form of bits, but instead in the form of real numbers. Each wire has a "weight" associated with it, such that when the wire is turned on, it outputs that value, rather than a simple binary 1. Now imagine that the neurons take all of these real numbered inputs, and apply some function to them to decide if the neuron wants to turn on or not. It might simply sum them, multiply them, or something more complex, but based on its inputs, it turns on. Its on signal then gets sent to all of the neurons that it points to, etc. The same way you can extract answers from a circuit of logic gates by seeing the output of the gates, you can extract answers from an NN by examining the output of certain neurons. This description is quite simplified and doesn't go into the architecture of ANNs, but if you are really having a lot of trouble grasping how ANNs work, this description should give you some intuition. The hardcore ML people will probably dislike it, but you have to start somewhere. After understanding it like this, I branched out quite a lot and now all of my academic research involves machine learning. But it took that initial breakthrough!
- Dewie 13y ago
  Using neural networks to solve AND, OR etc. was how the book[1] we used in our course introduced and compared neural networks. The simplest NN were just logical circuits. [1] http://books.google.no/books/about/Fundamentals_of_Neural_Networks.html?id=ONylQgAACAAJ&redir_esc=y http://books.google.no/books/about/Fundamentals_of_Neural_Ne...
- dhammack 13y ago
  I've managed to get an intuition for backpropagation (the way gradients are computed for neural networks) using a similar analogy. The basic idea is that it's just a signal moving in the opposite direction in the network - it starts with computing the derivative of the loss, and goes back through each layer, like a breadth-first search.
foobarqux 13y ago
Anyone have information about understanding how inputs affect outputs in a trained ANN? Specifically to better understand the system, which inputs have the most impact and how they interact with out inputs.
- TrainedMonkey 13y ago
  I would check Neural Network class on Coursera. Subject is so complex that you need to put time to get anywhere.
  - chmike 13y ago
    What is so complex ? Neurons compute a sum of its input. Neurons are connected by synapse. A synapse has a oefficient that weights the input. If the synapse coefficitent is the real value c and the value of the input neuron is i, then The sum will be computed with c*i. One can represent this by vector and matrix multiplication. In the most simple model, this sum is the output. In more sofisticated models the result of the sum is modified by a function with a sigmoid shape. One of these sigmoid function is infinitly derivable. This allows to fit any continuous function with a sum of these sigmoids. Different strategies have been suggested to do the fit. But all these ANN don't explain the biological neural networks found in our cortex.
    - TrainedMonkey 13y ago
      It is complex as soon as you try to do anything but simplest things with Neural network. What I really liked about Coursera's NN class is theory of how RBM's can be used to model associative memory.
      - chmike 13y ago
        Ah yes. I agree. Doing things with these simple neurons is complex. On the other hand a 3 year old kid can sing and dance at the same time. How can it be so complex? ;) It is a fascinating problem which is still lacking an answer.
- 13y ago
TrainedMonkey 13y ago
I've been messing with Neural Networks for years. Including doing AI college project and taking class on them later. What I learned - unless you structure and query your neural networks in a very special way all they do is approximate higher order functions. It is not trivial to correctly structure neural network and input/outputs. TL:DR Neural networks are super powerful, but really hard to use properly for anything slightly harder than simple functions.
cjauvin 13y ago
I have written a small series of posts which show (with some simple Python code and math) the relationship between the training procedure of an ANN (backpropagation) and some simpler, more basic machine learning algorithms (like logistic regression): http://cjauvin.blogspot.ca/2013/10/neural-network-101.html http://cjauvin.blogspot.ca/2013/10/neural-network-101.html
rdlecler1 13y ago
It becomes a lot easier to think about these circuits if you realize that many of the connections are actually spurious. That is, if you remove them and systematically test the function over a range you'll realize that many of these weights have no information bearing value. For instance, if you have a w_i_j=0.01 this may be totally insignificant for any sigmoidal function and for all intents and purposes should be w_i_j=0 in which case this complex web of neural connections is trimmed, resulting in the underlying circuitry. If you're interested I wrote a paper in Nature Systems Biology that shows this for artificial gene networks (just another branch of ANNs): Survival of the Sparsest: Robust Gene Networks are Parsimonious
rdlecler1 13y ago
It becomes a lot easier to think about these circuits if you realize that many of the connections are actually spurious. That is, if you remove them and systematically test the function over a range you'll realize that many of these weights have no information bearing value. For instance, if you have a w_i_j=0.01 this may be totally insignificant for any sigmoidal function and for all intents and purposes should be w_i_j=0 in which case this complex web of neural connections is trimmed, resulting in the underlying circuitry. If you're interested I wrote a paper in Nature Systems Biology that shows this for artificial gene networks (just another branch of ANNs): Survival of the Sparsest: Robust Gene Networks are Parsimonious
11001 13y ago
People! If you're interested in learning Neural Networks, please do yourself a favour, go learn them from a good textbook (e.g. http://www.cs.toronto.edu/~mackay/itprnn/book.html http://www.cs.toronto.edu/~mackay/itprnn/book.html is free online), not these "online tutorials". I know, math-y books don't often make it sexy "for hackers", but it is really the only way to learn a math subject.

168 points by bananacurve 13y ago

Does anyone here have any good guides/articles about the structure of the artificial network? At university I solved simple problems in a course with ANs, but the structure of my network (hidden layers, how they were connected etc.) were basically just try/fail until I achieved ok results.

howeman 13y ago
I think that's basically it. If you want to do a lot of work you can do somethings with cross validation (see which parameters perform the best), but that's a form of automatic try/fail
- pilooch 13y ago
  hyperparameter search is a solution. See http://jmlr.org/papers/v13/bergstra12a.html http://jmlr.org/papers/v13/bergstra12a.html and http://jaberg.github.io/hyperopt/ http://jaberg.github.io/hyperopt/
dave_sullivan 13y ago
There aren't any I'm aware of that tie this question together nicely, but here are the general structures in use right now. Google will lead you to more information on each. Standard feed forward deep net: like the ones you used at university but with a few important features. One, you can stack layers on top of each other then train the whole network with back propagation. The nonlinearity you use can be important (rectified nonlinear units--Max(0,x) are popular now). Regularization can be important (with dropout being a popular method). Pre-processing your data can also be important (eg scaling the inputs, subtracting the mean, ...). How you initialize your weights is important. Using tricks like momentum and learning rate decay are important too. Autoencoder: A standard feed forward deep net that tries to output an "uncorrupted" sample of a "corrupted" input. So basically, take an image, add noise, ask it to output the denoised image. Why you ask? Well, in doing this, the hidden units (the layers in between the input and output) tend to discover new features which can then be used elsewhere in machine learning pipelines. The big win here is automated feature engineering. RBM: Restricted boltzman machine, sort of like an autoencoder but not. I'm not convinced one is better than the other, but there are definitely differences. Recurrent neural networks: Like standard feed forward deep nets but extended to time series problems. The hidden units at each timestep feed in as additional input to the next timestep (along with the new input). So basically, each step it gets this as input: [whatever came before described as hidden units] + [input at T]. These are my personal favorite and currently hold state of the art in speech recognition, at least academically. Convolutional neural networks: State of the art in vision, this one is kind of hard to explain quickly, but you can think of it kind of like this: I take some image where I want to recognize objects in it. Then I train a bunch of neural nets that represent different aspects of an image (one net might look for vertical lines, another might look for horizontal. It figures out what it's looking for automatically and distributes what it thinks are important across these little networks. You can think of this as "feature detection") You then take a window on the image (400x400 image, maybe you've got a 20x20 window) which you then slide across the image and check each section for the presence of whatever the net is looking for (vertical/horizontal lines, etc.) There's several layers of this (specifically a convolution operation followed by a pooling operation) before the result is fed to a standard fully connected feed forward net which then outputs a prediction. While lower layers look for low level aspects of an image, each layer progressively looks for higher and higher level phenomena--for instance, when you hear about "the cat neuron" it comes from probing a neuron in the higher levels of a convolutional net and finding that pictures of cats happen to turn this neuron "on" while pictures of anything else don't. What is "high level" exactly? In this case, it really just means composed from "lower level" components... Some also say "higher level = more abstract" but I don't know this is quite the case--abstract means something different to me. An important consideration is that the conv net architecture allows you to cut down on the total number of parameters (good when you've got something high dimensional like image data) and it also takes advantage of the fact that you can make certain assumptions about images and how objects move/appear in space basically. Conv nets are probably the most difficult architecture to grasp and my explanation is extremely high level. Ha, that was a bit longer than I had hoped, but I hope it's helpful to someone...
- derefr 13y ago
  > Conv nets are probably the most difficult architecture to grasp I might be totally off-base, but is Numenta's HTM (Hierarchical Temporal Memory) model a conv net? If so, there are some really good introductory/TED-talk-like explanations out there for the idea.
  - dave_sullivan 13y ago
    Heh, I still can't figure out what an HTM is exactly, but I think it's sort of similar to a recurrent convolutional network. It handles learning completely differently (allegedly in a more biologically plausible way). Personally, I think there are two big issues with what numenta was doing: they diverged too far from mainline neural net research (which, when they started in ~2005 was just before things started getting interesting--it was also a time when "mainline neural net research" was widely assumed to be at a dead end) and they tried too hard to come up with something that was biologically plausible rather than mathematically expedient. Sort of how airplanes need wings but don't need feathers. As far as I am aware, HTMs really just don't work well in practice (and by that, I mean they are not anywhere near competitive with any of the architectures I listed above).
    - derefr 13y ago
      From what I remember, I think HTM learning is isomorphic to that of a recurrent conv net. You'll get roughly the same change given the same training data, but with a parallel-agent approach to learning, instead of the kind of monolithic computation you can throw a GPU at. (In other words, you flip the "data" and "instruction" streams from SIMD to get Multiple-Instruction-Same-Data processes.) You could, in other words, see a recurrent conv net as an opmitization of an HTM given a von Neumann architecture, or the reverse -- an HTM as an optimization of a recurrent conv net given a biological substrate (where it's much less costly to build tons of crummy processors and link them into an arbitrary graph, than it is to build a single fast processor.) Again, though, I'm not an ML person, so I might be way off.
    - chmike 13y ago
      What do you mean by things getting interresting ? Did I miss something after 2005 ? Do you have some reference on the web ?
bravura 13y ago
Quick and dirty guide: - Use the tanh activation function. - Use many hidden nodes, e.g. 100 or 1000 for complex tasks. - Tune the learning rate. - Add a penalty of k*w^2 to each hidden node. Tune the parameter k to minimize the held-out error. - If this isn't working, plot the error on the training set as you train. It should decrease to 0. Read Yann LeCun's efficient backprop work for more tips.
agibsonccc 13y ago
I'm just going to suggest this as well. When working with neural networks, you can see the learning curve for them go down massively if you understand linear algebra. Numpy based neural networks are a lot more digestable as far as understanding all of the moving parts due to the easy syntax. Neural nets are usually verbose if you use an object oriented representation. Things like feedforward and backpropagation don't require loops if you know how to think about individual neurons and weight matrices. Also, for working with them, here's something that cleared up a lot for me: when initializing neuron weights, it's randomly initialized. It seemed like magic to me at first and kept tricking me up when getting started with them. For those who do the JVM, I'm working on finishing up a library for myself to use in various bits of work I do. Java based, but you get access to fortran matrix routines, and it's interopable with scala and the like as well. https://github.com/agibsonccc/sda-jblas https://github.com/agibsonccc/sda-jblas For those of you who need hadoop or want to scale out neural networks, I'm going to suggest a friend of mine's iterative reduce project for neural nets on hadoop. https://github.com/jpatanooga/Metronome https://github.com/jpatanooga/Metronome I just finished up a parallel training setup I'm going to release based on this for neural nets as well. When working with them, you'll discover crazy long training times. Multicore tends to make things easy if you know how to balance out the weights though. I'd be happy to answer any other questions as well.

funky_vodka 13y ago

That guy did another nice tutorial on a simple evolutionary algorithm and how to implement it to solve the travelling salesman problem. His explanations are useful for beginners like me.

amazedsaint 13y ago

Cool, here is another one I wrote few years back - http://www.amazedsaint.com/2008/01/neural-networks-part-i-simple.html http://www.amazedsaint.com/2008/01/neural-networks-part-i-si...

justplay 13y ago

Please please do not stop writing about this. I really want to know more. thanks.

Anon84 13y ago
You should check out the Coursera ML class videos.

howeman 13y ago

If anyone is interested, I have a working neural net implementation https://github.com/btracey/nnet https://github.com/btracey/nnet . I don't make any stability promises (especially on the training routines, that's a mess at the moment), but sometimes it's nice to see code. The core of the net is https://github.com/btracey/nnet/blob/master/nnet/functions.go https://github.com/btracey/nnet/blob/master/nnet/functions.g... and https://github.com/btracey/nnet/blob/master/nnet/nnet.go https://github.com/btracey/nnet/blob/master/nnet/nnet.go

howeman 13y ago
On the stability, the point is that if you'd like to use it you should fork it.

pilooch 13y ago

A very good source of information, and tutorials with source code is http://deeplearning.net/tutorial/contents.html http://deeplearning.net/tutorial/contents.html

stillsut 13y ago
Also, Coursera partnered with Geoffrey Hinton (the big name in Deep Learning right now) to put together a pretty good course: https://www.coursera.org/course/neuralnets https://www.coursera.org/course/neuralnets

gamegoblin 13y ago

I have been messing with ANNs since I learned how to program. For years, understanding of ANNs eluded me. Here is the breakthrough I had some time ago that finally let me understand ANNs: Think about them like circuits. It is easy to see that a circuit full of NAND, AND, OR, etc can carry out computations. An AND gate is a gate that requires all of its inputs to be true and then it goes high. An OR gate requires at least 1 of its inputs to be true and it goes high. Think of a neuron as being somewhere in between that (I understand that in the article he shows how to simulate gates, but doesn't quite describe neurons as being fancy gates (which is where my breakthrough happening)). It requires N of its inputs to be true and it goes high. Then imagine that the "wires" in our neural network don't carry information in the form of bits, but instead in the form of real numbers. Each wire has a "weight" associated with it, such that when the wire is turned on, it outputs that value, rather than a simple binary 1. Now imagine that the neurons take all of these real numbered inputs, and apply some function to them to decide if the neuron wants to turn on or not. It might simply sum them, multiply them, or something more complex, but based on its inputs, it turns on. Its on signal then gets sent to all of the neurons that it points to, etc. The same way you can extract answers from a circuit of logic gates by seeing the output of the gates, you can extract answers from an NN by examining the output of certain neurons. This description is quite simplified and doesn't go into the architecture of ANNs, but if you are really having a lot of trouble grasping how ANNs work, this description should give you some intuition. The hardcore ML people will probably dislike it, but you have to start somewhere. After understanding it like this, I branched out quite a lot and now all of my academic research involves machine learning. But it took that initial breakthrough!

Dewie 13y ago
Using neural networks to solve AND, OR etc. was how the book[1] we used in our course introduced and compared neural networks. The simplest NN were just logical circuits. [1] http://books.google.no/books/about/Fundamentals_of_Neural_Networks.html?id=ONylQgAACAAJ&redir_esc=y http://books.google.no/books/about/Fundamentals_of_Neural_Ne...
dhammack 13y ago
I've managed to get an intuition for backpropagation (the way gradients are computed for neural networks) using a similar analogy. The basic idea is that it's just a signal moving in the opposite direction in the network - it starts with computing the derivative of the loss, and goes back through each layer, like a breadth-first search.

foobarqux 13y ago

Anyone have information about understanding how inputs affect outputs in a trained ANN? Specifically to better understand the system, which inputs have the most impact and how they interact with out inputs.

TrainedMonkey 13y ago
I would check Neural Network class on Coursera. Subject is so complex that you need to put time to get anywhere.
- chmike 13y ago
  What is so complex ? Neurons compute a sum of its input. Neurons are connected by synapse. A synapse has a oefficient that weights the input. If the synapse coefficitent is the real value c and the value of the input neuron is i, then The sum will be computed with c*i. One can represent this by vector and matrix multiplication. In the most simple model, this sum is the output. In more sofisticated models the result of the sum is modified by a function with a sigmoid shape. One of these sigmoid function is infinitly derivable. This allows to fit any continuous function with a sum of these sigmoids. Different strategies have been suggested to do the fit. But all these ANN don't explain the biological neural networks found in our cortex.
  - TrainedMonkey 13y ago
    It is complex as soon as you try to do anything but simplest things with Neural network. What I really liked about Coursera's NN class is theory of how RBM's can be used to model associative memory.
    - chmike 13y ago
      Ah yes. I agree. Doing things with these simple neurons is complex. On the other hand a 3 year old kid can sing and dance at the same time. How can it be so complex? ;) It is a fascinating problem which is still lacking an answer.
13y ago

TrainedMonkey 13y ago

I've been messing with Neural Networks for years. Including doing AI college project and taking class on them later. What I learned - unless you structure and query your neural networks in a very special way all they do is approximate higher order functions. It is not trivial to correctly structure neural network and input/outputs. TL:DR Neural networks are super powerful, but really hard to use properly for anything slightly harder than simple functions.

cjauvin 13y ago

I have written a small series of posts which show (with some simple Python code and math) the relationship between the training procedure of an ANN (backpropagation) and some simpler, more basic machine learning algorithms (like logistic regression): http://cjauvin.blogspot.ca/2013/10/neural-network-101.html http://cjauvin.blogspot.ca/2013/10/neural-network-101.html

rdlecler1 13y ago

It becomes a lot easier to think about these circuits if you realize that many of the connections are actually spurious. That is, if you remove them and systematically test the function over a range you'll realize that many of these weights have no information bearing value. For instance, if you have a w_i_j=0.01 this may be totally insignificant for any sigmoidal function and for all intents and purposes should be w_i_j=0 in which case this complex web of neural connections is trimmed, resulting in the underlying circuitry. If you're interested I wrote a paper in Nature Systems Biology that shows this for artificial gene networks (just another branch of ANNs): Survival of the Sparsest: Robust Gene Networks are Parsimonious

rdlecler1 13y ago

11001 13y ago

People! If you're interested in learning Neural Networks, please do yourself a favour, go learn them from a good textbook (e.g. http://www.cs.toronto.edu/~mackay/itprnn/book.html http://www.cs.toronto.edu/~mackay/itprnn/book.html is free online), not these "online tutorials". I know, math-y books don't often make it sexy "for hackers", but it is really the only way to learn a math subject.