3 ms·
I have been messing with ANNs since I learned how to program. For years, understanding of ANNs eluded me. Here is the breakthrough I had some time ago that fina
by gamegoblin 13y ago
I have been messing with ANNs since I learned how to program. For years, understanding of ANNs eluded me. Here is the breakthrough I had some time ago that finally let me understand ANNs:
Think about them like circuits. It is easy to see that a circuit full of NAND, AND, OR, etc can carry out computations. An AND gate is a gate that requires all of its inputs to be true and then it goes high. An OR gate requires at least 1 of its inputs to be true and it goes high.
Think of a neuron as being somewhere in between that (I understand that in the article he shows how to simulate gates, but doesn't quite describe neurons as being fancy gates (which is where my breakthrough happening)). It requires N of its inputs to be true and it goes high. Then imagine that the "wires" in our neural network don't carry information in the form of bits, but instead in the form of real numbers. Each wire has a "weight" associated with it, such that when the wire is turned on, it outputs that value, rather than a simple binary 1.
Now imagine that the neurons take all of these real numbered inputs, and apply some function to them to decide if the neuron wants to turn on or not. It might simply sum them, multiply them, or something more complex, but based on its inputs, it turns on. Its on signal then gets sent to all of the neurons that it points to, etc. The same way you can extract answers from a circuit of logic gates by seeing the output of the gates, you can extract answers from an NN by examining the output of certain neurons.
This description is quite simplified and doesn't go into the architecture of ANNs, but if you are really having a lot of trouble grasping how ANNs work, this description should give you some intuition. The hardcore ML people will probably dislike it, but you have to start somewhere. After understanding it like this, I branched out quite a lot and now all of my academic research involves machine learning. But it took that initial breakthrough!
- Dewie 13y agoUsing neural networks to solve AND, OR etc. was how the book[1] we used in our course introduced and compared neural networks. The simplest NN were just logical circuits. [1] http://books.google.no/books/about/Fundamentals_of_Neural_Networks.html?id=ONylQgAACAAJ&redir_esc=y http://books.google.no/books/about/Fundamentals_of_Neural_Ne...
- dhammack 13y agoI've managed to get an intuition for backpropagation (the way gradients are computed for neural networks) using a similar analogy. The basic idea is that it's just a signal moving in the opposite direction in the network - it starts with computing the derivative of the loss, and goes back through each layer, like a breadth-first search.
- agibsonccc 13y agoWhat are your thoughts on the vectorized represenation? I understood the object oriented representation after a while, but I've noticed that, despite what looks like magic at first, when there's a lot less code to look at with singular weight vectors for the whole network vs different kinds of neurons, setting up graph structures, etc.it's easier to grok. This is probably just a personal preference though. I find if you're tackling neural networks in the first place, it's great to have your fundamentals down in the first place though. I think a lot of the problem with people learning machine learning is that they just dive in to it without going back and understanding the moving parts that make up the whole. I made that mistake personally when I was starting out a few years ago. I've seen that with others as well.
- dhammack 13y agoI like to think of the vectorized representation as just a nonlinear transformation to a higher dimensional space with a classifier afterwords. If you're familiar with linear algebra, then z = Wx, where W is a matrix of weights and x is a feature vector maps x (which could be something like 5 dimensional) to a new space (which could be like 50 dimensional). z is the representation of x in that new space. After this linear mapping, we apply a nonlinear transform (sigmoid, rectifier, etc). If we didn't have the nonlinear transform, then the entire model would just be linear! This follows from the fact that the composition of linear functions is itself linear. The final layer is just a standard logistic regression classifier in the new (usually higher dimensional) space.
- agibsonccc 13y agoHaha I get that! I was just saying for someone learning it at first vs approaching it from an object oriented angle with individual neurons and having it be a graph data structure. It seems easier to tackle when you can summarize it exactly as you said there vs, oh there's these neurons with these connections and you forward propagate each individual weight vector then backpropagate etc...
- sitkack 13y agoFeynman would be proud of your explanation. Douglas Hofstadter, "Analogy as the Core of Cognition" http://www.youtube.com/watch?v=n8m7lFQ3njk http://www.youtube.com/watch?v=n8m7lFQ3njk
- pmelendez 13y agoActually that's the traditional analogy, and if I recall correctly in the process of thinking it as a circuit was how Marvin Minsky realized that the original perceptron is unable to mimic a XOR gate.
- gamegoblin 13y agoReally? In all of my perusing of various tutorials online, I never encountered it explained in this intuitive fashion, that neurons are just a generalization of logic gates.
- jonsen 13y agoThreshold Logic - a kind of in-between.
- deleted 13y ago[deleted]