4 ms·

It just seems so random. I understand that CNNs are better suited to process pixels in a 2D grid, it sort of makes sense b/c images are in a 2D grid and the ope

It just seems so random. I understand that CNNs are better suited to process pixels in a 2D grid, it sort of makes sense b/c images are in a 2D grid and the operations emulate convolutions used in lots of image processing. I also understand that RNNs used to process symbols, and their feedback mechanisms can emulate short-term memory. Still... even though I understand what works and what doesn't, I don't really understand why RNNs are better at processing symbols and CNNs are better at processing images. They hold the same information, and work the same way, they are just organized differently. It makes little sense.

candiodari 9y ago
Think of it this way: Neural networks are a particular kind of formula. For instance a 2x2 neural network takes 2 inputs, x, and y, has 2 biases, bx and by, and 4 weights w1 ... w4 and calculates 2 outputs ox, and oy. The formula it uses is: ox = w1 * x + w3 * y + bx oy = w2 * x + w4 * y + by And then learns how to change w1 ... w4, bx and by to get ox and oy to be more useful (less error). Is it so weird that a 2 layer network (which is the same formula, but it uses ox and oy as inputs to a second identical calculation to produce the final outputs) will produce better results, even with learning ? Even in a neural network so absurdly large as the human mind, it isn't just a bunch of wires connecting everything to everything. There are clear patterns, clear formulas that are dictated not by learning, but by the architecture itself (e.g. CNNs in the eyes and optical cortex, and something much more like LSTMs in memory heavy regions). In some sense one might even say it doesn't make a difference. A fully connected network with the same fanout as a CNN can do everything a CNN can, and more. Likewise, a network that is simply presented with the last 50 timesteps can do strictly better than an LSTM or GRU RNN would. But such a network would have much, much greater computational complexity (by a factor in the thousands at least, in the case of those CNNs by a factor that is roughly the number of pixels in the image). So they don't work better per se. In fact they are "worse" by the most important metric (error). But they are a good trade off : LSTMs are thousands of times faster, with only slightly worse results for sequence labeling or production (ie. text and audio comprehension). CNNs are millions of times faster at image comprehension than a fully connected network, and are only slightly worse at it.
- speedplane 9y ago
  I understand all your points, but they are just not satisfying. CNNs and RNNs are sufficiently different, I sort of see understand. Maybe CNNs are more like your eyeball and RNNs are more like your inner brain. But LSTMs... really? There are so many variations of them and they seem to be such trial and error hooking nodes up in weird ways. I'm not saying they don't work, and I'm comfortable not know why particular weights do what they do, but I it's hard to get understand why some configurations work so much better than others for certain problems.
- ChrisFoster 9y ago
  These are good points about computational complexity, but there's more to it: the NN architecture regularizes the representations which may be learned. A fully connected architecture may be fully general, but without powerful regularization you won't be able to train it to get the performance of an architecture which encodes some of the structure of the problem. Even disregarding the computational load of the increased number of parameters.
rjtavares 9y ago
The way I took it is that CNNs use local context and are translation neutral (so if a certain combination of pixels is an eye, they'll be an eye anywhere in the image), while RNNs can decide which information to keep and can decide in what way positioning affects result. So the thing CNNs are really good at are perfect for small images, but symbols already have that thing done. When CNNs and RNNs are used together (e.g. image segmentation), it's almost as if the CNN is creating symbols and RNN s processing them. Or, you know, maybe I'm far off. I'm in no way qualified to talk about this (just a fan).
- sdenton4 9y ago
  Here's another thing to consider. The brain is processing "video" in what we might call 3.5 dimensions. (2d images + time with around 6 channels... Plus sound). Image recognition with a CNN is working on still images, and dropping the time component entirely. RNNs are about handling sequential data, discrete or not... Meanwhile, the hardware in our heads is far more complex and capable than the models we're building. And we're not even sure how that hardware works, or what principles we can abstract out and simplify. A neuron getting electrical and hormonal signals is far more complex than an lstm cell... Does it need to be? Or are there biological requirements built in which we don't understand? And how many evolutionary accidents which never evolved away?
visarga 9y ago
> I don't really understand why RNNs are better at processing symbols and CNNs are better at processing images It's also possible to go the other way around. PixelRNN uses a kind of special convolutions to encode and generate images. And CNN's have been used for text translation and speech synthesis, especially because they are faster. The RNN and CNN contain prior knowledge about the domain. The RNN contains the idea that prior elements influence the following elements in a sequence. CNN's encode the idea that close pixels are related. Both imply some kind of domain knowledge. Multiple attention heads could potentially learn domain knowledge from data.

candiodari 9y ago

Think of it this way: Neural networks are a particular kind of formula. For instance a 2x2 neural network takes 2 inputs, x, and y, has 2 biases, bx and by, and 4 weights w1 ... w4 and calculates 2 outputs ox, and oy. The formula it uses is: ox = w1 * x + w3 * y + bx oy = w2 * x + w4 * y + by And then learns how to change w1 ... w4, bx and by to get ox and oy to be more useful (less error). Is it so weird that a 2 layer network (which is the same formula, but it uses ox and oy as inputs to a second identical calculation to produce the final outputs) will produce better results, even with learning ? Even in a neural network so absurdly large as the human mind, it isn't just a bunch of wires connecting everything to everything. There are clear patterns, clear formulas that are dictated not by learning, but by the architecture itself (e.g. CNNs in the eyes and optical cortex, and something much more like LSTMs in memory heavy regions). In some sense one might even say it doesn't make a difference. A fully connected network with the same fanout as a CNN can do everything a CNN can, and more. Likewise, a network that is simply presented with the last 50 timesteps can do strictly better than an LSTM or GRU RNN would. But such a network would have much, much greater computational complexity (by a factor in the thousands at least, in the case of those CNNs by a factor that is roughly the number of pixels in the image). So they don't work better per se. In fact they are "worse" by the most important metric (error). But they are a good trade off : LSTMs are thousands of times faster, with only slightly worse results for sequence labeling or production (ie. text and audio comprehension). CNNs are millions of times faster at image comprehension than a fully connected network, and are only slightly worse at it.

speedplane 9y ago
I understand all your points, but they are just not satisfying. CNNs and RNNs are sufficiently different, I sort of see understand. Maybe CNNs are more like your eyeball and RNNs are more like your inner brain. But LSTMs... really? There are so many variations of them and they seem to be such trial and error hooking nodes up in weird ways. I'm not saying they don't work, and I'm comfortable not know why particular weights do what they do, but I it's hard to get understand why some configurations work so much better than others for certain problems.
ChrisFoster 9y ago
These are good points about computational complexity, but there's more to it: the NN architecture regularizes the representations which may be learned. A fully connected architecture may be fully general, but without powerful regularization you won't be able to train it to get the performance of an architecture which encodes some of the structure of the problem. Even disregarding the computational load of the increased number of parameters.

rjtavares 9y ago

The way I took it is that CNNs use local context and are translation neutral (so if a certain combination of pixels is an eye, they'll be an eye anywhere in the image), while RNNs can decide which information to keep and can decide in what way positioning affects result. So the thing CNNs are really good at are perfect for small images, but symbols already have that thing done. When CNNs and RNNs are used together (e.g. image segmentation), it's almost as if the CNN is creating symbols and RNN s processing them. Or, you know, maybe I'm far off. I'm in no way qualified to talk about this (just a fan).

sdenton4 9y ago
Here's another thing to consider. The brain is processing "video" in what we might call 3.5 dimensions. (2d images + time with around 6 channels... Plus sound). Image recognition with a CNN is working on still images, and dropping the time component entirely. RNNs are about handling sequential data, discrete or not... Meanwhile, the hardware in our heads is far more complex and capable than the models we're building. And we're not even sure how that hardware works, or what principles we can abstract out and simplify. A neuron getting electrical and hormonal signals is far more complex than an lstm cell... Does it need to be? Or are there biological requirements built in which we don't understand? And how many evolutionary accidents which never evolved away?

visarga 9y ago

> I don't really understand why RNNs are better at processing symbols and CNNs are better at processing images It's also possible to go the other way around. PixelRNN uses a kind of special convolutions to encode and generate images. And CNN's have been used for text translation and speech synthesis, especially because they are faster. The RNN and CNN contain prior knowledge about the domain. The RNN contains the idea that prior elements influence the following elements in a sequence. CNN's encode the idea that close pixels are related. Both imply some kind of domain knowledge. Multiple attention heads could potentially learn domain knowledge from data.