4 ms·

(My take on this problem, feel free to correct any assumptions I make that are wrong) I think this is because neural nets are not actually "intelligent" in ter

(My take on this problem, feel free to correct any assumptions I make that are wrong) I think this is because neural nets are not actually "intelligent" in terms of the commonly accepted definition of intelligence. They are dumb. They try, usually with simple probabilistic techniques and input element-wise transforms, to mimic some function that produces approximations for a given set of inputs and outputs. The training data and the test data will always have underlying differences which will create gaps in the data generating distribution, assuming a reasonably large-sized set of data. This is contrary to the assumptions that (every?) machine learning algorithm makes, usually referred to as the i.i.d assumptions. The test and training data are assumed to be independent and identically distributed. Since practically no real world data sets are perfectly identically distributed, there will always be gaps in the learned model and the real model. Beyond that, the set of training data can never fully encompass the entire domain of possible input/outputs, else what need would we have for a machine to predict new ones? The oddities that researchers find are actually problems in the relationship of their training data to their testing data because the i.i.d assumptions are never actually true. We can only try to get as close as possible. The solution to this problem is just nigh impossible, so we try to reduce it as much as possible.

hiddencost 11y ago
That's pretty good. I wouldn't say things are impossible, but there're a lot of conversations happening about the problems with our current approach. Leon Bottou gave a great talk at ICML that covers stuff like this. https://www.google.com/url?sa=t&source=web&rct=j&url=http://icml.cc/2015/invited/LeonBottouICML2015.pdf&ved=0CBwQFjAAahUKEwjTpe3_tZXHAhVGEpIKHSOkB84&usg=AFQjCNEqCYMpVjZQs9-NXseTdgg2mCKDFA&sig2=tsR6pnvDP8GnJnFcoGMbQQ https://www.google.com/url?sa=t&source=web&rct=j&url=http://...
- crazypyro 11y ago
  Thanks for that slidedeck! It was really informative and enlightening to have some of these vague, hard-to-explain ideas that I've been wondering about cemented by Bottou.
noskynethere 11y ago
> They try, usually with simple probabilistic techniques and input element-wise transforms, to mimic some function that produces approximations for a given set of inputs and outputs It's my understanding that this is basically how the brain works. My personal theory is that enough of these "dumb" inputs, wired correctly together, leads to emergent behavior that is consciousness.
- crazypyro 11y ago
  I imagine the brain more like hundreds (thousands, millions, I'm not sure the magnitude) of different specialized neural networks. So you have a specific neural network for picking out colors and that feeds (along with a bunch of other inputs) into the neural network for picking out object boundaries and that feeds into the neural network for object recognition and so on. In comparison, most neural networks that are used in computer vision are generally trying to do the entire process in a single network (although they also use feedforward, so the difference is more complex than just composing the various layers). I think there is something to the idea that we need the neural network to have points where it can spit out a partial piece of the eventual goal model, things like object boundaries before recognizing the object, recognizing eyes before the entire face, etc. The key is being able to get those logical partial model results at various layers of the network.
  - noskynethere 11y ago
    I'm outside my depth here, but isn't that what hierarchical learning is? (I think it's popularly called "deep learning", which I assume means the neural nets have depth?) From what I've read, we aren't going more than a few dozens of levels deep. But it also sounds like this technique is very successful in image recognition. Am I incorrect in my understanding?
- deleted 11y ago
  [deleted]

hiddencost 11y ago

That's pretty good. I wouldn't say things are impossible, but there're a lot of conversations happening about the problems with our current approach. Leon Bottou gave a great talk at ICML that covers stuff like this. https://www.google.com/url?sa=t&source=web&rct=j&url=http://icml.cc/2015/invited/LeonBottouICML2015.pdf&ved=0CBwQFjAAahUKEwjTpe3_tZXHAhVGEpIKHSOkB84&usg=AFQjCNEqCYMpVjZQs9-NXseTdgg2mCKDFA&sig2=tsR6pnvDP8GnJnFcoGMbQQ https://www.google.com/url?sa=t&source=web&rct=j&url=http://...

crazypyro 11y ago
Thanks for that slidedeck! It was really informative and enlightening to have some of these vague, hard-to-explain ideas that I've been wondering about cemented by Bottou.

noskynethere 11y ago

> They try, usually with simple probabilistic techniques and input element-wise transforms, to mimic some function that produces approximations for a given set of inputs and outputs It's my understanding that this is basically how the brain works. My personal theory is that enough of these "dumb" inputs, wired correctly together, leads to emergent behavior that is consciousness.

crazypyro 11y ago
I imagine the brain more like hundreds (thousands, millions, I'm not sure the magnitude) of different specialized neural networks. So you have a specific neural network for picking out colors and that feeds (along with a bunch of other inputs) into the neural network for picking out object boundaries and that feeds into the neural network for object recognition and so on. In comparison, most neural networks that are used in computer vision are generally trying to do the entire process in a single network (although they also use feedforward, so the difference is more complex than just composing the various layers). I think there is something to the idea that we need the neural network to have points where it can spit out a partial piece of the eventual goal model, things like object boundaries before recognizing the object, recognizing eyes before the entire face, etc. The key is being able to get those logical partial model results at various layers of the network.
- noskynethere 11y ago
  I'm outside my depth here, but isn't that what hierarchical learning is? (I think it's popularly called "deep learning", which I assume means the neural nets have depth?) From what I've read, we aren't going more than a few dozens of levels deep. But it also sounds like this technique is very successful in image recognition. Am I incorrect in my understanding?
deleted 11y ago
[deleted]