3 ms·

I think author is stretching arguments here a bit - DL is just partitioning space according to some pre-baked associations given to it during training; in this

by bitL 9y ago

I think author is stretching arguments here a bit - DL is just partitioning space according to some pre-baked associations given to it during training; in this case it's more like a non-linear optimization where we want to end up with N-million dimensional objects of certain shape obtained by optimizing some objective function allowing predicting similar associations. It doesn't have much with the actual innate quality of understanding. Maybe reinforcement learning with deep learning together (DRL) can move us towards such a quality at least in a mechanical sense.

petters 9y ago
On one hand, I sort of agree with you. On the other hand, what you are saying feels a little bit like saying that humans aren't impressive, because we are just atoms. Sometimes interesting things arise from many small, simple parts.
jacquesm 9y ago
That's got to be one of the most concise but still complete descriptions of deep learning that I've seen so far. The question that it implicitly raises (at least, with me) is how can we tell the difference between 'understanding' and 'deep learning' if the end results are the same? To me 'reasoning' is a slow, conscious process, and understanding is a part of that. But classification problems , especially when done by humans when they try to work fast have no room for such conscious decision making, we go much faster than that and outsource the job to our subconscious. Predictably, the error rate goes up and in those kind of situations deep learning can today already outperform humans on the same tasks. The weird thing is that deep learning solutions can get simple cases completely wrong, where a human would never err, and yet get some of the hardest cases - where a human would be very likely to make an error - right. It's baffling.
akyu 9y ago
>It doesn't have much with the actual innate quality of understanding That was his entire point.
westoncb 9y ago
> DL is just partitioning space according to some pre-baked associations given to it during training What I wonder is whether that's not also maybe the cornerstone of human understanding. If I understand correctly, you are essentially saying that DL is forming categories, or developing a classification scheme. Granted, if we're only talking about supervised DL, and the program is practically told where to form the boundaries—then it's not very impressive. But if the software is extracting statistically prominent commonalities and using those to form category boundaries, and arranging them hierarchically—then while the implementation may be totally different from human understanding, the effect seems to strongly overlap. I assume I'm probably just missing something—anyone know what it is? (It seems clear that at least part of the problem here is that 'human understanding' has been left far more vague than DL, and in order to say one way or the other how much they have in common, we need to better define 'human understanding'.)
- bitL 9y ago
  I think what DL is capable of is already very impressive; sometimes while playing with DL in NLP I am wondering if our languages aren't way simpler than we think/hope and we aren't that intelligent either. There is one massive difference between humans and DL; humans can mimic something just from a single (even partial) observation; DL requires huge amounts of data and massive parallel processing, something that we got only recently with Big Data and GPUs. DL also suffers from the curse of dimensionality; in theory deep fully connected networks should be able to do everything better, in practice they are awful and only cleverly constructed schemes like CNNs, LSTMs etc. that assume data to be in certain format/domain bring impressive results when paired with optimizers/metrics that magically work on a given dataset. If you are able to construct a DNN that can figure out PCA/ICA/eigenvalues/etc. on its own during its training like it does with convolutions, that would again enable another set of magic tricks. In any case, humans still have to figure out the architecture of the network that works (even if we now have AutoML for figuring out best hyperparameters in parallel). Then the hard problem of consciousness; I personally believe we are far off and probably miss something very important in our understanding of Universe.
  - RivieraKid 9y ago

by bitL 9y ago

petters 9y ago

On one hand, I sort of agree with you. On the other hand, what you are saying feels a little bit like saying that humans aren't impressive, because we are just atoms. Sometimes interesting things arise from many small, simple parts.

jacquesm 9y ago

That's got to be one of the most concise but still complete descriptions of deep learning that I've seen so far. The question that it implicitly raises (at least, with me) is how can we tell the difference between 'understanding' and 'deep learning' if the end results are the same? To me 'reasoning' is a slow, conscious process, and understanding is a part of that. But classification problems , especially when done by humans when they try to work fast have no room for such conscious decision making, we go much faster than that and outsource the job to our subconscious. Predictably, the error rate goes up and in those kind of situations deep learning can today already outperform humans on the same tasks. The weird thing is that deep learning solutions can get simple cases completely wrong, where a human would never err, and yet get some of the hardest cases - where a human would be very likely to make an error - right. It's baffling.

akyu 9y ago

>It doesn't have much with the actual innate quality of understanding That was his entire point.

westoncb 9y ago

> DL is just partitioning space according to some pre-baked associations given to it during training What I wonder is whether that's not also maybe the cornerstone of human understanding. If I understand correctly, you are essentially saying that DL is forming categories, or developing a classification scheme. Granted, if we're only talking about supervised DL, and the program is practically told where to form the boundaries—then it's not very impressive. But if the software is extracting statistically prominent commonalities and using those to form category boundaries, and arranging them hierarchically—then while the implementation may be totally different from human understanding, the effect seems to strongly overlap. I assume I'm probably just missing something—anyone know what it is? (It seems clear that at least part of the problem here is that 'human understanding' has been left far more vague than DL, and in order to say one way or the other how much they have in common, we need to better define 'human understanding'.)

bitL 9y ago
I think what DL is capable of is already very impressive; sometimes while playing with DL in NLP I am wondering if our languages aren't way simpler than we think/hope and we aren't that intelligent either. There is one massive difference between humans and DL; humans can mimic something just from a single (even partial) observation; DL requires huge amounts of data and massive parallel processing, something that we got only recently with Big Data and GPUs. DL also suffers from the curse of dimensionality; in theory deep fully connected networks should be able to do everything better, in practice they are awful and only cleverly constructed schemes like CNNs, LSTMs etc. that assume data to be in certain format/domain bring impressive results when paired with optimizers/metrics that magically work on a given dataset. If you are able to construct a DNN that can figure out PCA/ICA/eigenvalues/etc. on its own during its training like it does with convolutions, that would again enable another set of magic tricks. In any case, humans still have to figure out the architecture of the network that works (even if we now have AutoML for figuring out best hyperparameters in parallel). Then the hard problem of consciousness; I personally believe we are far off and probably miss something very important in our understanding of Universe.
- RivieraKid 9y ago