5 ms·

What's really interesting is that these models are using some non-trivial portion of all easily accessible human writing -- yet humans learn language really wel

by bane 5y ago

What's really interesting is that these models are using some non-trivial portion of all easily accessible human writing -- yet humans learn language really well with significantly less input data. What's missing in the field to replicate human performance in learning?

dlkf 5y ago
Humans use language to accomplish tasks in their environment - establishing relationships, making deals, coaxing others, etc. By contrast, all neural language models do is predict the next word as a function of the previous word. So far, these language models have nothing at all to do with language learning. They're only valuable insofar as they advance downstream engineering tasks like machine translation.
whimsicalism 5y ago
https://arxiv.org/pdf/1802.10217.pdf https://arxiv.org/pdf/1802.10217.pdf This is the paper I love to link in response to these sort of objections.
- YeGoblynQueenne 5y ago
  Human priors are a feature, not a bug. The reason deep neural networks need so much data to train them (and still do not handle languag nearly as well as humans) is precisely that humans have "background knowledge", things that we already know and don't have to learn all over again from scratch. We bring this knowledge to bear in our ability to learn and understand language. Deep neural networks on the other hand have only a very, very limited ability to represent and therefore use background knowledge- and so they have to make up for it with astounding amounts of data. The OP's criticism is valid. Being forced to learn everything end-to-end, from scratch, is a severe limitation.
  - bane 5y ago
    I think what's very interesting is that most of the answers to my question sort of boil down (if you squint a little at the answers) to "this is hard, but general AI will make it easy".
    - YeGoblynQueenne 5y ago
      My apologies if the wording of my comment was confusing but what you say is not at all what I meant.
  - whimsicalism 5y ago
    > Human priors are a feature, not a bug. ... Being forced to learn everything end-to-end, from scratch, is a severe limitation. They're neither. When performing very human-adjacent tasks, it will certainly put the ML algorithm at a disadvantage compared to us. But for non-human adjacent tasks, say interpreting what a sequence of amino acids actually means, we can expect the computer to absolutely crush us because our stupid human heuristics take us absolutely nowhere, cause us to see patterns that aren't there, etc. etc. Regardless, this is irrelevant to the original point that I was making, which is that comparing the performance of DL on human adjacent tasks to the amount of time it takes a human to learn the same task is misguided because you are ignoring the million-year long optimization process to get there.
visarga 5y ago
Imagine you lived in a black room and all you can see is a buffer of text scrolling in front of you. Nobody explains what the symbols mean. You don't remember anything medium term, you can only access a short snippet of text at a time and form long term memories gradually. You just look and predict what will come next. Who could become an intelligent person being raised in these conditions? So they are missing 2 years worth of visual, auditory, tactile and other modalities (grounding), having direct access to change their environment (embodiment) and being part of our society or an AI society (social).
ipsum2 5y ago
One dimension that isn't being considered is that humans have had billions of years of evolution, while neural networks are essentially a blank slate.
sytelus 5y ago
We sure don't read entire wikipedia but out speech/text consumption is pretty high and we take long time to learn. At 150 words/minute, I would say babies probably consume about 10 million tokens before they start to speak. Baby's training time is much higher than just few days compared for a GPU cluster. Also, baby's vocab is very small and can do very limited things compared to these large models (for example, can't answer who is the king of England or what is the capital of Bangladesh). This is not to say that language models are efficient, of course. That's not even remotely true. But we seem to under-estimate how much time and resources we need to learn something.

by bane 5y ago

dlkf 5y ago

Humans use language to accomplish tasks in their environment - establishing relationships, making deals, coaxing others, etc. By contrast, all neural language models do is predict the next word as a function of the previous word. So far, these language models have nothing at all to do with language learning. They're only valuable insofar as they advance downstream engineering tasks like machine translation.

whimsicalism 5y ago

https://arxiv.org/pdf/1802.10217.pdf https://arxiv.org/pdf/1802.10217.pdf This is the paper I love to link in response to these sort of objections.

YeGoblynQueenne 5y ago
Human priors are a feature, not a bug. The reason deep neural networks need so much data to train them (and still do not handle languag nearly as well as humans) is precisely that humans have "background knowledge", things that we already know and don't have to learn all over again from scratch. We bring this knowledge to bear in our ability to learn and understand language. Deep neural networks on the other hand have only a very, very limited ability to represent and therefore use background knowledge- and so they have to make up for it with astounding amounts of data. The OP's criticism is valid. Being forced to learn everything end-to-end, from scratch, is a severe limitation.
- bane 5y ago
  I think what's very interesting is that most of the answers to my question sort of boil down (if you squint a little at the answers) to "this is hard, but general AI will make it easy".
  - YeGoblynQueenne 5y ago
    My apologies if the wording of my comment was confusing but what you say is not at all what I meant.
- whimsicalism 5y ago
  > Human priors are a feature, not a bug. ... Being forced to learn everything end-to-end, from scratch, is a severe limitation. They're neither. When performing very human-adjacent tasks, it will certainly put the ML algorithm at a disadvantage compared to us. But for non-human adjacent tasks, say interpreting what a sequence of amino acids actually means, we can expect the computer to absolutely crush us because our stupid human heuristics take us absolutely nowhere, cause us to see patterns that aren't there, etc. etc. Regardless, this is irrelevant to the original point that I was making, which is that comparing the performance of DL on human adjacent tasks to the amount of time it takes a human to learn the same task is misguided because you are ignoring the million-year long optimization process to get there.

visarga 5y ago

Imagine you lived in a black room and all you can see is a buffer of text scrolling in front of you. Nobody explains what the symbols mean. You don't remember anything medium term, you can only access a short snippet of text at a time and form long term memories gradually. You just look and predict what will come next. Who could become an intelligent person being raised in these conditions? So they are missing 2 years worth of visual, auditory, tactile and other modalities (grounding), having direct access to change their environment (embodiment) and being part of our society or an AI society (social).

ipsum2 5y ago

One dimension that isn't being considered is that humans have had billions of years of evolution, while neural networks are essentially a blank slate.

sytelus 5y ago

We sure don't read entire wikipedia but out speech/text consumption is pretty high and we take long time to learn. At 150 words/minute, I would say babies probably consume about 10 million tokens before they start to speak. Baby's training time is much higher than just few days compared for a GPU cluster. Also, baby's vocab is very small and can do very limited things compared to these large models (for example, can't answer who is the king of England or what is the capital of Bangladesh). This is not to say that language models are efficient, of course. That's not even remotely true. But we seem to under-estimate how much time and resources we need to learn something.