4 ms·

Humans are not exposed to "billions of words". They are exposed to continuous signals, and somehow extract meaningful representations, like "words" and "objects

Humans are not exposed to "billions of words". They are exposed to continuous signals, and somehow extract meaningful representations, like "words" and "objects", out of them. LLMs are exposed to human-curated data. Let's see an LLM curate its own data out of nothing but experience of raw continuous signals. Sure, GPT trained on the internet is a nice way to condense and retrieve human-curated data. But it's not going to give us a Lt. Data or C-3P0 that can learn and adapt in real time.

pbw 4y ago
> Humans are not exposed to "billions of words" ... continuous signals Yes humans need to learn to extract words from signals, but that does not mean it's not correct to count the number of words they've been exposed to in their lifetime. Your comment is like saying we can't count the number of hamburgers someone has eaten, because they actually eat myofibrillar proteins, not burgers.
- d0mine 4y ago
  Though millions seems more realistic (thousands of words per day)
andsoitis 4y ago
what do you think of midjourney's ability to describe a user-provided image in text? https://the-decoder.com/midjourney-new-image-tool-works-in-reverse/ https://the-decoder.com/midjourney-new-image-tool-works-in-r...
gunshai 4y ago
But humans are also trained on human-curated data so much so we put fairly hefty price tags on that curation. https://miro.medium.com/v2/resize:fit:1056/0*E1eNateTiDThGcYI.jpg https://miro.medium.com/v2/resize:fit:1056/0*E1eNateTiDThGcY...
echelon 4y ago
We've only just started. Now more capital and more minds will be pouring into solving this. Buckle up.
Sparkyte 4y ago
Also LLM are not AI just parts to AI you need a bunch of interfacing and injection components. The sum of which makes up AI. I see the progress of AI stagnating while people board the next equivalent of Crypto craze because someone want to financially profit off something not fully seeing realization. However it is not without merit which Crypto was completely without merit and something to show for itself. As flawed and imperfect ChatGPT is it clearly mirrors its creators and that is a compliment.
Femtodjy 4y ago
Why not? Humans learn based on human curated data too. School is not natural. Our whole env is neither. Babies can't survive. I would even go so far to say that the potential model a LLM would create internally might not be that far away of that of a human. And segment anything was just announced. The performance of zero shot systems is tremendous. It's not far fetched to assume that chatgpt combined with segment anything together would allow it to create an even more accurate model of the world.
- bluefirebrand 4y ago
  > Humans learn based on human curated data too. > School is not natural. Our whole env is neither. Babies can't survive A human goes to school to learn. Humans in general learned based on experience of the world around them. We invented language, no one taught it to us. We learned to make fire, forge tools, cook food, practice medicine, etc on our own. School is just how we pass down that learning.
  - dwaltrip 4y ago
    Humanity invented those things. Individual humans were taught those things, either directly or indirectly by observing / listening to others. We do learn through our experience, of course. But most of what we learn is from others in one form or another.
  - pixl97 4y ago
    If I took you and pitched you on an island as an exceptionally young child with no further training, most likely you would die. Even more likely you wouldn't have fire and tools. A high proportion of our behaviors can be traced as a continuous learning chain going back eons.
    - hnfong 4y ago
      https://en.wikipedia.org/wiki/Feral_child https://en.wikipedia.org/wiki/Feral_child A lot of these children had trouble with learning language after they returned to human civilization.
com2kid 4y ago
> Humans are not exposed to "billions of words". They are exposed to continuous signals, and somehow extract meaningful representations, like "words" and "objects", out of them. Plenty of studies showing long term academic achievement differences based on number of unique words babies are exposed to. And I promise you, having a baby/toddler is really damn close to doing data labeling. Reading picture books, you are basically labeling objects. Walking around the grocery store, you are labeling objects. All the time, again and again, and the same object will get labeled repeatedly, and sometimes it will even be fact checked. A toddler will point to something that they know is an orange, ask "apple?" and you had sure as hell better reply "orange".
- intalentive 4y ago
  >having a baby/toddler is really damn close to doing data labeling It's analogous, sure, but not really in the same ballpark. There is no AI system today that you can teach by pointing to things and making noises with your mouth. Why not? What's missing? An awful lot.
  - ben_w 4y ago
    Is anything missing? We've got gesture recognition, so we can do pointing; and that part of the system taken in isolation is not made any harder when the label for an image happens to be from, say, a latent space vector from the interior of an unrelated GAN whose sole purpose is to turn audio input into the specific set of muscle outputs which cause the vocal tract to produce output that sounds the same as a recently heard noise.
macrolocal 4y ago
I sometimes like to imagine the umwelt of an LLM. Tokens are their sensory atoms, and the statistical relationships between them form their entire reality. Of course, one could easily build a model whose tokens line up with qualia we experience.
- actionfromafar 4y ago
  Easily? In principle yes, but we don't have the data for it. (Yet.)
  - macrolocal 4y ago
    Sure, easily. On example is vision: you'd just have to adjust input preprocessing in existing models.
- kelseyfrog 4y ago
  Maybe, but I've been told by folks (and curiously LLMs-themselves) that LLMs are ontologically incapable of having a what-is-it-like-to-be-ness.
  - macrolocal 4y ago
    No matter what your ontological commitments happen to be, imagining what an LLM's umwelt might be like, and how it might approximate ours, is a useful (and fun!) exercise.
  - ben_w 4y ago
    Every time it comes up in discussion, I find the same patterns and the same conversations. Nobody really knows what is necessary or (let alone and) sufficient for having a what-is-it-like-to-be-ness. Lots of us claim we know, but whenever I've asked for details, it's always been some combination of (α) circular, e.g. "consciousness is sentience, sentience is consciousness"; (β) something that's so vague it accidentally includes VCRs because they can record information that can be recalled later; and/or (γ) that doesn't include all humans (most often by excluding those with aphantasia and/or amnesia, but not only). So… perhaps any given LLMs does, and perhaps it doesn't. Perhaps the structure can support qualia, but only actually gets them when the training process passes the analog of a thermodynamic phase change; or perhaps, even if that's the right general idea, qualia is a different phase change that this model can't support. Unfortunately, I can't find that phase change analogy with a quick web search, so I'm not sure if that analogy has ever really been made before or if it was just a dream I've misfiled as a memory of reality.
williamcotton 4y ago
I would wager $1,000 that many less words are required with a model that trains on audio, video and text. I bet an android-type machine that can move and interact and train without curated data on everyday human interactions would improve on current LLMs

pbw 4y ago

> Humans are not exposed to "billions of words" ... continuous signals Yes humans need to learn to extract words from signals, but that does not mean it's not correct to count the number of words they've been exposed to in their lifetime. Your comment is like saying we can't count the number of hamburgers someone has eaten, because they actually eat myofibrillar proteins, not burgers.

d0mine 4y ago
Though millions seems more realistic (thousands of words per day)

andsoitis 4y ago

what do you think of midjourney's ability to describe a user-provided image in text? https://the-decoder.com/midjourney-new-image-tool-works-in-reverse/ https://the-decoder.com/midjourney-new-image-tool-works-in-r...

gunshai 4y ago

But humans are also trained on human-curated data so much so we put fairly hefty price tags on that curation. https://miro.medium.com/v2/resize:fit:1056/0*E1eNateTiDThGcYI.jpg https://miro.medium.com/v2/resize:fit:1056/0*E1eNateTiDThGcY...

echelon 4y ago

We've only just started. Now more capital and more minds will be pouring into solving this. Buckle up.

Sparkyte 4y ago

Also LLM are not AI just parts to AI you need a bunch of interfacing and injection components. The sum of which makes up AI. I see the progress of AI stagnating while people board the next equivalent of Crypto craze because someone want to financially profit off something not fully seeing realization. However it is not without merit which Crypto was completely without merit and something to show for itself. As flawed and imperfect ChatGPT is it clearly mirrors its creators and that is a compliment.

Femtodjy 4y ago

Why not? Humans learn based on human curated data too. School is not natural. Our whole env is neither. Babies can't survive. I would even go so far to say that the potential model a LLM would create internally might not be that far away of that of a human. And segment anything was just announced. The performance of zero shot systems is tremendous. It's not far fetched to assume that chatgpt combined with segment anything together would allow it to create an even more accurate model of the world.

bluefirebrand 4y ago
> Humans learn based on human curated data too. > School is not natural. Our whole env is neither. Babies can't survive A human goes to school to learn. Humans in general learned based on experience of the world around them. We invented language, no one taught it to us. We learned to make fire, forge tools, cook food, practice medicine, etc on our own. School is just how we pass down that learning.
- dwaltrip 4y ago
  Humanity invented those things. Individual humans were taught those things, either directly or indirectly by observing / listening to others. We do learn through our experience, of course. But most of what we learn is from others in one form or another.
- pixl97 4y ago
  If I took you and pitched you on an island as an exceptionally young child with no further training, most likely you would die. Even more likely you wouldn't have fire and tools. A high proportion of our behaviors can be traced as a continuous learning chain going back eons.
  - hnfong 4y ago
    https://en.wikipedia.org/wiki/Feral_child https://en.wikipedia.org/wiki/Feral_child A lot of these children had trouble with learning language after they returned to human civilization.

com2kid 4y ago

> Humans are not exposed to "billions of words". They are exposed to continuous signals, and somehow extract meaningful representations, like "words" and "objects", out of them. Plenty of studies showing long term academic achievement differences based on number of unique words babies are exposed to. And I promise you, having a baby/toddler is really damn close to doing data labeling. Reading picture books, you are basically labeling objects. Walking around the grocery store, you are labeling objects. All the time, again and again, and the same object will get labeled repeatedly, and sometimes it will even be fact checked. A toddler will point to something that they know is an orange, ask "apple?" and you had sure as hell better reply "orange".

intalentive 4y ago
>having a baby/toddler is really damn close to doing data labeling It's analogous, sure, but not really in the same ballpark. There is no AI system today that you can teach by pointing to things and making noises with your mouth. Why not? What's missing? An awful lot.
- ben_w 4y ago
  Is anything missing? We've got gesture recognition, so we can do pointing; and that part of the system taken in isolation is not made any harder when the label for an image happens to be from, say, a latent space vector from the interior of an unrelated GAN whose sole purpose is to turn audio input into the specific set of muscle outputs which cause the vocal tract to produce output that sounds the same as a recently heard noise.

macrolocal 4y ago

I sometimes like to imagine the umwelt of an LLM. Tokens are their sensory atoms, and the statistical relationships between them form their entire reality. Of course, one could easily build a model whose tokens line up with qualia we experience.

actionfromafar 4y ago
Easily? In principle yes, but we don't have the data for it. (Yet.)
- macrolocal 4y ago
  Sure, easily. On example is vision: you'd just have to adjust input preprocessing in existing models.
kelseyfrog 4y ago
Maybe, but I've been told by folks (and curiously LLMs-themselves) that LLMs are ontologically incapable of having a what-is-it-like-to-be-ness.
- macrolocal 4y ago
  No matter what your ontological commitments happen to be, imagining what an LLM's umwelt might be like, and how it might approximate ours, is a useful (and fun!) exercise.
- ben_w 4y ago
  Every time it comes up in discussion, I find the same patterns and the same conversations. Nobody really knows what is necessary or (let alone and) sufficient for having a what-is-it-like-to-be-ness. Lots of us claim we know, but whenever I've asked for details, it's always been some combination of (α) circular, e.g. "consciousness is sentience, sentience is consciousness"; (β) something that's so vague it accidentally includes VCRs because they can record information that can be recalled later; and/or (γ) that doesn't include all humans (most often by excluding those with aphantasia and/or amnesia, but not only). So… perhaps any given LLMs does, and perhaps it doesn't. Perhaps the structure can support qualia, but only actually gets them when the training process passes the analog of a thermodynamic phase change; or perhaps, even if that's the right general idea, qualia is a different phase change that this model can't support. Unfortunately, I can't find that phase change analogy with a quick web search, so I'm not sure if that analogy has ever really been made before or if it was just a dream I've misfiled as a memory of reality.

williamcotton 4y ago

I would wager $1,000 that many less words are required with a model that trains on audio, video and text. I bet an android-type machine that can move and interact and train without curated data on everyday human interactions would improve on current LLMs