2 ms·

> Additionally, humans learn to speak and understand the world with a tiny fraction of the language data given to LLMs. Altough true that llms need more data t

> Additionally, humans learn to speak and understand the world with a tiny fraction of the language data given to LLMs. Altough true that llms need more data than a human, remember that these llms have a breadth of knowledge orders of magnitude larger than a single human. It is expected to understand every topic, every programming language and so on.

cma 4y ago
How well does it do given the linguistic data heard by a blind six year old in comparison to them? I would think they run circles around it in terms of correct grammar and that it would overfit to where more parameters would hurt. A newborn horse can almost immediately get up and walk around and navigate space. Animals have lots of innate stuff that doesn't use much training, and linguistic acquisition might have some similar innate system at its core.
- wizofaus 4y ago
  > linguistic acquisition might have some similar innate system I thought that was pretty well established. From memory, if certain parts of the brain are damaged at key developmental periods (generally before the age of 18), language is almost never acquired properly (even if the damage is repaired). I don't think there's any question human brains acquire language very differently to GPT technology. And they certainly acquire the ability to reason logically very differently - ChatGPT makes trivial errors with reasoning that it's hard to imagine an adult human doing (e.g. I just gave it a classic classification problem based on 3 made-up categories, and asked "if I see something green with three legs, what is it", and it replied (in part) "It is also not a foo, as all foos are green. Therefore, the green thing with three legs would be classified as...".
- mtlmtlmtlmtl 4y ago
  Afaik it's an open question what's innate and not. But it's a perfectly reasonable position that very little of it is. It seems more like what's innate is a tremendous ability to learn and shape the brain as it grows. And it's definitely the case that human brains can learn a lot more per amount of knlut data than current neural nets. And that's the big elephant in the room with these language models. It's a huge unsolved problem in machine learning. One clue to this is that it takes a pretty hefty artificial neural net to simulate a single cortical neuron. Neurons are a lot more intricate than a parameter in a linear equation. My feeling is that a fundamentally new approach to neural nets is needed for them to learn as efficiently as animal brains do.

cma 4y ago

How well does it do given the linguistic data heard by a blind six year old in comparison to them? I would think they run circles around it in terms of correct grammar and that it would overfit to where more parameters would hurt. A newborn horse can almost immediately get up and walk around and navigate space. Animals have lots of innate stuff that doesn't use much training, and linguistic acquisition might have some similar innate system at its core.

wizofaus 4y ago
> linguistic acquisition might have some similar innate system I thought that was pretty well established. From memory, if certain parts of the brain are damaged at key developmental periods (generally before the age of 18), language is almost never acquired properly (even if the damage is repaired). I don't think there's any question human brains acquire language very differently to GPT technology. And they certainly acquire the ability to reason logically very differently - ChatGPT makes trivial errors with reasoning that it's hard to imagine an adult human doing (e.g. I just gave it a classic classification problem based on 3 made-up categories, and asked "if I see something green with three legs, what is it", and it replied (in part) "It is also not a foo, as all foos are green. Therefore, the green thing with three legs would be classified as...".
mtlmtlmtlmtl 4y ago
Afaik it's an open question what's innate and not. But it's a perfectly reasonable position that very little of it is. It seems more like what's innate is a tremendous ability to learn and shape the brain as it grows. And it's definitely the case that human brains can learn a lot more per amount of knlut data than current neural nets. And that's the big elephant in the room with these language models. It's a huge unsolved problem in machine learning. One clue to this is that it takes a pretty hefty artificial neural net to simulate a single cortical neuron. Neurons are a lot more intricate than a parameter in a linear equation. My feeling is that a fundamentally new approach to neural nets is needed for them to learn as efficiently as animal brains do.