4 ms·

> Most humans don't need to ingest terabytes of domain specific data in order to engage in basic reading comprehension and reasoning. How much data do you thin

> Most humans don't need to ingest terabytes of domain specific data in order to engage in basic reading comprehension and reasoning. How much data do you think a 6 year old has processed by the time they read a book? Eyes and ears are pretty high bitrate input devices.

russdill 3y ago
Not only that, the built-in connection network that developed over hundreds of millions of years from untold trillions of organisms.
bluetomcat 3y ago
A child doesn’t process “data” in the form of a sequence of tokens. A child associates visual objects with similar traits to pronounced words, heard from their parents. It also acquires a basic feeling of temporal sequentiality and physical causality by throwing toys, for example. It learns the rules of the language grammar by subconsciously noticing the basic structure of sentences with objects, verbs and qualities.
- xboxnolifes 3y ago
  > A child doesn’t process “data” in the form of a sequence of tokens. A child associates visual objects with similar traits to pronounced words, heard from their parents. Objects and words are tokens. At least, objects and words are not fundamental types of things. Before they are tokens, they exist only through light and sound waves that we perceive (and other sensations such as touch, smell). Our eyes and ears need to process this information before it becomes objects and words.
korse 3y ago
No contest about the 'bitrate' for eyes and ears, but we don't have to feed kids the entire library of congress in order for them to reason about text. This is what gives me cold feet about the current wave of LLMs. Where is this going? We'll have something that people will dub 'AGI'. It will seem intelligent and may or may not be catastrophic, but whether or not it actually has a sense of self will be an open question, second only to the fact that it requires the power output of Argentina to survive? Seems like a dumb hack, not an advancement indicative of a thorough understanding of the problem. If and when, then maybe we can use it to bio-engineer our way to a better future?
- lostmsu 3y ago
  I think you are plainly wrong here. Imagine a thought experiment where you'd plug newborn's ears and cover eyes until they are 6yo and see how much data you'd have to feed to them before they can reason about text from this point. I imagine it will be in hundreds of terabytes or even petabytes. The idea that children learn from little is naive.
  - korse 3y ago
    I did not state that children learn from little. I did state that they are able to achieve impressive outcomes quickly with very small amounts of domain specific knowledge (movement not included). Our best language models at this point appear to be narrow and deep where the child model is wide and shallow. For example, how many pictures of apples do I need in order to train a robot to recognize apples? How many apples (or representations thereof) does a child need to see to identify an apple? I won't argue about image density, but human eyes aren't that high resolution and people don't seem to have excellent high resolution multi-frame memory. This just makes we wonder if we should be spending more time working on how we utilize our wide but shallow data corpus rather than playing I can't believe it's not statistics with hugely inefficient and focused data sets.
    - lostmsu 3y ago
      You are wrong again. There's no difference between ChatGPT and a child in that regard. That's like half of the point of GPT's zero shot performance on the tasks that involve made up language that you describe directly in the query. You are doubly wrong because somehow for a robot you are counting pictures of apples, and for a child you are counting distinct apples. The answer to "how many pictures of apples does child need to see" is somewhere in thousands. Remember, brain does an equivalent of 24 fps, so a child holding an apple for 5 seconds gets an equivalent of over 100 pictures.
    - lostmsu 3y ago
      Here, stumbled upon in a different thread: https://news.ycombinator.com/item?id=37273290 https://news.ycombinator.com/item?id=37273290 Too bad they are not saying how long it took to get to the recognition.

russdill 3y ago

Not only that, the built-in connection network that developed over hundreds of millions of years from untold trillions of organisms.

bluetomcat 3y ago

A child doesn’t process “data” in the form of a sequence of tokens. A child associates visual objects with similar traits to pronounced words, heard from their parents. It also acquires a basic feeling of temporal sequentiality and physical causality by throwing toys, for example. It learns the rules of the language grammar by subconsciously noticing the basic structure of sentences with objects, verbs and qualities.

xboxnolifes 3y ago
> A child doesn’t process “data” in the form of a sequence of tokens. A child associates visual objects with similar traits to pronounced words, heard from their parents. Objects and words are tokens. At least, objects and words are not fundamental types of things. Before they are tokens, they exist only through light and sound waves that we perceive (and other sensations such as touch, smell). Our eyes and ears need to process this information before it becomes objects and words.

korse 3y ago

No contest about the 'bitrate' for eyes and ears, but we don't have to feed kids the entire library of congress in order for them to reason about text. This is what gives me cold feet about the current wave of LLMs. Where is this going? We'll have something that people will dub 'AGI'. It will seem intelligent and may or may not be catastrophic, but whether or not it actually has a sense of self will be an open question, second only to the fact that it requires the power output of Argentina to survive? Seems like a dumb hack, not an advancement indicative of a thorough understanding of the problem. If and when, then maybe we can use it to bio-engineer our way to a better future?

lostmsu 3y ago
I think you are plainly wrong here. Imagine a thought experiment where you'd plug newborn's ears and cover eyes until they are 6yo and see how much data you'd have to feed to them before they can reason about text from this point. I imagine it will be in hundreds of terabytes or even petabytes. The idea that children learn from little is naive.
- korse 3y ago
  I did not state that children learn from little. I did state that they are able to achieve impressive outcomes quickly with very small amounts of domain specific knowledge (movement not included). Our best language models at this point appear to be narrow and deep where the child model is wide and shallow. For example, how many pictures of apples do I need in order to train a robot to recognize apples? How many apples (or representations thereof) does a child need to see to identify an apple? I won't argue about image density, but human eyes aren't that high resolution and people don't seem to have excellent high resolution multi-frame memory. This just makes we wonder if we should be spending more time working on how we utilize our wide but shallow data corpus rather than playing I can't believe it's not statistics with hugely inefficient and focused data sets.
  - lostmsu 3y ago
    You are wrong again. There's no difference between ChatGPT and a child in that regard. That's like half of the point of GPT's zero shot performance on the tasks that involve made up language that you describe directly in the query. You are doubly wrong because somehow for a robot you are counting pictures of apples, and for a child you are counting distinct apples. The answer to "how many pictures of apples does child need to see" is somewhere in thousands. Remember, brain does an equivalent of 24 fps, so a child holding an apple for 5 seconds gets an equivalent of over 100 pictures.
  - lostmsu 3y ago
    Here, stumbled upon in a different thread: https://news.ycombinator.com/item?id=37273290 https://news.ycombinator.com/item?id=37273290 Too bad they are not saying how long it took to get to the recognition.