3 ms·

No contest about the 'bitrate' for eyes and ears, but we don't have to feed kids the entire library of congress in order for them to reason about text. This is

by korse 3y ago

No contest about the 'bitrate' for eyes and ears, but we don't have to feed kids the entire library of congress in order for them to reason about text. This is what gives me cold feet about the current wave of LLMs. Where is this going? We'll have something that people will dub 'AGI'. It will seem intelligent and may or may not be catastrophic, but whether or not it actually has a sense of self will be an open question, second only to the fact that it requires the power output of Argentina to survive? Seems like a dumb hack, not an advancement indicative of a thorough understanding of the problem. If and when, then maybe we can use it to bio-engineer our way to a better future?

lostmsu 3y ago
I think you are plainly wrong here. Imagine a thought experiment where you'd plug newborn's ears and cover eyes until they are 6yo and see how much data you'd have to feed to them before they can reason about text from this point. I imagine it will be in hundreds of terabytes or even petabytes. The idea that children learn from little is naive.
- korse 3y ago
  I did not state that children learn from little. I did state that they are able to achieve impressive outcomes quickly with very small amounts of domain specific knowledge (movement not included). Our best language models at this point appear to be narrow and deep where the child model is wide and shallow. For example, how many pictures of apples do I need in order to train a robot to recognize apples? How many apples (or representations thereof) does a child need to see to identify an apple? I won't argue about image density, but human eyes aren't that high resolution and people don't seem to have excellent high resolution multi-frame memory. This just makes we wonder if we should be spending more time working on how we utilize our wide but shallow data corpus rather than playing I can't believe it's not statistics with hugely inefficient and focused data sets.
  - lostmsu 3y ago
    You are wrong again. There's no difference between ChatGPT and a child in that regard. That's like half of the point of GPT's zero shot performance on the tasks that involve made up language that you describe directly in the query. You are doubly wrong because somehow for a robot you are counting pictures of apples, and for a child you are counting distinct apples. The answer to "how many pictures of apples does child need to see" is somewhere in thousands. Remember, brain does an equivalent of 24 fps, so a child holding an apple for 5 seconds gets an equivalent of over 100 pictures.
  - lostmsu 3y ago
    Here, stumbled upon in a different thread: https://news.ycombinator.com/item?id=37273290 https://news.ycombinator.com/item?id=37273290 Too bad they are not saying how long it took to get to the recognition.

by korse 3y ago

lostmsu 3y ago

I think you are plainly wrong here. Imagine a thought experiment where you'd plug newborn's ears and cover eyes until they are 6yo and see how much data you'd have to feed to them before they can reason about text from this point. I imagine it will be in hundreds of terabytes or even petabytes. The idea that children learn from little is naive.

korse 3y ago
I did not state that children learn from little. I did state that they are able to achieve impressive outcomes quickly with very small amounts of domain specific knowledge (movement not included). Our best language models at this point appear to be narrow and deep where the child model is wide and shallow. For example, how many pictures of apples do I need in order to train a robot to recognize apples? How many apples (or representations thereof) does a child need to see to identify an apple? I won't argue about image density, but human eyes aren't that high resolution and people don't seem to have excellent high resolution multi-frame memory. This just makes we wonder if we should be spending more time working on how we utilize our wide but shallow data corpus rather than playing I can't believe it's not statistics with hugely inefficient and focused data sets.
- lostmsu 3y ago
  You are wrong again. There's no difference between ChatGPT and a child in that regard. That's like half of the point of GPT's zero shot performance on the tasks that involve made up language that you describe directly in the query. You are doubly wrong because somehow for a robot you are counting pictures of apples, and for a child you are counting distinct apples. The answer to "how many pictures of apples does child need to see" is somewhere in thousands. Remember, brain does an equivalent of 24 fps, so a child holding an apple for 5 seconds gets an equivalent of over 100 pictures.
- lostmsu 3y ago
  Here, stumbled upon in a different thread: https://news.ycombinator.com/item?id=37273290 https://news.ycombinator.com/item?id=37273290 Too bad they are not saying how long it took to get to the recognition.