3 ms·

> a machine was built that writes like a human But the real hero here is not the LLM, but the training set. It took ages to collect all the knowledge, ideas an

> a machine was built that writes like a human But the real hero here is not the LLM, but the training set. It took ages to collect all the knowledge, ideas and methods we put in books. It cost a lot of human effort to provide the data. Without the data we would have nothing. Without GPT we could use RWKV, Mamba, S4, etc and still get similar results. It's the data not the model. > the intuition that language is central to thought predates LLMs Language carries AI and humans. The same distribution of language can be the software running in our brains and in LLMs. I think humans act like conditional language models with multi modality and actions. We use language to plan and solve our problem, work together and learn (a lot) from others. Language itself is an evolutionary system and a self replicator. Its speed is much faster than biology. We've been on the language exponential for millennia, but just now hit the critical mass for LLMs to be possible. It's not so important that GPT-4 is a 2T weights model, what matters is that it was trained on 13T tokens of human experience and it now "writes like a human". Does that mean humans also learn the same skills GPT-4 has learned from its training set mostly by language as well?

AlecSchueler 3y ago
> But the real hero here is not the LLM, but the training set. And in the case of windmills the hero is the wind. But the mill is still a fantastic achievement.
- mewpmewp2 3y ago
  But can mills easily move? No, people can, so people are still viable.
  - coldtea 3y ago
    >But can mills easily move? Yes, that's their whole thing: moving when the air blows.
    - mewpmewp2 3y ago
      But can they relocate?
- riwsky 3y ago
  And if you rearrange the letters in “mill”? “I, LLM.” qed
  - razodactyl 3y ago
    Correlation checks out.
nickpsecurity 3y ago
Indeed, they are mostly parroting that knowledge with some reasoning from combinations of those patterns. That counts for something. It’s not what we do, though. Or not all we do. Humans can just sit around aimlessly toying with stuff, reading books, etc. They’ll figure out some of these patterns on their own. Whereas, we have to give these things a ton of highly-curated, pre-processed data made by human minds of all kinds. Then, it’s usually 800GB-4TB for the good ones. They’re appear to be not in our league yet as learning machines. We’ll be able to assess it better as multimodal models come online. We can train them like infants, then children, on books, random observations through cameras, TV, people reading to them, supervised feedback… all the stuff we do with humans. Then see if and how they match up in performance.

AlecSchueler 3y ago

> But the real hero here is not the LLM, but the training set. And in the case of windmills the hero is the wind. But the mill is still a fantastic achievement.

mewpmewp2 3y ago
But can mills easily move? No, people can, so people are still viable.
- coldtea 3y ago
  >But can mills easily move? Yes, that's their whole thing: moving when the air blows.
  - mewpmewp2 3y ago
    But can they relocate?
riwsky 3y ago
And if you rearrange the letters in “mill”? “I, LLM.” qed
- razodactyl 3y ago
  Correlation checks out.

nickpsecurity 3y ago

Indeed, they are mostly parroting that knowledge with some reasoning from combinations of those patterns. That counts for something. It’s not what we do, though. Or not all we do. Humans can just sit around aimlessly toying with stuff, reading books, etc. They’ll figure out some of these patterns on their own. Whereas, we have to give these things a ton of highly-curated, pre-processed data made by human minds of all kinds. Then, it’s usually 800GB-4TB for the good ones. They’re appear to be not in our league yet as learning machines. We’ll be able to assess it better as multimodal models come online. We can train them like infants, then children, on books, random observations through cameras, TV, people reading to them, supervised feedback… all the stuff we do with humans. Then see if and how they match up in performance.