4 ms·

Babies don't come installed with those brains. They take a decade or two to pick up most skills that have any value. And after all that, the vast majority end u

by cen4 2y ago

Babies don't come installed with those brains. They take a decade or two to pick up most skills that have any value. And after all that, the vast majority end up playing replaceable support npc roles. What's the energy consumption? To get from - why is the compass pointing north to electromagnetism, it took thousands of years and the activity of millions of brains. We see a highly misleading and compressed version of that computing process in our textbooks. As they say knowledge is inherited wealth. People over estimate how much they deserve it or have earned it.

visarga 2y ago
A child learns to talk in about 150M tokens, GPT-4 needs 15T tokens. Humanity needed the total speech of 110B people to evolve to current level, which comes about 51,395,437,500,000,000,000 tokens, about 3.4 million times more than GPT-4's training run, and 342M times more than a child needs to learn. That is to show the relative cost between imitation and discovery. Evolution is on the order of 0.3 trillion times slower than catching up by imitation.
- defrost 2y ago
  > Humanity needed the total speech of 110B people to evolve to current level Seems a loooong stretch to say the least.
  - visarga 2y ago
    I find it interesting to estimate on tokens so we can meaningfully compare people with LLMs.
    - defrost 2y ago
      It's more the implied dependancy tree ... did Isaac Newton rest on the shoulders of 110 B people to conceive of calculus just as Liebnitz did the same? How much input did the Kalahari bushmen of that time, and their ancestors also, have to that infintesimal leap forwards?
      - visarga 2y ago
        The dependency tree is not made only of the winning "happy route", it contains all sorts of abandoned branches, otherwise it wouldn't function as an evolutionary system.
- continuational 2y ago
  That's apples to oranges. No human ingests that many tokens of speech; individually we learn from far fewer tokens.
  - famouswaffles 2y ago
    No human brain is the near blank slate an untrained transformer is either. "We only need x tokens of ingested speech" to learn language doesn't hit the same when you have billions of years of the brain baking in an oven to get to that point. But i agree, it's not a direct comparison.
    - Jensson 2y ago
      > No human brain is the near blank slate an untrained transformer is either. Why can't we start out a transformer at a better state and then teach it language in few tokens? Seems like a problem with architecture.
      - famouswaffles 2y ago
        Because we don't know or understand what a better state would be and so by far the most successful attempts are with architectures with as clean a slate/free of bias as possible. There's nothing wrong with blank slates. It's not a problem, whatever that means. It just is.
WheatMillington 2y ago
[flagged]
achierius 2y ago
>support npc roles This might sound trite, but are your parents "replaceable"? Even in a strict mechanistic sense, even a "baseline" parent possesses innumerable qualities that we have absolutely no clue how to build in a robot. They have a well-tuned cost function that keeps them or anything else from killing the baby; they come to learn the child well enough to understand what they're thinking, to correct them & guide them; they themselves are reliable enough to broadly survive until the child reaches adulthood (can you imagine swapping out your robot parents every iPhone generation?). Even their fine motor skills alone are far beyond what we today understand how to build (to cradle, feed, wash, &c). And that's not the only thing, just a convenient example of how the worldview you present -- one where the only value someone can provide is that which can be extracted by the owning class via their "job" -- is not only unkind but also incoherent. I don't even know what you mean by "deserve" or "earned": yes, none of us did anything to gain the attributes that we were born with, but so what? We are who we are, and who we are is defined in scope by who we were born to be. We are all given options and choose between them day by day. But nobody -- no thing -- chooses which options they are given.

by cen4 2y ago

visarga 2y ago

A child learns to talk in about 150M tokens, GPT-4 needs 15T tokens. Humanity needed the total speech of 110B people to evolve to current level, which comes about 51,395,437,500,000,000,000 tokens, about 3.4 million times more than GPT-4's training run, and 342M times more than a child needs to learn. That is to show the relative cost between imitation and discovery. Evolution is on the order of 0.3 trillion times slower than catching up by imitation.

defrost 2y ago
> Humanity needed the total speech of 110B people to evolve to current level Seems a loooong stretch to say the least.
- visarga 2y ago
  I find it interesting to estimate on tokens so we can meaningfully compare people with LLMs.
  - defrost 2y ago
    It's more the implied dependancy tree ... did Isaac Newton rest on the shoulders of 110 B people to conceive of calculus just as Liebnitz did the same? How much input did the Kalahari bushmen of that time, and their ancestors also, have to that infintesimal leap forwards?
    - visarga 2y ago
      The dependency tree is not made only of the winning "happy route", it contains all sorts of abandoned branches, otherwise it wouldn't function as an evolutionary system.
continuational 2y ago
That's apples to oranges. No human ingests that many tokens of speech; individually we learn from far fewer tokens.
- famouswaffles 2y ago
  No human brain is the near blank slate an untrained transformer is either. "We only need x tokens of ingested speech" to learn language doesn't hit the same when you have billions of years of the brain baking in an oven to get to that point. But i agree, it's not a direct comparison.
  - Jensson 2y ago
    > No human brain is the near blank slate an untrained transformer is either. Why can't we start out a transformer at a better state and then teach it language in few tokens? Seems like a problem with architecture.
    - famouswaffles 2y ago
      Because we don't know or understand what a better state would be and so by far the most successful attempts are with architectures with as clean a slate/free of bias as possible. There's nothing wrong with blank slates. It's not a problem, whatever that means. It just is.

WheatMillington 2y ago

[flagged]

achierius 2y ago

>support npc roles This might sound trite, but are your parents "replaceable"? Even in a strict mechanistic sense, even a "baseline" parent possesses innumerable qualities that we have absolutely no clue how to build in a robot. They have a well-tuned cost function that keeps them or anything else from killing the baby; they come to learn the child well enough to understand what they're thinking, to correct them & guide them; they themselves are reliable enough to broadly survive until the child reaches adulthood (can you imagine swapping out your robot parents every iPhone generation?). Even their fine motor skills alone are far beyond what we today understand how to build (to cradle, feed, wash, &c). And that's not the only thing, just a convenient example of how the worldview you present -- one where the only value someone can provide is that which can be extracted by the owning class via their "job" -- is not only unkind but also incoherent. I don't even know what you mean by "deserve" or "earned": yes, none of us did anything to gain the attributes that we were born with, but so what? We are who we are, and who we are is defined in scope by who we were born to be. We are all given options and choose between them day by day. But nobody -- no thing -- chooses which options they are given.