6 ms·

Look i'm not going to say the transformer is as efficient as the brain but you are not starting from Zero. any Code LLM will be learning language and code and

Look i'm not going to say the transformer is as efficient as the brain but you are not starting from Zero. any Code LLM will be learning language and code and everything else, with absolutely no predisposition to either. Your brain is baked with millions of years of evolution with specific areas already predisposed to certain types of processing before you ever utter a word.

coldtea 3y ago
>Look i'm not going to say the transformer is as efficient as the brain but you are not starting from Zero. Still, we can rewrite the parent's argument as: If we train an AI on the amount of non-code-related dataset (writing read/speech heard) I've consumed, and then add to it all the amount of code-related writing/speech (coding book, coding lessons taken, code and manual pages read, man pages, etc.) I've consumed, would it even remotely as good as coding as me? Or even as good as itself is now? I'd guess no. It's less effiecient, and thus needs way more coding dataset to get the point of coding than a human. Which brings us to: >Your brain is baked with millions of years of evolution with specific areas already predisposed to certain types of processing before you ever utter a word. Isn't that the whole point the parent is making? That our advantage isn't about dataset-volume, but architecture.
- famouswaffles 3y ago
  The closest biological equivalent to a parameter in an ann is a synapse. Well humans have about 100 trillion synapses. We already know that the higher the parameter count, the lower the training data required. a 50 billion parameter model will far outperform a 5 billion one trained on the same data. and a 500b one would far outperform that 50 billion one. Current LLMs are actually nowhere near the scale of the human brain, either in parameters/neurons or training data (all the text we've ever trained an LLM on would be dwarfed by all the sense data humans perceive), as well as not having the headstart the human brain has. It's a bogus comparison when you really think about it. You could easily make the case that LLMs are far more efficient.
  - chaxor 3y ago
    It is a bogus comparison, because typical language models used already have textual representations, and output textual representations; which is a very small fraction of 'the brain'. An astounding portion of the brain's neurons really go towards proprioception, smell, taste, motor function, etc. Which are not at all even slightly part of most models today. Wernicke's area, a small sliver of frontal cortex, and a dash of dopaminergic circuitry is maybe the best 'coverage' made in these models if you want to be exceedingly facile/ham-handed about the analogies here. That's a very small portion of cortex, and much closer than what you may think in terms of capability/ TEPS and unit count.
    - reaperman 3y ago
      It's not clear how many classical calculations a single human neuron is equivalent to. There's a strong analog component in multiple domains (strength, frequency, timing) and each neuron can connect to up to 15,000 other neurons. Assuming the brain's neurons are (probably unrealistically) fairly 'digital' we get an estimation of the human brain being equivalent to 1 exaflop (this is the currently accepted lower bound, and rather disputed as being too low). Current TPUv4 pods currently provide approximately 9 exaflops. I don't think we're currently reaching human-level learning rates. There’s currently no accepted “upper bound” on estimates of FLOP equivalency to a human brain. https://www.youtube.com/watch?v=HB5TrK7A4pI https://www.youtube.com/watch?v=HB5TrK7A4pI is a recently posted video to HN Frontpage which was summarized as such: > Though we have been building and programming computing machines for about 60 years and have learned a great deal about composition and abstraction, we have just begun to scratch the surface. > A mammalian neuron takes about ten milliseconds to respond to a stimulus. A driver can respond to a visual stimulus in a few hundred milliseconds, and decide an action, such as making a turn. So the computational depth of this behavior is only a few tens of steps. We don't know how to make such a machine, and we wouldn't know how to program it. > The human genome -- the information required to build a human from a single, undifferentiated eukariotic cell -- is about 1GB. The instructions to build a mammal are written in very dense code, and the program is extremely flexible. Only small patches to the human genome are required to build a cow or a dog rather than a human. Bigger patches result in a frog or a snake. We don't have any idea how to make a description of such a complex machine that is both dense and flexible. > New design principles and new linguistic support are needed. I will address this issue and show some ideas that can perhaps get us to the next phase of engineering design. > Gerald Sussman Massachusetts Institute of Technology
- dwallin 3y ago
  That’s an arbitrary line to make though. The human brain starts with a general architecture that has been gone through billions and trillions of generations of evolutionary training, before being fine tuned in a single individual over decades, and then you did a little bit of fine tuning and few shot at the end and claim that is comparable to the entire training of a LLM from scratch? Not to mention the many more orders of magnitude of neurons that a human brain has. I could equally argue that an LLM takes zero training, since we have ALREADY trained models and I can just copy the model and run it and get a new “brain” performing a task, instead of taking decades of training to get there. Even your statement about programming skills is debatable, it depends on how you measure programming skill. They certainly are faster at it, and they know more computer languages than most people have even heard of. In fact, human programming strength seems to be more about general logic and planning skills over programming-specific skills, both things where the bulk of training happened evolutionarily and more generally over the course of a life. The truth is, the two are not directly comparable. They are completely different architectures, at completely different scales, with entirely different strengths and weaknesses.
aledalgrande 3y ago
Also multi modal / multi topic learning benefits the brain too. Range by David Epstein is an interesting read
hannasm 3y ago
The training process is finding local minimums based around an initialization vector of random numbers. 1000s of years of evolution probably mean you were initialized better than a baby AI using a pseudo random number generator.
- ddalex 3y ago
  Watching my kid learn to speak made it clear to me that we are pre-wired for the acquisition of language - we're not building the structures from scratch, we are learning to put words into the empty spaces. Probably natural evolution at best - no speech, lower chance of gene propagation
Taek 3y ago
I feel like there should be a basic 'curriculum' that gets passed to all foundational LLMs that teaches them the basics of language. Maybe 100 million files where the first 10 million are all first grade reading level, the second 10 million are all second grade reading level, etc. Ideally this includes a bunch of text books. That should give the LLM time to grok language before it starts training on more difficult texts.
visarga 3y ago
> Your brain is baked with millions of years of evolution Exactly, and not only that, we are agents from birth, so we enjoy the 5 E's: "Embodied, Embedded, Enacted and Extended into the Environment" LLM's can't even write a script to see if it works, they got no feedback and can't make any meaningful choice with consequences. They are trained with "teacher forcing" and self-supervised methods, no deviation allowed. On the other hand we got THE WORLD which is infinitely more rich than any simulation, and human society which is made of super-GPT agents, and search based access to information. Remember, most LLMs work closed book and train on a dry, static dataset. Don't directly compare them with humans. Humans can't write great code top-down without computers either. We are trial-and-feedback monkeys, without feedback we're just no good.
manojlds 3y ago
And the point of aeroplanes wasn't to mimic birds but to fly.

coldtea 3y ago

>Look i'm not going to say the transformer is as efficient as the brain but you are not starting from Zero. Still, we can rewrite the parent's argument as: If we train an AI on the amount of non-code-related dataset (writing read/speech heard) I've consumed, and then add to it all the amount of code-related writing/speech (coding book, coding lessons taken, code and manual pages read, man pages, etc.) I've consumed, would it even remotely as good as coding as me? Or even as good as itself is now? I'd guess no. It's less effiecient, and thus needs way more coding dataset to get the point of coding than a human. Which brings us to: >Your brain is baked with millions of years of evolution with specific areas already predisposed to certain types of processing before you ever utter a word. Isn't that the whole point the parent is making? That our advantage isn't about dataset-volume, but architecture.

famouswaffles 3y ago
The closest biological equivalent to a parameter in an ann is a synapse. Well humans have about 100 trillion synapses. We already know that the higher the parameter count, the lower the training data required. a 50 billion parameter model will far outperform a 5 billion one trained on the same data. and a 500b one would far outperform that 50 billion one. Current LLMs are actually nowhere near the scale of the human brain, either in parameters/neurons or training data (all the text we've ever trained an LLM on would be dwarfed by all the sense data humans perceive), as well as not having the headstart the human brain has. It's a bogus comparison when you really think about it. You could easily make the case that LLMs are far more efficient.
- chaxor 3y ago
  It is a bogus comparison, because typical language models used already have textual representations, and output textual representations; which is a very small fraction of 'the brain'. An astounding portion of the brain's neurons really go towards proprioception, smell, taste, motor function, etc. Which are not at all even slightly part of most models today. Wernicke's area, a small sliver of frontal cortex, and a dash of dopaminergic circuitry is maybe the best 'coverage' made in these models if you want to be exceedingly facile/ham-handed about the analogies here. That's a very small portion of cortex, and much closer than what you may think in terms of capability/ TEPS and unit count.
  - reaperman 3y ago
    It's not clear how many classical calculations a single human neuron is equivalent to. There's a strong analog component in multiple domains (strength, frequency, timing) and each neuron can connect to up to 15,000 other neurons. Assuming the brain's neurons are (probably unrealistically) fairly 'digital' we get an estimation of the human brain being equivalent to 1 exaflop (this is the currently accepted lower bound, and rather disputed as being too low). Current TPUv4 pods currently provide approximately 9 exaflops. I don't think we're currently reaching human-level learning rates. There’s currently no accepted “upper bound” on estimates of FLOP equivalency to a human brain. https://www.youtube.com/watch?v=HB5TrK7A4pI https://www.youtube.com/watch?v=HB5TrK7A4pI is a recently posted video to HN Frontpage which was summarized as such: > Though we have been building and programming computing machines for about 60 years and have learned a great deal about composition and abstraction, we have just begun to scratch the surface. > A mammalian neuron takes about ten milliseconds to respond to a stimulus. A driver can respond to a visual stimulus in a few hundred milliseconds, and decide an action, such as making a turn. So the computational depth of this behavior is only a few tens of steps. We don't know how to make such a machine, and we wouldn't know how to program it. > The human genome -- the information required to build a human from a single, undifferentiated eukariotic cell -- is about 1GB. The instructions to build a mammal are written in very dense code, and the program is extremely flexible. Only small patches to the human genome are required to build a cow or a dog rather than a human. Bigger patches result in a frog or a snake. We don't have any idea how to make a description of such a complex machine that is both dense and flexible. > New design principles and new linguistic support are needed. I will address this issue and show some ideas that can perhaps get us to the next phase of engineering design. > Gerald Sussman Massachusetts Institute of Technology
dwallin 3y ago
That’s an arbitrary line to make though. The human brain starts with a general architecture that has been gone through billions and trillions of generations of evolutionary training, before being fine tuned in a single individual over decades, and then you did a little bit of fine tuning and few shot at the end and claim that is comparable to the entire training of a LLM from scratch? Not to mention the many more orders of magnitude of neurons that a human brain has. I could equally argue that an LLM takes zero training, since we have ALREADY trained models and I can just copy the model and run it and get a new “brain” performing a task, instead of taking decades of training to get there. Even your statement about programming skills is debatable, it depends on how you measure programming skill. They certainly are faster at it, and they know more computer languages than most people have even heard of. In fact, human programming strength seems to be more about general logic and planning skills over programming-specific skills, both things where the bulk of training happened evolutionarily and more generally over the course of a life. The truth is, the two are not directly comparable. They are completely different architectures, at completely different scales, with entirely different strengths and weaknesses.

aledalgrande 3y ago

Also multi modal / multi topic learning benefits the brain too. Range by David Epstein is an interesting read

hannasm 3y ago

The training process is finding local minimums based around an initialization vector of random numbers. 1000s of years of evolution probably mean you were initialized better than a baby AI using a pseudo random number generator.

ddalex 3y ago
Watching my kid learn to speak made it clear to me that we are pre-wired for the acquisition of language - we're not building the structures from scratch, we are learning to put words into the empty spaces. Probably natural evolution at best - no speech, lower chance of gene propagation

Taek 3y ago

I feel like there should be a basic 'curriculum' that gets passed to all foundational LLMs that teaches them the basics of language. Maybe 100 million files where the first 10 million are all first grade reading level, the second 10 million are all second grade reading level, etc. Ideally this includes a bunch of text books. That should give the LLM time to grok language before it starts training on more difficult texts.

visarga 3y ago

> Your brain is baked with millions of years of evolution Exactly, and not only that, we are agents from birth, so we enjoy the 5 E's: "Embodied, Embedded, Enacted and Extended into the Environment" LLM's can't even write a script to see if it works, they got no feedback and can't make any meaningful choice with consequences. They are trained with "teacher forcing" and self-supervised methods, no deviation allowed. On the other hand we got THE WORLD which is infinitely more rich than any simulation, and human society which is made of super-GPT agents, and search based access to information. Remember, most LLMs work closed book and train on a dry, static dataset. Don't directly compare them with humans. Humans can't write great code top-down without computers either. We are trial-and-feedback monkeys, without feedback we're just no good.

manojlds 3y ago

And the point of aeroplanes wasn't to mimic birds but to fly.