4 ms·

> Considering this, it seems perfectly possible that a model like GPT-4 is just a hair's breadth away from vastly superhuman performance. Except that structura

> Considering this, it seems perfectly possible that a model like GPT-4 is just a hair's breadth away from vastly superhuman performance. Except that structurally the brain is clearly has vastly more capacity than the GPT-4 model. So sure one brain doesn't look that much different to the other - and it's in the details of the learning, wiring. But the brain, looks vastly different from a GPT-4 model in terms of capacity - with trillions of connections - with each connection and internal state being more subtle as well. > vastly superhuman performance In terms of specific tasks computers ( whether you write the program explicitly or it's learnt by tweaking params in a network ) have been there for decades. So the question is really around which tasks can you apply computers successful to. Neural nets are allowing programs to be written that weren't possible be hand. I find it amusing that people worry about ChatGPT etc al putting programmers out of a job, when it already has in the sense that CHapGPT is a program that was built by another program already.

twobitshifter 3y ago
I’ve heard that the human brain may have more connections but with AI we’ve made something that is more efficient. If we make it a hardware issue, if provided with the brains’ capacity, what would AI look like?
lostmsu 3y ago
> brain is clearly has vastly more capacity I would not be so sure about "clearly" bit. Brain has 100B neurons x 1000 connections. GPT3 has 175B connections, but to implement operations used in those connections Nvidia H100 uses about 5M transistors, which in human brain would have to be copied to each connection because nature didn't invent software. (assumes one needs one full CUDA core to implement necessary ops, and that H100's 80B transistors are evenly divided between its 16k cores)
- DrScientist 3y ago
  Your argument only makes sense if you consider GPT3 equivalent to the human brain - yep the brain has more neurons - but it's also doing a heck of a lot more. Also each neuron can hold much more state than a transistor. For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. In addition the neural connections themselves have properties. > because nature didn't invent software. Eh? How do you explain the fact that brains can learn and aren't a fixed input, output engine? Neural nets are software - just ones programmed by trial and error. Similarly for the brain.
  - lostmsu 3y ago
    > but it's also doing a heck of a lot more What are you referring to? > Also each neuron can hold much more state than a transistor. But can it hold much more state than 8 transistors? 16 transistors? 32 transistors? > For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. But they can't fire 3B times a second. Transistors though can accumulate a number and fire when it reaches a threshold much faster (although that is not really used in transformers). There are differences, but it is unclear how to quantify them. At least I don't immediately see how that feature is a definite advantage, and not simply an implementation detail. > In addition the neural connections themselves have properties. This is very technically correct, which is great because in this form it shows to be a non-argument. Individual transistors also have properties. Even molecules do. Which of those properties are essential for the learning process is the important bit, and it is quite possible that none are. > Eh? How do you explain the fact that brains can learn and aren't a fixed input, output engine? Eh? Brains learn by physically growing parts of neurons. That growing can not be moved around to another set of neurons. That property is essential to be deemed "software".
    - DrScientist 3y ago
      GPT is doing a single task - my brain as I type is thinking about the content, controlling the fingers, scanning the visual input for danger, processing the continuous signal for other senses, smell, touch, sound, and also wondering what's for dinner, moving stuff from short term to long term memory, and under going continual training as well as generating this text.
    - DrScientist 3y ago
      > Eh? Brains learn by physically growing parts of neurons. Hmm - that's quite a slow process - sure it happens - but I think you'll find that brains can learn on a faster timescale than the growth of connections would enable. > That growing can not be moved around to another set of neurons, That property is essential to be deemed "software". Not sure the ability to easily copy is the definition of software. Connections can be activated or attenuated without growth - that's the equivalent of learning your weights. Isn't that cumulative set of attentuations/activations software - ie you remap the input/output engine dynamically?

twobitshifter 3y ago

I’ve heard that the human brain may have more connections but with AI we’ve made something that is more efficient. If we make it a hardware issue, if provided with the brains’ capacity, what would AI look like?

lostmsu 3y ago

> brain is clearly has vastly more capacity I would not be so sure about "clearly" bit. Brain has 100B neurons x 1000 connections. GPT3 has 175B connections, but to implement operations used in those connections Nvidia H100 uses about 5M transistors, which in human brain would have to be copied to each connection because nature didn't invent software. (assumes one needs one full CUDA core to implement necessary ops, and that H100's 80B transistors are evenly divided between its 16k cores)

DrScientist 3y ago
Your argument only makes sense if you consider GPT3 equivalent to the human brain - yep the brain has more neurons - but it's also doing a heck of a lot more. Also each neuron can hold much more state than a transistor. For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. In addition the neural connections themselves have properties. > because nature didn't invent software. Eh? How do you explain the fact that brains can learn and aren't a fixed input, output engine? Neural nets are software - just ones programmed by trial and error. Similarly for the brain.
- lostmsu 3y ago
  > but it's also doing a heck of a lot more What are you referring to? > Also each neuron can hold much more state than a transistor. But can it hold much more state than 8 transistors? 16 transistors? 32 transistors? > For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. But they can't fire 3B times a second. Transistors though can accumulate a number and fire when it reaches a threshold much faster (although that is not really used in transformers). There are differences, but it is unclear how to quantify them. At least I don't immediately see how that feature is a definite advantage, and not simply an implementation detail. > In addition the neural connections themselves have properties. This is very technically correct, which is great because in this form it shows to be a non-argument. Individual transistors also have properties. Even molecules do. Which of those properties are essential for the learning process is the important bit, and it is quite possible that none are. > Eh? How do you explain the fact that brains can learn and aren't a fixed input, output engine? Eh? Brains learn by physically growing parts of neurons. That growing can not be moved around to another set of neurons. That property is essential to be deemed "software".
  - DrScientist 3y ago
    GPT is doing a single task - my brain as I type is thinking about the content, controlling the fingers, scanning the visual input for danger, processing the continuous signal for other senses, smell, touch, sound, and also wondering what's for dinner, moving stuff from short term to long term memory, and under going continual training as well as generating this text.
  - DrScientist 3y ago
    > Eh? Brains learn by physically growing parts of neurons. Hmm - that's quite a slow process - sure it happens - but I think you'll find that brains can learn on a faster timescale than the growth of connections would enable. > That growing can not be moved around to another set of neurons, That property is essential to be deemed "software". Not sure the ability to easily copy is the definition of software. Connections can be activated or attenuated without growth - that's the equivalent of learning your weights. Isn't that cumulative set of attentuations/activations software - ie you remap the input/output engine dynamically?