3 ms·

Your argument only makes sense if you consider GPT3 equivalent to the human brain - yep the brain has more neurons - but it's also doing a heck of a lot more.

Your argument only makes sense if you consider GPT3 equivalent to the human brain - yep the brain has more neurons - but it's also doing a heck of a lot more. Also each neuron can hold much more state than a transistor. For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. In addition the neural connections themselves have properties. > because nature didn't invent software. Eh? How do you explain the fact that brains can learn and aren't a fixed input, output engine? Neural nets are software - just ones programmed by trial and error. Similarly for the brain.

lostmsu 3y ago
> but it's also doing a heck of a lot more What are you referring to? > Also each neuron can hold much more state than a transistor. But can it hold much more state than 8 transistors? 16 transistors? 32 transistors? > For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. But they can't fire 3B times a second. Transistors though can accumulate a number and fire when it reaches a threshold much faster (although that is not really used in transformers). There are differences, but it is unclear how to quantify them. At least I don't immediately see how that feature is a definite advantage, and not simply an implementation detail. > In addition the neural connections themselves have properties. This is very technically correct, which is great because in this form it shows to be a non-argument. Individual transistors also have properties. Even molecules do. Which of those properties are essential for the learning process is the important bit, and it is quite possible that none are. > Eh? How do you explain the fact that brains can learn and aren't a fixed input, output engine? Eh? Brains learn by physically growing parts of neurons. That growing can not be moved around to another set of neurons. That property is essential to be deemed "software".
- DrScientist 3y ago
  GPT is doing a single task - my brain as I type is thinking about the content, controlling the fingers, scanning the visual input for danger, processing the continuous signal for other senses, smell, touch, sound, and also wondering what's for dinner, moving stuff from short term to long term memory, and under going continual training as well as generating this text.
- DrScientist 3y ago
  > Eh? Brains learn by physically growing parts of neurons. Hmm - that's quite a slow process - sure it happens - but I think you'll find that brains can learn on a faster timescale than the growth of connections would enable. > That growing can not be moved around to another set of neurons, That property is essential to be deemed "software". Not sure the ability to easily copy is the definition of software. Connections can be activated or attenuated without growth - that's the equivalent of learning your weights. Isn't that cumulative set of attentuations/activations software - ie you remap the input/output engine dynamically?
- DrScientist 3y ago
  > But they can't fire 3B times a second. Fair point - but I'd argue the power comes for the combinatoric complexity, not the speed of operation. So those timing effects can also combine with the simple connections - so if a neuron has 2 connections - the output can be controlled by not just the last signals for the 2 connections, but the relative timing of each of them. Suddenly your input space you are operating over is much much bigger - you've got 2 inputs and hundreds of ways the timing could be adjusted. Massively bigger. The fact that the whole thing doesn't operate in a big synchronous cycle is a huge jump in possible states.

lostmsu 3y ago

> but it's also doing a heck of a lot more What are you referring to? > Also each neuron can hold much more state than a transistor. But can it hold much more state than 8 transistors? 16 transistors? 32 transistors? > For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. But they can't fire 3B times a second. Transistors though can accumulate a number and fire when it reaches a threshold much faster (although that is not really used in transformers). There are differences, but it is unclear how to quantify them. At least I don't immediately see how that feature is a definite advantage, and not simply an implementation detail. > In addition the neural connections themselves have properties. This is very technically correct, which is great because in this form it shows to be a non-argument. Individual transistors also have properties. Even molecules do. Which of those properties are essential for the learning process is the important bit, and it is quite possible that none are. > Eh? How do you explain the fact that brains can learn and aren't a fixed input, output engine? Eh? Brains learn by physically growing parts of neurons. That growing can not be moved around to another set of neurons. That property is essential to be deemed "software".

DrScientist 3y ago
GPT is doing a single task - my brain as I type is thinking about the content, controlling the fingers, scanning the visual input for danger, processing the continuous signal for other senses, smell, touch, sound, and also wondering what's for dinner, moving stuff from short term to long term memory, and under going continual training as well as generating this text.
DrScientist 3y ago
> Eh? Brains learn by physically growing parts of neurons. Hmm - that's quite a slow process - sure it happens - but I think you'll find that brains can learn on a faster timescale than the growth of connections would enable. > That growing can not be moved around to another set of neurons, That property is essential to be deemed "software". Not sure the ability to easily copy is the definition of software. Connections can be activated or attenuated without growth - that's the equivalent of learning your weights. Isn't that cumulative set of attentuations/activations software - ie you remap the input/output engine dynamically?
DrScientist 3y ago
> But they can't fire 3B times a second. Fair point - but I'd argue the power comes for the combinatoric complexity, not the speed of operation. So those timing effects can also combine with the simple connections - so if a neuron has 2 connections - the output can be controlled by not just the last signals for the 2 connections, but the relative timing of each of them. Suddenly your input space you are operating over is much much bigger - you've got 2 inputs and hundreds of ways the timing could be adjusted. Massively bigger. The fact that the whole thing doesn't operate in a big synchronous cycle is a huge jump in possible states.