4 ms·

Does the human brain use transformers?

exe34 3y ago
No, but they can both implement a language virtual machine which appears to be able to produce intelligent behaviour with unknown bounds.
mirekrusin 3y ago
Yes, through ie. services like openai's chatgpt.
dartos 3y ago
What?
__loam 3y ago
Anyone telling you it does is a fraud.
- utopcell 3y ago
  Here [1] are some "frauds" from Stanford University, Oxford University and University College London telling you exactly that. From their abstract: ``One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version.`` [1] https://arxiv.org/abs/2112.04035 https://arxiv.org/abs/2112.04035
  - __loam 3y ago
    Making the claim that transformers are a good candidate model for certain neural pathways is a pretty different claim than saying the brain is literally using transformers.
    - Rastonbury 3y ago
      In an alternative timeline, humans were being hooked up and used as GPUs in The Matrix
willy_k 3y ago
I’m assuming you are asking if the brain uses transformer-like structures or otherwise exhibits similar behavior. I don’t know, but it does share some processes with simpler ML ideas, and I’d be very interested to see if it uses anything resembling a transformer.
- patcon 3y ago
  Forward-forward algorithm is more like the brain. As I understand, backpropagation transformers require storing data, doing calculations on that aggregate, and sending it back through, which no neural structures can do anything like. https://medium.com/@Mosbeh_Barhoumi/forward-forward-algorithm-ac24d0d9ffd https://medium.com/@Mosbeh_Barhoumi/forward-forward-algorith...
nostrademons 3y ago
Yes. I still remember Optimus Prime and Bumblebee from my youth.

No, but they can both implement a language virtual machine which appears to be able to produce intelligent behaviour with unknown bounds.

mirekrusin 3y ago

Yes, through ie. services like openai's chatgpt.

dartos 3y ago

What?

__loam 3y ago

Anyone telling you it does is a fraud.

utopcell 3y ago
Here [1] are some "frauds" from Stanford University, Oxford University and University College London telling you exactly that. From their abstract: ``One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version.`` [1] https://arxiv.org/abs/2112.04035 https://arxiv.org/abs/2112.04035
- __loam 3y ago
  Making the claim that transformers are a good candidate model for certain neural pathways is a pretty different claim than saying the brain is literally using transformers.
  - Rastonbury 3y ago
    In an alternative timeline, humans were being hooked up and used as GPUs in The Matrix

willy_k 3y ago

I’m assuming you are asking if the brain uses transformer-like structures or otherwise exhibits similar behavior. I don’t know, but it does share some processes with simpler ML ideas, and I’d be very interested to see if it uses anything resembling a transformer.

patcon 3y ago
Forward-forward algorithm is more like the brain. As I understand, backpropagation transformers require storing data, doing calculations on that aggregate, and sending it back through, which no neural structures can do anything like. https://medium.com/@Mosbeh_Barhoumi/forward-forward-algorithm-ac24d0d9ffd https://medium.com/@Mosbeh_Barhoumi/forward-forward-algorith...

nostrademons 3y ago

Yes. I still remember Optimus Prime and Bumblebee from my youth.