5 ms·

> we're not magic either We pretty much are compared to present-day neural architectures. How many simulated neurons and synapses are in the largest architectu

> we're not magic either We pretty much are compared to present-day neural architectures. How many simulated neurons and synapses are in the largest architectures, and how do those numbers compare to humans?

dartos 3y ago
It’s a non starter to assume that virtual “synapses and neurons” behave like ours do. We barely understand how ours works. Also, modern LLMs built on the transformers architecture no longer use the neuron-inspired perceptron style topology for most of their compute. I’ve heard that spiking NNs are supposed to mimic organic brains more closely, but I haven’t read into them much yet.
- samus 3y ago
  The attention mechanism is in practice implemented using three linear layers. The matrix multiplication to average the output and to implement the masking is the only non-neuronal part of that computation, but it can be seen as an activation function. Usually, linear perceptrons and ReLUs or GeLUs are used. Due to the enormous compute requirements to evaluate models of interesting size, other types of neuronal networks and activation functions have received very little attention (pun intended) so far.
  - dartos 3y ago
    Using ReLU instead of sigmoid is a significant departure with regards to how closely it models actual neurons. Using non fully connected layers is as well. Our brains likely aren’t fully connected, but the connections that matter are made stronger through living life and learning. If you squint, it’s kind of like training a dense series of linear layers, but that’s not what we’re doing anymore (for the better) Comparing NNs to organic brains is an apples to oranges comparison, is what I’m saying.
    - deleted 3y ago
      [deleted]
    - samus 3y ago
      I agree that the biggest difference is the missing ability of an artificial neuronal network to adapt.
      - ben_w 3y ago
        Lack of adaption is mainly a feature, we choose not to train them in real-time and instead make available fixed models with repeatable behaviour. We could, if we wanted to, update the model weights continuously in response to feedback. I think the biggest difference is that they need far more examples than we need, to learn anything.
ben_w 3y ago
Unknown for the actual largest due to secrecy; 1% for the largest public models… but also organic ones are definitely a bit different from digital ones, and the jury is still out if those differences matter and if so by how much. The comparison would therefore be with a mid-sized rodent, horse, or raven rather than a human. (But even that's misleading, because the LLM doesn't have to use tokens to represent "contract left supracoracoideus" and "lay egg"). Edit: also, I've not heard much suggestion that anyone knows how certain genes do things like giving humans the inherent capability to recognise and create smiles or other similar reflexes, so we don't really know how much of our brains a pre-trained by evolution; furthermore, I think organic life is more sample-efficient for learning things than any AI so far.
- samus 3y ago
  Tokens aren't a necessary differentiator here. There is no fundamental technical reason why tokenization is used, it just has certain practical advantages. And the distinction almost disappears when we look at multimodal transformers, which process images, audio, and video broken apart into sequences of blocks of binary data.
  - ben_w 3y ago
    There's no reason for any specific tokenisation, but the Transformer always has some tokenisation. Tokens are allowed to be blocks of pixels, for example. No reason we couldn't have a token be a specific muscle or sensory nerve. What I'm saying is that Large Language Models don't have a body, so no nerves and muscles to have to be represented within them; conversely, organic life does have those things and thus organic brains must spend some of their complexity on those things. This means they have the possibility to equal us for language even with no capacity for vision, walking, tying shoelaces, or playing catch.

dartos 3y ago

It’s a non starter to assume that virtual “synapses and neurons” behave like ours do. We barely understand how ours works. Also, modern LLMs built on the transformers architecture no longer use the neuron-inspired perceptron style topology for most of their compute. I’ve heard that spiking NNs are supposed to mimic organic brains more closely, but I haven’t read into them much yet.

samus 3y ago
The attention mechanism is in practice implemented using three linear layers. The matrix multiplication to average the output and to implement the masking is the only non-neuronal part of that computation, but it can be seen as an activation function. Usually, linear perceptrons and ReLUs or GeLUs are used. Due to the enormous compute requirements to evaluate models of interesting size, other types of neuronal networks and activation functions have received very little attention (pun intended) so far.
- dartos 3y ago
  Using ReLU instead of sigmoid is a significant departure with regards to how closely it models actual neurons. Using non fully connected layers is as well. Our brains likely aren’t fully connected, but the connections that matter are made stronger through living life and learning. If you squint, it’s kind of like training a dense series of linear layers, but that’s not what we’re doing anymore (for the better) Comparing NNs to organic brains is an apples to oranges comparison, is what I’m saying.
  - deleted 3y ago
    [deleted]
  - samus 3y ago
    I agree that the biggest difference is the missing ability of an artificial neuronal network to adapt.
    - ben_w 3y ago
      Lack of adaption is mainly a feature, we choose not to train them in real-time and instead make available fixed models with repeatable behaviour. We could, if we wanted to, update the model weights continuously in response to feedback. I think the biggest difference is that they need far more examples than we need, to learn anything.

ben_w 3y ago

Unknown for the actual largest due to secrecy; 1% for the largest public models… but also organic ones are definitely a bit different from digital ones, and the jury is still out if those differences matter and if so by how much. The comparison would therefore be with a mid-sized rodent, horse, or raven rather than a human. (But even that's misleading, because the LLM doesn't have to use tokens to represent "contract left supracoracoideus" and "lay egg"). Edit: also, I've not heard much suggestion that anyone knows how certain genes do things like giving humans the inherent capability to recognise and create smiles or other similar reflexes, so we don't really know how much of our brains a pre-trained by evolution; furthermore, I think organic life is more sample-efficient for learning things than any AI so far.

samus 3y ago
Tokens aren't a necessary differentiator here. There is no fundamental technical reason why tokenization is used, it just has certain practical advantages. And the distinction almost disappears when we look at multimodal transformers, which process images, audio, and video broken apart into sequences of blocks of binary data.
- ben_w 3y ago
  There's no reason for any specific tokenisation, but the Transformer always has some tokenisation. Tokens are allowed to be blocks of pixels, for example. No reason we couldn't have a token be a specific muscle or sensory nerve. What I'm saying is that Large Language Models don't have a body, so no nerves and muscles to have to be represented within them; conversely, organic life does have those things and thus organic brains must spend some of their complexity on those things. This means they have the possibility to equal us for language even with no capacity for vision, walking, tying shoelaces, or playing catch.