4 ms·

Building Transformers from Neurons and Astrocytes

34 points by sebg 4y ago

mr_usb_porto 4y ago
This is interesting and important in understanding the modern ai revolutions connection with biologically plausible neural computation. Very cool stuff!
dontreact 4y ago
It seems like almost no matter what the AI folks come up with, there will be some potential way of interpreting neural phenomena as loosely mapping onto it.
- puttycat 4y ago
  This. Thank you
- mjburgess 4y ago
  They came up with the function, f(x; W, W', ...) = ...max(0, W''max(0, W'max(0, Wx))) And called it a "neural network" When your opening gambit is that ridiculous, you really can't expect them to improve. We're not dealing with people interested in science in the ordinary sense.
mjburgess 4y ago
This reads like something a retired professor of physics used to write: "how differential calculus solves the need for talking therapy". Ie., a slightly cooky misapplication of the tools of physics to some problem more complex than the author realises. I think we're in an age of pseudoscience where rather than wear lab coats, pseudoscientists "wear statistics" in a self-deluding attempt to ape science. However, I think this paper falls closer on the "cooky speculation" side. It does, nevertheless, participate in the culture of pseudoscience around these issues. It would be great if the paper owned up to its radical speculative nature, and didnt brush aside it's lack of parameterisation on time (which is pretty fatal to taking this seriously). Then it would be a pretty good model for what "radical considered speculation" looks like which we could contrast with the genuine pseudoscience of many papers in this area.
- dr_dshiv 4y ago
  Labeling a scientific paper as pseudoscience is an intense insult. Why not explain the point that you disagree with and explain why you disagree? That would be valuable for the conversation.
  - mjburgess 4y ago
    Well my comment is that this paper isnt pseudoscience, but falls into the category of "cooky speculative science", which I'm generally reasonably OK with. But I would prefer it makes that point clearer. It is certainly part of the "pseudoscience culture" wherein many papers are just "not even wrong". The reason it's, at best, speculative is that we have no idea at what layer the brain-body-env performs computations. If we're interested in a dynamical model of the brain-body-env, then we want a series of partial differential equations which are parameterised by space and time. Just as we would describe the electrical field of a digital machine. However since we know that +/- voltage states of the machines we create are the relevant discrete states to give a computational description, we can also give a non-dynamical account of a digital computer (ie., we can say it realises the sequence: 0001, 1001, 0101, etc. What this paper does it take arbitrary aspects of arbitrary parts of the brain, and tries to marry them to some existing computational framework. If an alien were to analyse a CPU that way, for example, it could easily conclude that it's the heat of air particles warmed by the CPU which "did computational work". Since they do computational work, but none relevant to the digital computer we are using. This paper treads the line, some others cross it, in breathlessly claiming their series of wild, indefensible, and pretty clearly wrong assumptions which motivate their fashionable computational framework (ie., NNs, etc.) show that their press-released new projects (eg., GPT, etc.) are like brains somehow.
    - dr_dshiv 4y ago
      > What this paper does it take arbitrary aspects of arbitrary parts of the brain, and tries to marry them to some existing computational framework. Can you try describing what they specifically claim that you find arbitrary? Like, what are the “clearly wrong assumptions?”
leokoz8 4y ago
Hi, one of the authors here (Leo). To clear up some confusion: 1) The paper proves an architectural equivalence between two models: neuron-astrocyte networks and Transformers. This is different from studies of representational similarity, which look at Transformer "responses" and compare them to brain responses using regression techniques. 2) The paper provides testable hypotheses for what astrocytes are doing. We can either succeed or fail at rejecting those hypotheses after comparing to real brain data. That is future work. Happy to answer questions. - Leo

34 points by sebg 4y ago

This is interesting and important in understanding the modern ai revolutions connection with biologically plausible neural computation. Very cool stuff!

dontreact 4y ago

It seems like almost no matter what the AI folks come up with, there will be some potential way of interpreting neural phenomena as loosely mapping onto it.

puttycat 4y ago
This. Thank you
mjburgess 4y ago
They came up with the function, f(x; W, W', ...) = ...max(0, W''max(0, W'max(0, Wx))) And called it a "neural network" When your opening gambit is that ridiculous, you really can't expect them to improve. We're not dealing with people interested in science in the ordinary sense.

mjburgess 4y ago

This reads like something a retired professor of physics used to write: "how differential calculus solves the need for talking therapy". Ie., a slightly cooky misapplication of the tools of physics to some problem more complex than the author realises. I think we're in an age of pseudoscience where rather than wear lab coats, pseudoscientists "wear statistics" in a self-deluding attempt to ape science. However, I think this paper falls closer on the "cooky speculation" side. It does, nevertheless, participate in the culture of pseudoscience around these issues. It would be great if the paper owned up to its radical speculative nature, and didnt brush aside it's lack of parameterisation on time (which is pretty fatal to taking this seriously). Then it would be a pretty good model for what "radical considered speculation" looks like which we could contrast with the genuine pseudoscience of many papers in this area.

dr_dshiv 4y ago
Labeling a scientific paper as pseudoscience is an intense insult. Why not explain the point that you disagree with and explain why you disagree? That would be valuable for the conversation.
- mjburgess 4y ago
  Well my comment is that this paper isnt pseudoscience, but falls into the category of "cooky speculative science", which I'm generally reasonably OK with. But I would prefer it makes that point clearer. It is certainly part of the "pseudoscience culture" wherein many papers are just "not even wrong". The reason it's, at best, speculative is that we have no idea at what layer the brain-body-env performs computations. If we're interested in a dynamical model of the brain-body-env, then we want a series of partial differential equations which are parameterised by space and time. Just as we would describe the electrical field of a digital machine. However since we know that +/- voltage states of the machines we create are the relevant discrete states to give a computational description, we can also give a non-dynamical account of a digital computer (ie., we can say it realises the sequence: 0001, 1001, 0101, etc. What this paper does it take arbitrary aspects of arbitrary parts of the brain, and tries to marry them to some existing computational framework. If an alien were to analyse a CPU that way, for example, it could easily conclude that it's the heat of air particles warmed by the CPU which "did computational work". Since they do computational work, but none relevant to the digital computer we are using. This paper treads the line, some others cross it, in breathlessly claiming their series of wild, indefensible, and pretty clearly wrong assumptions which motivate their fashionable computational framework (ie., NNs, etc.) show that their press-released new projects (eg., GPT, etc.) are like brains somehow.
  - dr_dshiv 4y ago
    > What this paper does it take arbitrary aspects of arbitrary parts of the brain, and tries to marry them to some existing computational framework. Can you try describing what they specifically claim that you find arbitrary? Like, what are the “clearly wrong assumptions?”

leokoz8 4y ago

Hi, one of the authors here (Leo). To clear up some confusion: 1) The paper proves an architectural equivalence between two models: neuron-astrocyte networks and Transformers. This is different from studies of representational similarity, which look at Transformer "responses" and compare them to brain responses using regression techniques. 2) The paper provides testable hypotheses for what astrocytes are doing. We can either succeed or fail at rejecting those hypotheses after comparing to real brain data. That is future work. Happy to answer questions. - Leo