4 ms·

That's a stupid analogy because you're comparing a brainprocess to a full animal. Only a small part of that 60kcal is used for learning, and for that same 60 k

That's a stupid analogy because you're comparing a brainprocess to a full animal. Only a small part of that 60kcal is used for learning, and for that same 60 kcal you get an actual physical being that is able to procreate, eat, do things and fend for and maintain itself. Also you cannot compare neuron firing rates with clockspeed. Afaik each neuron in a ml-model can have code that takes several clock cycles to complete. Also an neuron in ml is just a weighted value, a biological neuron does much more than that. For example neurons communicate using neuro transmitters as well as using voltage potentials. The actual date rate of biological neurons is therfore much higher and complex. Basically your analogy is false because your napkin-math basically forgets that the rat is an actual biological rat and not something as neatly defined as a computer chip

ben_w 2y ago
> Also an neuron in ml is just a weighted value, a biological neuron does much more than that. For example neurons communicate using neuro transmitters as well as using voltage potentials. The actual date rate of biological neurons is therfore much higher and complex. The conclusion does not follow from the premise. The observed maximum rate of the inter-neuron communication is important, the mechanism is not. > Also you cannot compare neuron firing rates with clockspeed. Afaik each neuron in a ml-model can have code that takes several clock cycles to complete. Depends how you're doing it. Jupyter notebook? Python in general? Sure. A100s etc., not so much — those are specialist systems designed for this task: """1024 dense FP16/FP32 FMA operations per clock""" - https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf https://images.nvidia.com/aem-dam/en-zz/Solutions/data-cente... "FMA" meaning "fused multiply-add". It's the unit that matters for synapse-equivalents. (Even that doesn't mean they're perfect fits: IMO a "perfect fit" would likely be using transistors as analog rather than digital elements, and then you get to run them at the native transistor speed of ~100 GHz or so and don't worry too much about how many bits you need to represent the now-analog weights and biases, but that's one of those things which is easy to say from a comfortable armchair and very hard to turn into silicon). > Basically your analogy is false because your napkin-math basically forgets that the rat is an actual biological rat and not something as neatly defined as a computer chip Any of those biological functions that don't correspond to intelligence, make the comparison more extreme in favour of the computer. This is, after all, a question of their mere intelligence, not how well LLMs (or indeed any AI) do or don't function as von Neumann replicators, which is where things like "procreate, eat, do things and fend for and maintain itself" would actually matter.
- imtringued 2y ago
  You're so deep into this nonsense I don't think anything I could possibly say to you would change your mind, so I'll try something different. Have you thought about stepping back from all of this for a few days and notice that you are wasting your time with these arguments? It doesn't matter how fast you can calculate a dot product or evaluate an activation function if the weights in question do not change. NNs as of right now are the equivalent of a brain scan. You can simulate how that brain scan would answer a question, but the moment you close the Q and A session, you will have to start from scratch. Making higher resolution brain scans may help you get more precise answers to more questions, but it will never change the questions that it can answer after you have made the brain scan.
  - ben_w 2y ago
    > Have you thought about stepping back from all of this for a few days and notice that you are wasting your time with these arguments? Num fecisti? > It doesn't matter how fast you can calculate a dot product or evaluate an activation function if the weights in question do not change. That's a deliberate choice, not a fundamental requirement. Models get frozen in order to become a product someone can put a version number on and ship, not because they must be, as demonstrated both by fine-tuning and by the initial training process — both of which update the weights. > NNs as of right now are the equivalent of a brain scan. First: see above. Second: even if it were, so what? Look at the context I'm replying to, this is about energy efficiency — and applies just fine even when calculated for training the whole thing from scratch. To put it another way: how long would it take a mouse to read 13 trillion tokens? The energy cost of silicon vs. biology is lower than people realise, because people read the power consumption without considering that the speed of silicon is much higher: at the lowest level, the speed of silicon computation literally — not metaphorically, really literally — outpaces biological computation by the same magnitude to which jogging outpaces continental drift.
- Jensson 2y ago
  > "FMA" meaning "fused multiply-add". It's the unit that matters for synapse-equivalents. Neurons do so much more than a single math operation. A single cell can act as an intelligent little animal on its own, they are nothing like a neural network "neuron". And note that all neurons act in parallel, so they are billions times more parallel than GPU's even if the operations would be the same.

ben_w 2y ago

> Also an neuron in ml is just a weighted value, a biological neuron does much more than that. For example neurons communicate using neuro transmitters as well as using voltage potentials. The actual date rate of biological neurons is therfore much higher and complex. The conclusion does not follow from the premise. The observed maximum rate of the inter-neuron communication is important, the mechanism is not. > Also you cannot compare neuron firing rates with clockspeed. Afaik each neuron in a ml-model can have code that takes several clock cycles to complete. Depends how you're doing it. Jupyter notebook? Python in general? Sure. A100s etc., not so much — those are specialist systems designed for this task: """1024 dense FP16/FP32 FMA operations per clock""" - https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf https://images.nvidia.com/aem-dam/en-zz/Solutions/data-cente... "FMA" meaning "fused multiply-add". It's the unit that matters for synapse-equivalents. (Even that doesn't mean they're perfect fits: IMO a "perfect fit" would likely be using transistors as analog rather than digital elements, and then you get to run them at the native transistor speed of ~100 GHz or so and don't worry too much about how many bits you need to represent the now-analog weights and biases, but that's one of those things which is easy to say from a comfortable armchair and very hard to turn into silicon). > Basically your analogy is false because your napkin-math basically forgets that the rat is an actual biological rat and not something as neatly defined as a computer chip Any of those biological functions that don't correspond to intelligence, make the comparison more extreme in favour of the computer. This is, after all, a question of their mere intelligence, not how well LLMs (or indeed any AI) do or don't function as von Neumann replicators, which is where things like "procreate, eat, do things and fend for and maintain itself" would actually matter.

imtringued 2y ago
You're so deep into this nonsense I don't think anything I could possibly say to you would change your mind, so I'll try something different. Have you thought about stepping back from all of this for a few days and notice that you are wasting your time with these arguments? It doesn't matter how fast you can calculate a dot product or evaluate an activation function if the weights in question do not change. NNs as of right now are the equivalent of a brain scan. You can simulate how that brain scan would answer a question, but the moment you close the Q and A session, you will have to start from scratch. Making higher resolution brain scans may help you get more precise answers to more questions, but it will never change the questions that it can answer after you have made the brain scan.
- ben_w 2y ago
  > Have you thought about stepping back from all of this for a few days and notice that you are wasting your time with these arguments? Num fecisti? > It doesn't matter how fast you can calculate a dot product or evaluate an activation function if the weights in question do not change. That's a deliberate choice, not a fundamental requirement. Models get frozen in order to become a product someone can put a version number on and ship, not because they must be, as demonstrated both by fine-tuning and by the initial training process — both of which update the weights. > NNs as of right now are the equivalent of a brain scan. First: see above. Second: even if it were, so what? Look at the context I'm replying to, this is about energy efficiency — and applies just fine even when calculated for training the whole thing from scratch. To put it another way: how long would it take a mouse to read 13 trillion tokens? The energy cost of silicon vs. biology is lower than people realise, because people read the power consumption without considering that the speed of silicon is much higher: at the lowest level, the speed of silicon computation literally — not metaphorically, really literally — outpaces biological computation by the same magnitude to which jogging outpaces continental drift.
Jensson 2y ago
> "FMA" meaning "fused multiply-add". It's the unit that matters for synapse-equivalents. Neurons do so much more than a single math operation. A single cell can act as an intelligent little animal on its own, they are nothing like a neural network "neuron". And note that all neurons act in parallel, so they are billions times more parallel than GPU's even if the operations would be the same.