3 ms·

I can't see a pathway for these things to be able to learn from experience in any meaningful way - the energy budget seems to prohibit it. Artificial neural ne

I can't see a pathway for these things to be able to learn from experience in any meaningful way - the energy budget seems to prohibit it. Artificial neural networks are already many orders of magnitude more energy intensive than natural neural networks. Despite this, large nervous systems are very energy intensive as well. For example in humans 20% of the energy budget goes to the brain which is 5% of the body weight. So I refrer to LLMs as language extrusion confabulation machines. Language extrusion was a term I heard the linguist Emily Bender use. Confabulation because my observation is that talking to an LLM is very much similar to my experience of interacting with Korsakov syndrome patients some years ago. I look forward to the hype settling down to see what we end up with.

rcxdude 18d ago
> Artificial neural networks are already many orders of magnitude more energy intensive than natural neural networks. What metric are you using to compare? By most counts, the energy budget of an instance of an LLM in a datacenter is lower than the energy a person uses. Of course, if you count energy per neuron connections vs weights then you'll likely get a quite different number. But then again LLMs do a lot of things with far fewer weights than the brain does neuron connections, even if you only count neurons in some parts of the brain. And of course you can point to capabilities that the brain has but LLMs lack, but on the whole it feels like it's pretty difficult to make a useful like-for-like comparison here.
- singingfish 17d ago
  I can't make sense of your comment. Firstly because of the obvious massive over-build of GPU infrastructure the AI companies are engaging in. Secondly because the instance of the LLM in the data centre that users interact with is only a small part of the story. The training phase is clearly prohibitively expensive, thus the fact that these things have no way to learn from experience except by smoke and mirrors. Also your comment feels like the classic climate denial discourse - say something a bit complicated and a bit difficult to follow that looks at a very small out of context part of the story to cast doubt.
- singingfish 17d ago
  Maybe I shouldn't have been pissed of by your commient. In which case - the excess costs of the training, output, feedback, and training cycle of artificial neural networks seem to prohibit the nightly training consolidation cycle (i.e. sleep) that is an important part of natural neural networks.
  - rcxdude 17d ago
    There's two sides to efficiency: one of them is how much you put in and the other is how much you get out. I was asking the question because the answer of the relative efficiency of a human vs an LLM strongly depends on what you put on either side of the equation. It's a really easy pitfall to look at one side of one version of that equation and presume a really high inefficiency when that's not necessarily the case. It doesn't help that the data necessary to fully answer any version of this isn't publicly available. That's why I was asking for something more concrete in terms of how you were making the comparison. If you're looking at training costs, there is definitely one aspect in which LLMs are obviously significantly less efficient: the amount of information needed for the initial training. This does translate into some pretty high costs but it only needs to be paid once for the amount of work that any given LLM does. In terms of fine-tuning LLMs can get significantly more data-efficient than the initial model, which also means energy-efficient, and it's not obvious to me that it would be drastically worse than a human (though again, only thinking in terms of doing the energy input for the kind of work that an LLM is good at). (The increased efficiency in comparison to humans is part of the reason why you see Jevon's paradox mentioned a lot whenever concerns about the resources used by LLMs are mentioned: more efficiency can easily mean more resource use in total)
    - singingfish 14d ago
      I'm only concerned with the input efficiency, but the output energy efficiency is quite bad too. I can not see a pathway to generating economies of scale for this stuff.

rcxdude 18d ago

> Artificial neural networks are already many orders of magnitude more energy intensive than natural neural networks. What metric are you using to compare? By most counts, the energy budget of an instance of an LLM in a datacenter is lower than the energy a person uses. Of course, if you count energy per neuron connections vs weights then you'll likely get a quite different number. But then again LLMs do a lot of things with far fewer weights than the brain does neuron connections, even if you only count neurons in some parts of the brain. And of course you can point to capabilities that the brain has but LLMs lack, but on the whole it feels like it's pretty difficult to make a useful like-for-like comparison here.

singingfish 17d ago
I can't make sense of your comment. Firstly because of the obvious massive over-build of GPU infrastructure the AI companies are engaging in. Secondly because the instance of the LLM in the data centre that users interact with is only a small part of the story. The training phase is clearly prohibitively expensive, thus the fact that these things have no way to learn from experience except by smoke and mirrors. Also your comment feels like the classic climate denial discourse - say something a bit complicated and a bit difficult to follow that looks at a very small out of context part of the story to cast doubt.
singingfish 17d ago
Maybe I shouldn't have been pissed of by your commient. In which case - the excess costs of the training, output, feedback, and training cycle of artificial neural networks seem to prohibit the nightly training consolidation cycle (i.e. sleep) that is an important part of natural neural networks.
- rcxdude 17d ago
  There's two sides to efficiency: one of them is how much you put in and the other is how much you get out. I was asking the question because the answer of the relative efficiency of a human vs an LLM strongly depends on what you put on either side of the equation. It's a really easy pitfall to look at one side of one version of that equation and presume a really high inefficiency when that's not necessarily the case. It doesn't help that the data necessary to fully answer any version of this isn't publicly available. That's why I was asking for something more concrete in terms of how you were making the comparison. If you're looking at training costs, there is definitely one aspect in which LLMs are obviously significantly less efficient: the amount of information needed for the initial training. This does translate into some pretty high costs but it only needs to be paid once for the amount of work that any given LLM does. In terms of fine-tuning LLMs can get significantly more data-efficient than the initial model, which also means energy-efficient, and it's not obvious to me that it would be drastically worse than a human (though again, only thinking in terms of doing the energy input for the kind of work that an LLM is good at). (The increased efficiency in comparison to humans is part of the reason why you see Jevon's paradox mentioned a lot whenever concerns about the resources used by LLMs are mentioned: more efficiency can easily mean more resource use in total)
  - singingfish 14d ago
    I'm only concerned with the input efficiency, but the output energy efficiency is quite bad too. I can not see a pathway to generating economies of scale for this stuff.