9 ms·

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

24 points by pythonic_hell 13d ago

iLoveOncall 11d ago
> We propose intelligence per watt (IPW), task accuracy per unit of power Stupid metric. It's not because a model is better performing that it necessarily requires more energy or compute.
- utopiah 11d ago
  I don't think that's what they are saying. In fact if they did the metric would be pointless. Rather they are saying by estimating that value on different architectures, one can find more efficient ones. They use open model to be able to remove unknowns. They aren't advocating for one model or another, only more efficient architectures.
  - the8472 11d ago
    Intelligence per Joule would be more appropriate in many cases. If a model can do the same work but takes 10 times as long as a bigger one that can still be useful (e.g. due to memory constraints), but at the same wattage it burns 10 times the energy. Even more so on mobile devices.
    - frumiousirc 11d ago
      They also define and measure an "IPJ" as well as "IPW" > the NVIDIA B200 achieves 1.6× to 2.3× higher intelligence per joule than the APPLE M4 MAX across QWEN 3 and GPT-OSS model variants The B200 = "cloud", M4 = "local". So "cloud" does even better in energy than it does in power compared to "local". Or, to flip it, "local" is both slower and more expensive than "cloud".
  - iLoveOncall 11d ago
    Then the metric should be Watts per Intelligence, not the contrary.
    - lioeters 11d ago
      Is there any practical difference? One is just an inverse of the other and can be trivially derived.
      - iLoveOncall 11d ago
        Yes, there is a massive difference, and you cannot invert them to get the other. Watt per Intelligence means that you have a fixed, deterministic, measure of intelligence, and you calculate how many watts it takes to get there. If your goal is to measure which model can reach a specific outcome with the least energy possible (which is what GP says the goal is for this metric), then you cannot have a variable outcome, which is what intelligence per watt describes. As opposed to watt per intelligence, where the outcome is fixed and the numerator defines how much energy expenditure is needed to reach this fixed outcome.
        lioeters 11d ago
        Here is a more everyday example: distance traveled by a vehicle and the amount of fuel consumed. > Fuel efficiency can be expressed in terms of the volume of fuel to travel a given distance, such as in litres per 100 kilometres, or through its inverse, the distance traveled per unit volume of fuel consumed, as in kilometres per litre. If you have a car that consumes 1 liter of fuel per 100 kilometers, it's the same as saying it travels 1 kilometer per 0.01 liter of fuel. They are equivalent, describing the same relationship and proportion. It makes no difference whether the quantity is a "deterministic" or "variable" measurement. You can use either unit, depending on the aim of your calculation.
        iLoveOncall 11d ago
        This has absolutely nothing to do with the problem at end. I'm so tired of people bringing up random metaphors that don't apply to justify obvious bullshit when it comes to AI.
- _diyar 11d ago
  > We propose miles per hour (MPH), distance travelled per unit of time Stupid metric. It‘s not because you spend more time that you travel farther. s/
- fizzbuzzbarbazz 11d ago
  Not a well-thought-out take.
api 11d ago
Unless I misread it, are they saying local GPUs use less energy? That’s surprising, almost unbelievable, due to batching. Local is usually not batched.
- stymaar 11d ago
  Small models are much smaller than frontier models though, which is how they end up consuming less energy despite low batch count. (Though with local models growing strong agentic capabilities, batching becomes a reality with local models as well).
- frumiousirc 11d ago
  > Unless I misread it, are they saying local GPUs use less energy? You misread it. From the abstract: > local accelerators achieve at least 1.4× lower IPW than cloud accelerators running identical models That's "intelligence per watt". They also have IPJ, per Joule. So, they find local is 40% "dumber" than cloud for the same power or 40% more power for the same "intelligence". Tables 13 and 14 summarize their IPW and IPJ metrics. But, to your actual point, I think the "local is 40% dumber per watt than cloud" message is still an understatement. And maybe this is something I failed to find in the paper but they seem to ignore the "idle baseline" costs and talks about explicitly focusing on the power consumption of just the accelerator under load. There is a large baseline power consumption just to support the accelerator. CPUs, memory, PS losses, network, fans, general environment cooling. This "cost floor" is different for data centers and a "random local computer" and I think must be in favor of data centers which are designed and built with efficiency in mind. Idleness should also be considered. My local GPUs at $WORK and home are idle more than they are used. Idle time energy in real world scenarios should be somehow attributed to those brief, punctuated times when LLM functions are actually active on the accelerator. Actual, local LLM usage of a GPU is brief (assuming one user per PC). Even with my heavy usage developing s/w I'd guess I heat up a GPU about one hour per day total, sometimes much less. If that is local then one must pay 23 hours of idleness for that 1 hour of "intelligence". Of course a local PC is used for other things and the idleness penalty must somehow account for that. OTOH, data centers try to maximize utilization so their idle time penalty would be much less, perhaps close to zero, by construction.
kaziava 11d ago
this is the metric i've been waiting for. we run everything local (ollama + neo4j) for compliance reasons, so 'quality per watt' is literally our budget line. one data point from our setup: qwen2.5:3b on an m2 macbook handles nl-to-cypher for simple graph schemas at ~3-5s per answer, and the energy cost is a rounding error compared to shipping the same queries to a frontier api. the hard part was never the model though, it was parsing pdfs locally without a vision model. would love to see parsing/ocr covered in future benchmarks.
jmiskovic 11d ago
Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winter time the GPU also doubles as a 300W in-house heater. Not to mention avoiding personal data collection and re-selling.
- jgalt212 11d ago
  > It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. This is only true if your local model is already resident in RAM / VRAM.
  - jmiskovic 11d ago
    In my case the bad internet also tips the scale.
- rpozarickij 11d ago
  > During the winter time the GPU also doubles as a 300W in-house heater There could be a service that works in reverse where if someone needs a heater for a few months, they could rent a portable server (e.g. using older repurposed GPUs) with a built-in 5G modem that would run inference on LLM queries. As an incentive perhaps renting itself could be free (or you could earn money?), but you'd still have to pay your electricity bill.
  - HPsquared 11d ago
    I remember a company that sold or used bitcoin mining rigs as swimming pool heaters.
    - teamonkey 11d ago
      There was one that tried to sell them as free heating for the elderly.
Havoc 11d ago
Saw some measurements on SBC NPUs (3588) and that did seem to have a decent win on power over CPU...but also a perplexity loss relative to CPU so think this will prove quite hard to reliably quantify in practice.
scottcha 11d ago
Very cool paper and some interesting things for local serving. Though its a little apples-to-oranges we do publish live energy stats for all models on our service here https://portal.neuralwatt.com/energy-pricing https://portal.neuralwatt.com/energy-pricing in case you are interested in what this looks like on the cloud side. FWIW DSV4.1 flash is really getting popular due to its IPW. Some of the items like model routing, if you do it per request instead of per session, can break down on the cloud from an energy and cost POV since one of the best things you can do for both is to maintain the KV cache which both reduces time component of energy and the quite expensive prefill energy. I am keen on the future where we have local/cloud hybrid serving which is cache aware. I do think that could be the best use of energy resources for AI.
nilsharker 11d ago
[dead]
surprisetalk 11d ago
I rather like this paper, but I think it is generous to say their benchmarks measure "intelligence". [0] https://arxiv.org/pdf/1911.01547 https://arxiv.org/pdf/1911.01547 We have no dang clue what intelligence is, nor how to measure it. [1] https://taylor.town/crowpower https://taylor.town/crowpower
- paimapi 11d ago
  it really does feel like 'intelligence' is becoming so much more of a marketing term moreso than it is any actual definitive measure. I was acquaintances with a cognitive science post-doc many years ago who trained under the same program that Douglas Hofstadter was part of (and who he saw as a great popular science communicator but perhaps a bit over-rated when it came to his own research) my very naive question to him back then was how close we were to understanding human cognition. we were both fans of grand strategy games (though the few hundred hours of Stellaris I played vs his thousands in EU4 paled in comparison) and I was asking if it was possible to map human cognition to the same array of interdependent logical chains-of-reasoning that games like that could be boiled down to his answer, in short, was 'we are so, so, so far away and no, that's, at best, a reductive mental model of intelligence' I keep that conversation in mind whenever I hear about all this talk of AGI - that realistically we're so far away from actual AGI in the same way that the inventor(s) of the wheel were from a gas-powered car, and there's many paradigm shifts to go in how we even understand what the nature of intelligence is before we get there
  - mswphd 11d ago
    there's a funny paper on this theme, titles "Could a Neuroscientist Understand a Microprocessor" https://pmc.ncbi.nlm.nih.gov/articles/PMC5230747/ https://pmc.ncbi.nlm.nih.gov/articles/PMC5230747/ roughly, it reviews common techniques in neuroscience, and comes to the conclusion that they would not be able to understand even simple computing platforms that we have perfect information for (and can perfectly stimulate any internal connection, can perfectly read out the values on any internal connection, etc).
- RunSet 11d ago
  "We see a future where intelligence is a utility like electricity or water and people buy it from us on a meter" - Sam Altman[0] "Can you define intelligence?" "Yes, it is this many moneys." [0] https://www.businessinsider.com/sam-altman-ai-utility-electricity-water-openai-2026-3 https://www.businessinsider.com/sam-altman-ai-utility-electr...
djb_hackernews 11d ago
ok now let's do some math to calculate the current levels of daily intelligence to calculate how much power we need to replace 10% of it with AI. Never mind new use cases or approaching human replacement. My guess is it is a few orders of magnitude more power than we produce today.
polotics 11d ago
Did I miss something or does this article not bother to indicate how much RAM their M4-Max had?
- washadjeffmad 11d ago
  [dead]
WASDx 11d ago
I tried calculating historical "intelligence per cost" recently but stopped when I realized intelligence is not linear. For any meaningful "x per y" you can just double "y" if you have a half as efficient system to get the same result but so-called intelligence doesn't work like that.
arttaboi 11d ago
[dead]
nickyocean 11d ago
[flagged]
sarfaraznaushad 10d ago
[flagged]

24 points by pythonic_hell 13d ago

> We propose intelligence per watt (IPW), task accuracy per unit of power Stupid metric. It's not because a model is better performing that it necessarily requires more energy or compute.

utopiah 11d ago
I don't think that's what they are saying. In fact if they did the metric would be pointless. Rather they are saying by estimating that value on different architectures, one can find more efficient ones. They use open model to be able to remove unknowns. They aren't advocating for one model or another, only more efficient architectures.
- the8472 11d ago
  Intelligence per Joule would be more appropriate in many cases. If a model can do the same work but takes 10 times as long as a bigger one that can still be useful (e.g. due to memory constraints), but at the same wattage it burns 10 times the energy. Even more so on mobile devices.
  - frumiousirc 11d ago
    They also define and measure an "IPJ" as well as "IPW" > the NVIDIA B200 achieves 1.6× to 2.3× higher intelligence per joule than the APPLE M4 MAX across QWEN 3 and GPT-OSS model variants The B200 = "cloud", M4 = "local". So "cloud" does even better in energy than it does in power compared to "local". Or, to flip it, "local" is both slower and more expensive than "cloud".
- iLoveOncall 11d ago
  Then the metric should be Watts per Intelligence, not the contrary.
  - lioeters 11d ago
    Is there any practical difference? One is just an inverse of the other and can be trivially derived.
    - iLoveOncall 11d ago
      Yes, there is a massive difference, and you cannot invert them to get the other. Watt per Intelligence means that you have a fixed, deterministic, measure of intelligence, and you calculate how many watts it takes to get there. If your goal is to measure which model can reach a specific outcome with the least energy possible (which is what GP says the goal is for this metric), then you cannot have a variable outcome, which is what intelligence per watt describes. As opposed to watt per intelligence, where the outcome is fixed and the numerator defines how much energy expenditure is needed to reach this fixed outcome.
      - lioeters 11d ago
        Here is a more everyday example: distance traveled by a vehicle and the amount of fuel consumed. > Fuel efficiency can be expressed in terms of the volume of fuel to travel a given distance, such as in litres per 100 kilometres, or through its inverse, the distance traveled per unit volume of fuel consumed, as in kilometres per litre. If you have a car that consumes 1 liter of fuel per 100 kilometers, it's the same as saying it travels 1 kilometer per 0.01 liter of fuel. They are equivalent, describing the same relationship and proportion. It makes no difference whether the quantity is a "deterministic" or "variable" measurement. You can use either unit, depending on the aim of your calculation.
        iLoveOncall 11d ago
        This has absolutely nothing to do with the problem at end. I'm so tired of people bringing up random metaphors that don't apply to justify obvious bullshit when it comes to AI.
_diyar 11d ago
> We propose miles per hour (MPH), distance travelled per unit of time Stupid metric. It‘s not because you spend more time that you travel farther. s/
fizzbuzzbarbazz 11d ago
Not a well-thought-out take.

api 11d ago

Unless I misread it, are they saying local GPUs use less energy? That’s surprising, almost unbelievable, due to batching. Local is usually not batched.

stymaar 11d ago
Small models are much smaller than frontier models though, which is how they end up consuming less energy despite low batch count. (Though with local models growing strong agentic capabilities, batching becomes a reality with local models as well).
frumiousirc 11d ago
> Unless I misread it, are they saying local GPUs use less energy? You misread it. From the abstract: > local accelerators achieve at least 1.4× lower IPW than cloud accelerators running identical models That's "intelligence per watt". They also have IPJ, per Joule. So, they find local is 40% "dumber" than cloud for the same power or 40% more power for the same "intelligence". Tables 13 and 14 summarize their IPW and IPJ metrics. But, to your actual point, I think the "local is 40% dumber per watt than cloud" message is still an understatement. And maybe this is something I failed to find in the paper but they seem to ignore the "idle baseline" costs and talks about explicitly focusing on the power consumption of just the accelerator under load. There is a large baseline power consumption just to support the accelerator. CPUs, memory, PS losses, network, fans, general environment cooling. This "cost floor" is different for data centers and a "random local computer" and I think must be in favor of data centers which are designed and built with efficiency in mind. Idleness should also be considered. My local GPUs at $WORK and home are idle more than they are used. Idle time energy in real world scenarios should be somehow attributed to those brief, punctuated times when LLM functions are actually active on the accelerator. Actual, local LLM usage of a GPU is brief (assuming one user per PC). Even with my heavy usage developing s/w I'd guess I heat up a GPU about one hour per day total, sometimes much less. If that is local then one must pay 23 hours of idleness for that 1 hour of "intelligence". Of course a local PC is used for other things and the idleness penalty must somehow account for that. OTOH, data centers try to maximize utilization so their idle time penalty would be much less, perhaps close to zero, by construction.

kaziava 11d ago

this is the metric i've been waiting for. we run everything local (ollama + neo4j) for compliance reasons, so 'quality per watt' is literally our budget line. one data point from our setup: qwen2.5:3b on an m2 macbook handles nl-to-cypher for simple graph schemas at ~3-5s per answer, and the energy cost is a rounding error compared to shipping the same queries to a frontier api. the hard part was never the model though, it was parsing pdfs locally without a vision model. would love to see parsing/ocr covered in future benchmarks.

jmiskovic 11d ago

Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winter time the GPU also doubles as a 300W in-house heater. Not to mention avoiding personal data collection and re-selling.

jgalt212 11d ago
> It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. This is only true if your local model is already resident in RAM / VRAM.
- jmiskovic 11d ago
  In my case the bad internet also tips the scale.
rpozarickij 11d ago
> During the winter time the GPU also doubles as a 300W in-house heater There could be a service that works in reverse where if someone needs a heater for a few months, they could rent a portable server (e.g. using older repurposed GPUs) with a built-in 5G modem that would run inference on LLM queries. As an incentive perhaps renting itself could be free (or you could earn money?), but you'd still have to pay your electricity bill.
- HPsquared 11d ago
  I remember a company that sold or used bitcoin mining rigs as swimming pool heaters.
  - teamonkey 11d ago
    There was one that tried to sell them as free heating for the elderly.

Havoc 11d ago

Saw some measurements on SBC NPUs (3588) and that did seem to have a decent win on power over CPU...but also a perplexity loss relative to CPU so think this will prove quite hard to reliably quantify in practice.

scottcha 11d ago

Very cool paper and some interesting things for local serving. Though its a little apples-to-oranges we do publish live energy stats for all models on our service here https://portal.neuralwatt.com/energy-pricing https://portal.neuralwatt.com/energy-pricing in case you are interested in what this looks like on the cloud side. FWIW DSV4.1 flash is really getting popular due to its IPW. Some of the items like model routing, if you do it per request instead of per session, can break down on the cloud from an energy and cost POV since one of the best things you can do for both is to maintain the KV cache which both reduces time component of energy and the quite expensive prefill energy. I am keen on the future where we have local/cloud hybrid serving which is cache aware. I do think that could be the best use of energy resources for AI.

nilsharker 11d ago

[dead]

surprisetalk 11d ago

I rather like this paper, but I think it is generous to say their benchmarks measure "intelligence". [0] https://arxiv.org/pdf/1911.01547 https://arxiv.org/pdf/1911.01547 We have no dang clue what intelligence is, nor how to measure it. [1] https://taylor.town/crowpower https://taylor.town/crowpower

paimapi 11d ago
it really does feel like 'intelligence' is becoming so much more of a marketing term moreso than it is any actual definitive measure. I was acquaintances with a cognitive science post-doc many years ago who trained under the same program that Douglas Hofstadter was part of (and who he saw as a great popular science communicator but perhaps a bit over-rated when it came to his own research) my very naive question to him back then was how close we were to understanding human cognition. we were both fans of grand strategy games (though the few hundred hours of Stellaris I played vs his thousands in EU4 paled in comparison) and I was asking if it was possible to map human cognition to the same array of interdependent logical chains-of-reasoning that games like that could be boiled down to his answer, in short, was 'we are so, so, so far away and no, that's, at best, a reductive mental model of intelligence' I keep that conversation in mind whenever I hear about all this talk of AGI - that realistically we're so far away from actual AGI in the same way that the inventor(s) of the wheel were from a gas-powered car, and there's many paradigm shifts to go in how we even understand what the nature of intelligence is before we get there
- mswphd 11d ago
  there's a funny paper on this theme, titles "Could a Neuroscientist Understand a Microprocessor" https://pmc.ncbi.nlm.nih.gov/articles/PMC5230747/ https://pmc.ncbi.nlm.nih.gov/articles/PMC5230747/ roughly, it reviews common techniques in neuroscience, and comes to the conclusion that they would not be able to understand even simple computing platforms that we have perfect information for (and can perfectly stimulate any internal connection, can perfectly read out the values on any internal connection, etc).
RunSet 11d ago
"We see a future where intelligence is a utility like electricity or water and people buy it from us on a meter" - Sam Altman[0] "Can you define intelligence?" "Yes, it is this many moneys." [0] https://www.businessinsider.com/sam-altman-ai-utility-electricity-water-openai-2026-3 https://www.businessinsider.com/sam-altman-ai-utility-electr...

djb_hackernews 11d ago

ok now let's do some math to calculate the current levels of daily intelligence to calculate how much power we need to replace 10% of it with AI. Never mind new use cases or approaching human replacement. My guess is it is a few orders of magnitude more power than we produce today.

polotics 11d ago

Did I miss something or does this article not bother to indicate how much RAM their M4-Max had?

washadjeffmad 11d ago
[dead]

WASDx 11d ago

I tried calculating historical "intelligence per cost" recently but stopped when I realized intelligence is not linear. For any meaningful "x per y" you can just double "y" if you have a half as efficient system to get the same result but so-called intelligence doesn't work like that.

arttaboi 11d ago

[dead]

nickyocean 11d ago

[flagged]

sarfaraznaushad 10d ago

[flagged]