2 ms·

> We propose intelligence per watt (IPW), task accuracy per unit of power Stupid metric. It's not because a model is better performing that it necessarily requ

> We propose intelligence per watt (IPW), task accuracy per unit of power Stupid metric. It's not because a model is better performing that it necessarily requires more energy or compute.

utopiah 10d ago
I don't think that's what they are saying. In fact if they did the metric would be pointless. Rather they are saying by estimating that value on different architectures, one can find more efficient ones. They use open model to be able to remove unknowns. They aren't advocating for one model or another, only more efficient architectures.
- the8472 10d ago
  Intelligence per Joule would be more appropriate in many cases. If a model can do the same work but takes 10 times as long as a bigger one that can still be useful (e.g. due to memory constraints), but at the same wattage it burns 10 times the energy. Even more so on mobile devices.
  - frumiousirc 10d ago
    They also define and measure an "IPJ" as well as "IPW" > the NVIDIA B200 achieves 1.6× to 2.3× higher intelligence per joule than the APPLE M4 MAX across QWEN 3 and GPT-OSS model variants The B200 = "cloud", M4 = "local". So "cloud" does even better in energy than it does in power compared to "local". Or, to flip it, "local" is both slower and more expensive than "cloud".
- iLoveOncall 10d ago
  Then the metric should be Watts per Intelligence, not the contrary.
  - lioeters 10d ago
    Is there any practical difference? One is just an inverse of the other and can be trivially derived.
    - iLoveOncall 10d ago
      Yes, there is a massive difference, and you cannot invert them to get the other. Watt per Intelligence means that you have a fixed, deterministic, measure of intelligence, and you calculate how many watts it takes to get there. If your goal is to measure which model can reach a specific outcome with the least energy possible (which is what GP says the goal is for this metric), then you cannot have a variable outcome, which is what intelligence per watt describes. As opposed to watt per intelligence, where the outcome is fixed and the numerator defines how much energy expenditure is needed to reach this fixed outcome.
      - lioeters 10d ago
        Here is a more everyday example: distance traveled by a vehicle and the amount of fuel consumed. > Fuel efficiency can be expressed in terms of the volume of fuel to travel a given distance, such as in litres per 100 kilometres, or through its inverse, the distance traveled per unit volume of fuel consumed, as in kilometres per litre. If you have a car that consumes 1 liter of fuel per 100 kilometers, it's the same as saying it travels 1 kilometer per 0.01 liter of fuel. They are equivalent, describing the same relationship and proportion. It makes no difference whether the quantity is a "deterministic" or "variable" measurement. You can use either unit, depending on the aim of your calculation.
        iLoveOncall 10d ago
        This has absolutely nothing to do with the problem at end. I'm so tired of people bringing up random metaphors that don't apply to justify obvious bullshit when it comes to AI.
_diyar 10d ago
> We propose miles per hour (MPH), distance travelled per unit of time Stupid metric. It‘s not because you spend more time that you travel farther. s/
fizzbuzzbarbazz 10d ago
Not a well-thought-out take.

> We propose intelligence per watt (IPW), task accuracy per unit of power Stupid metric. It's not because a model is better performing that it necessarily requires more energy or compute.

utopiah 10d ago

I don't think that's what they are saying. In fact if they did the metric would be pointless. Rather they are saying by estimating that value on different architectures, one can find more efficient ones. They use open model to be able to remove unknowns. They aren't advocating for one model or another, only more efficient architectures.

the8472 10d ago
Intelligence per Joule would be more appropriate in many cases. If a model can do the same work but takes 10 times as long as a bigger one that can still be useful (e.g. due to memory constraints), but at the same wattage it burns 10 times the energy. Even more so on mobile devices.
- frumiousirc 10d ago
  They also define and measure an "IPJ" as well as "IPW" > the NVIDIA B200 achieves 1.6× to 2.3× higher intelligence per joule than the APPLE M4 MAX across QWEN 3 and GPT-OSS model variants The B200 = "cloud", M4 = "local". So "cloud" does even better in energy than it does in power compared to "local". Or, to flip it, "local" is both slower and more expensive than "cloud".
iLoveOncall 10d ago
Then the metric should be Watts per Intelligence, not the contrary.
- lioeters 10d ago
  Is there any practical difference? One is just an inverse of the other and can be trivially derived.
  - iLoveOncall 10d ago
    Yes, there is a massive difference, and you cannot invert them to get the other. Watt per Intelligence means that you have a fixed, deterministic, measure of intelligence, and you calculate how many watts it takes to get there. If your goal is to measure which model can reach a specific outcome with the least energy possible (which is what GP says the goal is for this metric), then you cannot have a variable outcome, which is what intelligence per watt describes. As opposed to watt per intelligence, where the outcome is fixed and the numerator defines how much energy expenditure is needed to reach this fixed outcome.
    - lioeters 10d ago
      Here is a more everyday example: distance traveled by a vehicle and the amount of fuel consumed. > Fuel efficiency can be expressed in terms of the volume of fuel to travel a given distance, such as in litres per 100 kilometres, or through its inverse, the distance traveled per unit volume of fuel consumed, as in kilometres per litre. If you have a car that consumes 1 liter of fuel per 100 kilometers, it's the same as saying it travels 1 kilometer per 0.01 liter of fuel. They are equivalent, describing the same relationship and proportion. It makes no difference whether the quantity is a "deterministic" or "variable" measurement. You can use either unit, depending on the aim of your calculation.
      - iLoveOncall 10d ago
        This has absolutely nothing to do with the problem at end. I'm so tired of people bringing up random metaphors that don't apply to justify obvious bullshit when it comes to AI.

_diyar 10d ago

> We propose miles per hour (MPH), distance travelled per unit of time Stupid metric. It‘s not because you spend more time that you travel farther. s/

fizzbuzzbarbazz 10d ago

Not a well-thought-out take.