4 ms·

Measuring energy usage: regular code vs. SIMD code

62 points by ashvardanian 3y ago

SomeoneFromCA 3y ago
everyone who has early batches of Alder Lake should try AVX512-FP16. Extremely fast for AI tasks.
BenoitP 3y ago
Please look at these number with the following grain of salt, when optimizing programs. They are trumped by cache usage patterns [1]: Loading (fig IV) Location Energy (pJ = pico Joules) L1 64 pJ/Byte L2 121 pJ/Byte L3 254 pJ/Byte RAM 1250 pJ/Byte Adding integers with SIMD (fig VI) 428 pJ/op, for 8 Byte/op; this means: 53 pJ/byte So it takes actually 20 times to fetch data from RAM than to add it to something! And most often this is also the source of latency. Generally energy is linear to the distance signal has to travel, and that's the same for latency. That's why successful data structures are sized a tiny bit under L1/L2 sizes! (BTree chunks, ring buffers). If you've been following hardware, it's all about putting RAM closer to compute at the moment with chiplets at the moment. [1] https://tu-dresden.de/zih/forschung/ressourcen/dateien/abgeschlossene-projekte/benchit/2010_IGCC_authors_version.pdf?lang=en https://tu-dresden.de/zih/forschung/ressourcen/dateien/abges...
jsheard 3y ago
It makes intuitive sense that going wide in SIMD uses less power than going wide over cores, because extra SIMD lanes take up much less silicon area than adding another whole core. If they didn't then we wouldn't bother with SIMD, we'd just make more cores and save ourselves the hassle of juggling two different types of parallelism. GPUs take this even further by going even wider, with each "core" typically executing 1024-bit SIMD operations (i.e. 32x FP32, 64x FP16, etc). CPUs have roughly settled at 128-bit (most ARM) or 256-bit (most x86) with a little bit of 512-bit (x86 with AVX512) which grew out of Intels earlier aborted attempt at making a dedicated GPU.
- dist-epoch 3y ago
  Not only that, but you need to decode the SIMD instruction only once, and not N times. Same with other stages of the pipeline.
  - sakras 3y ago
    Yep, I've heard that actually something like 80% of the energy of the CPU is used by the front-end! Beyond decoding, you also have all the energy spent on scoreboarding data dependencies and reordering the instructions and scheduling to execution ports. If you can do that 8x less as in the case for SIMD, that's a huge win!
    - jsheard 3y ago
      Memory traffic too, assuming the SIMD code is properly optimized, you can slam nicely aligned blocks of memory straight into a SIMD register rather than loading each element one at a time.
      - adgjlsfhk1 3y ago
        this is less important than you would expect because memory gets fetched into caches 512 bits at a time. As such, the difference in power use is only the cost of loading from L1 cache (since the first instruction will bring the line to l1)
    - adgjlsfhk1 3y ago
      That's definitely wrong (unless you use an incredibly expansive definition of "frontend"). The simplest way to prove that it's wrong is that heavily vectorized workloads can easily cause a 30% increase in power consumption over scalar workloads even though the instruction density is similar.
      - totallyabstract 3y ago
        Wouldn't that support the idea that most energy consumption is in decoding? If you're getting 2x, 4x, 8x ect as much value computation per instruction and yet only a 30% increase in power then clearly most the power is not used by computing the values.
hansonpeter 3y ago
[dead]
andy99 3y ago
There's a talk by a Nvidia scientist I just saw where he talks about compute efficiency in terms of flops/joule and explicitly mentions vectorized instructions as one of the energy savers. I'll see if I can find it. Edit: found it (I think) I can't remember where in the presentation but he does mention energy efficiency as a proxy for performance. https://m.youtube.com/watch?v=kLiwvnr4L80 https://m.youtube.com/watch?v=kLiwvnr4L80 Edit 2: about 26:40 he starts talking about energy use in the context of performance.
- 01HNNWZ0MV43FF 3y ago
  I guess it also makes sense if you figure that dedicated hardware for things like AES, video codecs, etc. are similar to SIMD
zokier 3y ago
> if your tasks complete much faster, you are likely to use less power, even if you are using more energy per unit of time. isn't this another formulation of the race-to-idle thinking?
ashvardanian 3y ago
The 3.5x energy-efficiency gap between serial and SIMD code becomes even larger when A. you do byte-level processing instead of float words; B. you use embedded, IoT, and other low-energy devices. A few years ago I've compared Nvidia Jetson Xavier (long before the Orin release), Intel-based MacBook Pro with Core i9, and AVX-512 capable CPUs on substring search benchmarks. On Xavier one can quite easily disable/enable cores and reconfigure power usage. At peak I got to 4.2 GB/J which was an 8.3x improvement in inefficiency over LibC in substring search operations. The comparison table is still available in the older README: https://github.com/ashvardanian/StringZilla/tree/v2.0.2?tab=readme-ov-file#substring-search https://github.com/ashvardanian/StringZilla/tree/v2.0.2?tab=...
Havoc 3y ago
Anybody know whether the same is true for gpu vid decode va cpu?

62 points by ashvardanian 3y ago

everyone who has early batches of Alder Lake should try AVX512-FP16. Extremely fast for AI tasks.

Please look at these number with the following grain of salt, when optimizing programs. They are trumped by cache usage patterns [1]: Loading (fig IV) Location Energy (pJ = pico Joules) L1 64 pJ/Byte L2 121 pJ/Byte L3 254 pJ/Byte RAM 1250 pJ/Byte Adding integers with SIMD (fig VI) 428 pJ/op, for 8 Byte/op; this means: 53 pJ/byte So it takes actually 20 times to fetch data from RAM than to add it to something! And most often this is also the source of latency. Generally energy is linear to the distance signal has to travel, and that's the same for latency. That's why successful data structures are sized a tiny bit under L1/L2 sizes! (BTree chunks, ring buffers). If you've been following hardware, it's all about putting RAM closer to compute at the moment with chiplets at the moment. [1] https://tu-dresden.de/zih/forschung/ressourcen/dateien/abgeschlossene-projekte/benchit/2010_IGCC_authors_version.pdf?lang=en https://tu-dresden.de/zih/forschung/ressourcen/dateien/abges...

jsheard 3y ago

It makes intuitive sense that going wide in SIMD uses less power than going wide over cores, because extra SIMD lanes take up much less silicon area than adding another whole core. If they didn't then we wouldn't bother with SIMD, we'd just make more cores and save ourselves the hassle of juggling two different types of parallelism. GPUs take this even further by going even wider, with each "core" typically executing 1024-bit SIMD operations (i.e. 32x FP32, 64x FP16, etc). CPUs have roughly settled at 128-bit (most ARM) or 256-bit (most x86) with a little bit of 512-bit (x86 with AVX512) which grew out of Intels earlier aborted attempt at making a dedicated GPU.

dist-epoch 3y ago
Not only that, but you need to decode the SIMD instruction only once, and not N times. Same with other stages of the pipeline.
- sakras 3y ago
  Yep, I've heard that actually something like 80% of the energy of the CPU is used by the front-end! Beyond decoding, you also have all the energy spent on scoreboarding data dependencies and reordering the instructions and scheduling to execution ports. If you can do that 8x less as in the case for SIMD, that's a huge win!
  - jsheard 3y ago
    Memory traffic too, assuming the SIMD code is properly optimized, you can slam nicely aligned blocks of memory straight into a SIMD register rather than loading each element one at a time.
    - adgjlsfhk1 3y ago
      this is less important than you would expect because memory gets fetched into caches 512 bits at a time. As such, the difference in power use is only the cost of loading from L1 cache (since the first instruction will bring the line to l1)
  - adgjlsfhk1 3y ago
    That's definitely wrong (unless you use an incredibly expansive definition of "frontend"). The simplest way to prove that it's wrong is that heavily vectorized workloads can easily cause a 30% increase in power consumption over scalar workloads even though the instruction density is similar.
    - totallyabstract 3y ago
      Wouldn't that support the idea that most energy consumption is in decoding? If you're getting 2x, 4x, 8x ect as much value computation per instruction and yet only a 30% increase in power then clearly most the power is not used by computing the values.

hansonpeter 3y ago

[dead]

andy99 3y ago

There's a talk by a Nvidia scientist I just saw where he talks about compute efficiency in terms of flops/joule and explicitly mentions vectorized instructions as one of the energy savers. I'll see if I can find it. Edit: found it (I think) I can't remember where in the presentation but he does mention energy efficiency as a proxy for performance. https://m.youtube.com/watch?v=kLiwvnr4L80 https://m.youtube.com/watch?v=kLiwvnr4L80 Edit 2: about 26:40 he starts talking about energy use in the context of performance.

01HNNWZ0MV43FF 3y ago
I guess it also makes sense if you figure that dedicated hardware for things like AES, video codecs, etc. are similar to SIMD

zokier 3y ago

> if your tasks complete much faster, you are likely to use less power, even if you are using more energy per unit of time. isn't this another formulation of the race-to-idle thinking?

ashvardanian 3y ago

The 3.5x energy-efficiency gap between serial and SIMD code becomes even larger when A. you do byte-level processing instead of float words; B. you use embedded, IoT, and other low-energy devices. A few years ago I've compared Nvidia Jetson Xavier (long before the Orin release), Intel-based MacBook Pro with Core i9, and AVX-512 capable CPUs on substring search benchmarks. On Xavier one can quite easily disable/enable cores and reconfigure power usage. At peak I got to 4.2 GB/J which was an 8.3x improvement in inefficiency over LibC in substring search operations. The comparison table is still available in the older README: https://github.com/ashvardanian/StringZilla/tree/v2.0.2?tab=readme-ov-file#substring-search https://github.com/ashvardanian/StringZilla/tree/v2.0.2?tab=...

Havoc 3y ago

Anybody know whether the same is true for gpu vid decode va cpu?