3 ms·

Since there is no summary, these are the benchmark findings: AMX co-processor 2 TFLOPS FP32 GPU 8 TFLOPS FP32 Neural Engine 5.5 TFLOPS FP16

Firadeoclus 4y ago
Note that AMX can achieve roughly double the FLOPS with FP16, and 8 TFLOPS for the GPU is only about 77% of peak. You can do better than that, especially using FP16 90+% is possible (which is >9.4 TFLOPS).
londons_explore 4y ago
So why would you choose to use the Neural Engine rather than the GPU? Just power efficiency?
- potatolicious 4y ago
  That and if you want to use the GPU at the same time.
londons_explore 4y ago
Is there any easy way to use all of these at the same time? Ie. some library you can ask to do a big matrix multiply and it will loadbalance between the bits of hardware? Or do you have to manually split the computation between them?
- thewebcount 4y ago
  I’m by no means an expert in any of this. I mainly work on video processing using the GPU. That said, I would think if any library would do load balancing between them, it would likely be the Accelerate.framework that ships with the system. However, I do have some experience with having the same code run on the GPU and the CPU. In my work, we have tried breaking images (usually frames of video) into various sized chunks and processing on both the CPU and GPU at the same time. Our conclusion is that the overhead of using both outweighs any benefit you’d get. The GPU is so much faster than the CPU, there’s no point in involving the CPU at all. These experiments were done several years ago, so perhaps the landscape has changed since then, but that was what we found.
  - jasonwatkinspdx 4y ago
    You might find David Wright's presentations about Unreal 5 interesting: https://highperformancegraphics.org/slides22/Journey_to_Nanite.pdf https://highperformancegraphics.org/slides22/Journey_to_Nani... https://advances.realtimerendering.com/s2022/index.html#Lumen https://advances.realtimerendering.com/s2022/index.html#Lume... They're great presentations with a lot of depth in the notes. I think videos are around somewhere if you prefer that. Two specifics I'd mention: It seems a lot of games now use feedback between frames as a way to tolerate the latency of moving data between CPU and GPU. Eg the CPU will use GPU crunched data from the previous frame as a source for CPU crunching that optimizes what data gets passed to the GPU next. The other is that fixed functionality is moving into shaders. Unreal 5 uses a mix of hardware rasterization and software rasterization in a shader (and path tracing now as well). There the tradeoff between the two is triangle size in pixels.
    - thewebcount 4y ago
      Oh wow! Thanks! That looks really cool.
      - jasonwatkinspdx 4y ago
        They're great. I dunno if you find 3d interesting vs video, but the section in that nanite presentation where he goes through how he arrived at the LoD clustering design is some of the smartest stuff I've ever seen any developer say, ever. Like John Carmack probably saw this and went "dang, wish I'd thought of that" levels of smart.

Firadeoclus 4y ago

Note that AMX can achieve roughly double the FLOPS with FP16, and 8 TFLOPS for the GPU is only about 77% of peak. You can do better than that, especially using FP16 90+% is possible (which is >9.4 TFLOPS).

londons_explore 4y ago

So why would you choose to use the Neural Engine rather than the GPU? Just power efficiency?

potatolicious 4y ago
That and if you want to use the GPU at the same time.

londons_explore 4y ago

Is there any easy way to use all of these at the same time? Ie. some library you can ask to do a big matrix multiply and it will loadbalance between the bits of hardware? Or do you have to manually split the computation between them?

thewebcount 4y ago
I’m by no means an expert in any of this. I mainly work on video processing using the GPU. That said, I would think if any library would do load balancing between them, it would likely be the Accelerate.framework that ships with the system. However, I do have some experience with having the same code run on the GPU and the CPU. In my work, we have tried breaking images (usually frames of video) into various sized chunks and processing on both the CPU and GPU at the same time. Our conclusion is that the overhead of using both outweighs any benefit you’d get. The GPU is so much faster than the CPU, there’s no point in involving the CPU at all. These experiments were done several years ago, so perhaps the landscape has changed since then, but that was what we found.
- jasonwatkinspdx 4y ago
  You might find David Wright's presentations about Unreal 5 interesting: https://highperformancegraphics.org/slides22/Journey_to_Nanite.pdf https://highperformancegraphics.org/slides22/Journey_to_Nani... https://advances.realtimerendering.com/s2022/index.html#Lumen https://advances.realtimerendering.com/s2022/index.html#Lume... They're great presentations with a lot of depth in the notes. I think videos are around somewhere if you prefer that. Two specifics I'd mention: It seems a lot of games now use feedback between frames as a way to tolerate the latency of moving data between CPU and GPU. Eg the CPU will use GPU crunched data from the previous frame as a source for CPU crunching that optimizes what data gets passed to the GPU next. The other is that fixed functionality is moving into shaders. Unreal 5 uses a mix of hardware rasterization and software rasterization in a shader (and path tracing now as well). There the tradeoff between the two is triangle size in pixels.
  - thewebcount 4y ago
    Oh wow! Thanks! That looks really cool.
    - jasonwatkinspdx 4y ago
      They're great. I dunno if you find 3d interesting vs video, but the section in that nanite presentation where he goes through how he arrived at the LoD clustering design is some of the smartest stuff I've ever seen any developer say, ever. Like John Carmack probably saw this and went "dang, wish I'd thought of that" levels of smart.