3 ms·

I am really surprised to see the performance of the CPU and especially the latency characteristics are so poor. The article alludes to the design likely being t

I am really surprised to see the performance of the CPU and especially the latency characteristics are so poor. The article alludes to the design likely being tuned for specific workloads, which seems like a good explanation. But I can't help wonder if throughput at the cost of high memory latency is just not a good strategy for CPUs even with the excellent branch predictors and clever OOO work that modern CPUs bring to the table. Is this a bad take? Are we just not seeing the intended use-case where this thing really shines compared to anything else?

p1necone 2y ago
This kind of hardware makes sense for video games, and I guess GPU heavy workloads like AI might be similar? Most games have middling compute requirements but will take as much GPU power as you can give them if you're trying to run at high resolutions/settings. Although getting smooth gameplay at very high frame rates (~120hz+) does need a decent CPU in a lot of games. Look at how atrocious the CPUs were in the PS4/Xbone generation for an example of this.
- wmf 2y ago
  Grace Hopper was not designed for games though.
- pjmlp 2y ago
  And yet PS 4 / XBox ONE rule the games console market still, because only more polygons isn't worth buying a PS 5 or XSeries, for a large market segment, hence the negative sales and trying to cater to PC gamers as alternative.
  - p1necone 2y ago
    Yes I was making the point that the CPUs in PS4/Xbone were terrible (seriously look up benchmarks - they're basically underclocked pre ryzen AMD) and that didn't matter for performance because video game workloads are so heavily skewed towards GPU. I know they were successful.
edward28 2y ago
These CPUs are intended just to run miscellaneous tasks, such as loading AI models or running the cluster operating system. They don't need to be performant, just efficient, as the GPU does all the heavy lifting. NVIDIA also provides an option to swap the grace chips out with an x86 chip, which could deliver better performance depending on the remaining power budget though.
- MobiusHorizons 2y ago
  If this is all there is to it, why do they have the high frequency and high l3 cache? Those seem to be optimizing for something, not just a “good enough” configuration for a part that is not the bottleneck
  - riotnrrd 2y ago
    Data augmentation in CPU-space is often compute-light, but requires rapid access to memory. There are libraries (like NVIDIA's Dali) that can do augmentation on the GPU, but this takes up GPU resources that could be used by training. Having a multi-core CPU with fast caches is a good compromise.
tonyarkles 2y ago
We’re using the Orin AGX for edge ML. Not the same setup (Ampere) but it’s a similar situation. The GPU is excellent for what we need it to do, but the CPU cores are painful. We’re lucky… the CPUs aren’t great but there’s 12 of them and we can get away with carefully pipelining our data flows across multiple threads to get the throughput we need even though some individual stage latencies aren’t what we’d like.
freeqaz 2y ago
What's the point of having the GPU on die for this? Are they expecting people to deploy one of these nodes without dedicated GPUs? It has a ton of NVLink connections which makes me think that these will often be deployed alongside GPUs which feels weird. The flip side of this is if the GPU can access the main system memory then I could see this being useful for loading big models with much more efficient "offloading" of layers. Even though bandwidth between GPU->LPDDR5 is going to be slow, it's still faster than what traditional PCI-E would allow. The caveat here is that I imagine these machines are $$$ and enterprise only. If something like this was brought to the consumer market though I think it would be very enticing. (If anybody from AMD is reading this, I feel like an architecture like this would be awesome to have. I would love to run Llama 3.1 405b at home and today I see zero path towards doing that for any "reasonable" amount of money (<$10k?).) Edit: It's at the bottom of the article. These are designed to be meshed together via NVLink into one big cluster. Makes sense. I'm really curious how the system RAM would be used in LLM training scenarios, or if these boxes are going to be used for totally different tasks that I have little context into.
weebull 2y ago
I'd really like to get hold of a model for a modern CPU and properly analyse what all the performance features actually get us in terms of performance. - Branch prediction and speculative execution - Out of order execution - Massive physical register files and register renaming - Cache predictors - and many more I'm sure. Speculative execution is the big one for me, just because of the information leakage possible through it. It's there because you'd have to pause fetching new instructions until the result of a conditional branch is known, which has knock-on effects to instruction scheduling... But how big are these effects? Do some certain combinations of features supercharge or work against each other? I'm sure there's people looking at such things inside Intel and AMD, but it doesn't seem like there's much out there for public consumption.

p1necone 2y ago

This kind of hardware makes sense for video games, and I guess GPU heavy workloads like AI might be similar? Most games have middling compute requirements but will take as much GPU power as you can give them if you're trying to run at high resolutions/settings. Although getting smooth gameplay at very high frame rates (~120hz+) does need a decent CPU in a lot of games. Look at how atrocious the CPUs were in the PS4/Xbone generation for an example of this.

wmf 2y ago
Grace Hopper was not designed for games though.
pjmlp 2y ago
And yet PS 4 / XBox ONE rule the games console market still, because only more polygons isn't worth buying a PS 5 or XSeries, for a large market segment, hence the negative sales and trying to cater to PC gamers as alternative.
- p1necone 2y ago
  Yes I was making the point that the CPUs in PS4/Xbone were terrible (seriously look up benchmarks - they're basically underclocked pre ryzen AMD) and that didn't matter for performance because video game workloads are so heavily skewed towards GPU. I know they were successful.

edward28 2y ago

These CPUs are intended just to run miscellaneous tasks, such as loading AI models or running the cluster operating system. They don't need to be performant, just efficient, as the GPU does all the heavy lifting. NVIDIA also provides an option to swap the grace chips out with an x86 chip, which could deliver better performance depending on the remaining power budget though.

MobiusHorizons 2y ago
If this is all there is to it, why do they have the high frequency and high l3 cache? Those seem to be optimizing for something, not just a “good enough” configuration for a part that is not the bottleneck
- riotnrrd 2y ago
  Data augmentation in CPU-space is often compute-light, but requires rapid access to memory. There are libraries (like NVIDIA's Dali) that can do augmentation on the GPU, but this takes up GPU resources that could be used by training. Having a multi-core CPU with fast caches is a good compromise.

tonyarkles 2y ago

We’re using the Orin AGX for edge ML. Not the same setup (Ampere) but it’s a similar situation. The GPU is excellent for what we need it to do, but the CPU cores are painful. We’re lucky… the CPUs aren’t great but there’s 12 of them and we can get away with carefully pipelining our data flows across multiple threads to get the throughput we need even though some individual stage latencies aren’t what we’d like.

freeqaz 2y ago

What's the point of having the GPU on die for this? Are they expecting people to deploy one of these nodes without dedicated GPUs? It has a ton of NVLink connections which makes me think that these will often be deployed alongside GPUs which feels weird. The flip side of this is if the GPU can access the main system memory then I could see this being useful for loading big models with much more efficient "offloading" of layers. Even though bandwidth between GPU->LPDDR5 is going to be slow, it's still faster than what traditional PCI-E would allow. The caveat here is that I imagine these machines are $$$ and enterprise only. If something like this was brought to the consumer market though I think it would be very enticing. (If anybody from AMD is reading this, I feel like an architecture like this would be awesome to have. I would love to run Llama 3.1 405b at home and today I see zero path towards doing that for any "reasonable" amount of money (<$10k?).) Edit: It's at the bottom of the article. These are designed to be meshed together via NVLink into one big cluster. Makes sense. I'm really curious how the system RAM would be used in LLM training scenarios, or if these boxes are going to be used for totally different tasks that I have little context into.

weebull 2y ago

I'd really like to get hold of a model for a modern CPU and properly analyse what all the performance features actually get us in terms of performance. - Branch prediction and speculative execution - Out of order execution - Massive physical register files and register renaming - Cache predictors - and many more I'm sure. Speculative execution is the big one for me, just because of the information leakage possible through it. It's there because you'd have to pause fetching new instructions until the result of a conditional branch is known, which has knock-on effects to instruction scheduling... But how big are these effects? Do some certain combinations of features supercharge or work against each other? I'm sure there's people looking at such things inside Intel and AMD, but it doesn't seem like there's much out there for public consumption.