6 ms·

This is a fairly ridiculous amount of performance, all things considered. It always seemed to me like SIMD/AVX/etc would eventually come for the GPU's lunch mo

This is a fairly ridiculous amount of performance, all things considered. It always seemed to me like SIMD/AVX/etc would eventually come for the GPU's lunch money... How many more product generations of "SIMD on steroids" before this is practically true? The latency factor is the biggest thing for me. The GPU is a turtle compared to CPU-bound techniques. I can see emerging applications for this in real-time/streaming where every millisecond counts.

thechao 4y ago
That's how we felt when we were writing the software rasterizer for Larrabee! The issue is that that 1.5TFLOP is probably way more power than the M1 GPU's ~2.5TFLOP. The second issue is that a SW rasterizer is going to spend ~50% of its budget emulating fixed function. So, now you're way more power, for 1/4 the perf (best case). Also, you can't run any other apps, and you're probably going to have bandwidth issues to the display controller. GPUs are an optimization to try to use the excess Moore's law we have to get to the ghost of Dennard's law.
- bob1029 4y ago
  I think a power/latency/perf tradeoff could be agreeable for certain applications. GPUs in the cloud are not exactly cheap. Many gaming experiences do not require nanite-level graphics. Building something that can reliably output reasonable-quality 3d graphics without relying on specific GPU technologies will give you a much broader realm to operate with. I believe something along this path is the solution for streaming gaming. I perceive the failure of Stadia, et. al. as being a consequence of trying to bolt streaming onto existing GPU-based, local gaming solutions. Build something from scratch with streaming/latency as a #1 priority, and you can dramatically expand the operational radius of each datacenter (e.g. ~100km per millisecond saved).
  - jvanderbot 4y ago
    Light is 300km/ms in a vacuum. Is it that much slower through switched fiber?
    - rkangel 4y ago
      Speed of light in glass is about 2/3 of the speed of light in a vacuum (refractive index of glass is around 1.5).
    - Someone 4y ago
      Signal speed in fiber is about ⅔ of that in vacuum (https://en.wikipedia.org/wiki/Optical_fiber#Refractive_index https://en.wikipedia.org/wiki/Optical_fiber#Refractive_index), but fiber won’t be straight-line between sender and receiver, light doesn’t move in a straight line inside the fiber, and the switched adds delays. https://www.pingdom.com/blog/theoretical-vs-real-world-speed-limit-of-ping/ https://www.pingdom.com/blog/theoretical-vs-real-world-speed...: “you should probably double the “ideal” response times shown above for a more realistic target to aim at“ So yes, ⅓ of light speed in vacuum seems a decent heuristic.
    - MobiusHorizons 4y ago
      Round trip latency matters here, which would get you down to 150km without any slowdown through fiber.
  - thechao 4y ago
    Well; except for the fact that the 1.5TFLOP quoted in the article is because of the AMX part. The actually useful throughput of the big-core is probably more like 35GFLOP peak. This compares to the 1–2TFLOP throughput of the GPU. The CPU is easily going to be 50–100x slower than the GPU. If you're talking full-screen Angry Birds with, say, a 2x average compositing, you're going to be fine on the CPU; but, energy- and jitter- wise you'll still be happier with the GPU, overall.
  - dotnet00 4y ago
    I feel like that's a somewhat out-of-touch interpretation, as Stadia failed largely because of Google's terrible reputation and the (completely valid) concerns from gamers about companies intending to turn even single player games into fragmented streaming platforms where the content is entirely dependent on the whims of the company (a fitting example being Google doing its thing and killing Stadia). They had no shortage of GPUs. NVIDIA's streaming service is doing relatively fine in comparison. They simply share a GPU between several users for anything that isn't demanding enough. They also get around some of the concerns about gaming being turned into another streaming style fragmented mess by not actually selling the games. You simply log into your account on Steam/GOG/whatever and play the games you already own as you might on a local PC. Additionally, "building something that can reliably output reasonable-quality 3d graphics without relying on specific GPU technologies" doesn't make much sense to me. If it's an accelerator designed to handle relatively modern 3d graphics, due to the programmability of a modern graphics pipeline it's effectively just a GPU. There aren't any underlying technologies that are required to be used as long as they can produce a similar output (mobile GPUs tend to have a different approach to how they implement the graphics pipeline compared to desktop GPUs for instance).
  - kllrnohj 4y ago
    well 2 things: 1) 1.5 TFLOPS is already less than the GPUs in most current phones. Like you're not exactly talking a big number here. You're talking 14 year old graphics (when desktop GPUs crossed the 1TFLOP mark) 2) And, this is the bigger issue, this AMX unit only does matrix multiplies. I'd be fascinated to see someone create a 3d renderer with only matrix multiplication. I'm sure it's possible but this 1.5 tflops ain't remotely comparable to a GPU's 1.5 tflops. Like if you tried to do "traditional" pixel shader rendering code on this unit you'd instantly be cutting that 1.5 tflops to 1/16th the performance - you're now at a theoretical peak of a mere 93 gflops of GPU-comparable performance.
    - brrrrrm 4y ago
      well, AMX does asynchronous outer products from byte-addressable buffers with accumulation. it's not like tensor cores or intel's AMX where the silicon literally only does a matrix multiplication. you can easily do efficient sliding window math from the buffers (convolutions) or sparse data loads (since the instructions run asynchronously from the CPU)
- hajile 4y ago
  There’s nothing to preclude a CPU from adding fixed function units they can call for certain things. Larabee 2 (one of the knights series) actually included these on the chip (though they were disabled in the final compute only product). You also forgot to mention the usability bit. Branching is bad for GPUs. They either take both branches or in the optimized version, they stall the pipeline until they get a result then attempt to optimize which units need to execute on which branch. The first is simply inefficient. The last is efficient, but impracticality slow on branchy code. On the power side, bloated x86 is hardly a good showcase for what such an architecture could do. I also suspect that an automatic thread manager like most modern GPUs use could also add a lot of value.
  - kllrnohj 4y ago
    > Branching is bad for GPUs. Branching is bad for wide SIMD workloads of which GPUs are the most common. But just shoving that SIMD under the control of a CPU instead won't fix your branching issues. You're still probably going to choose the path of just executing both branches & masking off the results.
  - thechao 4y ago
    I worked on the LRB through LRB3. I wrote the non-polygon pipeline for omatic (lines, points, billboards); later I switched to the shader compiler (masher) under mattomatic. Larrabee 1 (all of omatic) had fixed function on the turns: texture units, and pointer chasing. The later parts did not have GPU FF; they didn’t even have some of the rasterization opcodes (faddsets & fmad233). I seem to remember that LRB could test & jmp in 4 cycles, if you were careful. Since it was 4x barrel processed, those jmps were “free”. I moved to integrated GPUs, later. x86 is bloated, but LRB was not. Also, the decoder — even a big one like x86 — isn’t a major HW problem. I’d say x86’s memory hierarchy is more of an issue.
jasonwatkinspdx 4y ago
One interesting data point here is the Fugaku supercomputer is based around ARM's scalable vector stuff (basically Cray style variable length vectors vs short vector SIMD like AVX) and no gpu. Using HBM is a key enabler here. I'm not sure GPUs will be displaced, looking at the difficulties Larrabe had on the driver side, but I do think we'll see more flexible alternatives becoming popular.
fnordpiglet 4y ago
A typical goal is 60hz, which is 17k microseconds. My cursory research says as of 10 years ago a typical write/receive latency for an nvidia card, i7, pcie2.0 is 20 microseconds. That gives you a large budget despite the fact SIMD on chip is measured in cycles not microseconds. Inside the GPU you have a huge amount of space and resources to do highly specialized operations in vast concurrency, I.e., bandwidth for compute is huge and specialized. I don’t see how CPU’s or SOC will solve this without vastly increasing die sizes and heat and power consumption to be close to that of a GPU with all its cooling requirements and heavy power needs. That said I think the “good enough” metric is already there and unless you’re doing hardware ray tracing or extreme details at high resolutions you won’t need or care about a GPU any more. Latency though isn’t the issue. The times involved for human perception are long and not getting shorter.
- fragmede 4y ago
  Things have been "good enough" since 2012. But then VSCode and bigger webpages came along and suddenly a Core2Duo just doesn't cut it anymore. ML models need somewhere to run, locally, and both Apple and Google have dedicated hardware on smartphones for that. Support for bigger and bigger models (read GPU performance) in smaller and smaller packages is just the latest iteration of progress.
  - fnordpiglet 4y ago
    Yes I agree. Except I think real time ray tracing really is that much better and shifts the goal posts again.
zozbot234 4y ago
The GPU is more like a slow U-Haul truck, whereas the CPU is a super fast race car. Both have merit in their own domain. And GPU training is pretty solidly in the "slow and steady" camp.
- pletnes 4y ago
  Training in production, yes. Developing locally is still a thing for many reasons. More importantly, inference is more «sports car» - you want the app to stay interactive!
touisteur 4y ago
The GPU people are also reaching for simd and fixed matmul hw to increase perf. Tensor Cores (int, fp16, tf32 and even fp64 on A100) and the new DPX instructions. RT cores are a different kind of horse but still specialized for BVH traversal and ray-triangle intersection.
- theLiminator 4y ago
  We're reaching a point where CPUs are increasingly getting more specialized, and GPUs are becoming increasingly generalized. Going at improvements from both sides of the sandwich.
kllrnohj 4y ago
You'd need a fairly drastic shift in the memory architecture of CPUs for that. Not something unheard of, such as Intel's new Xeon Max beast with HBM 2e on the CPU module. But it's definitely not an issue of just throwing some big SIMD blocks onto the die & calling it a day. That is, after all, basically what AVX-512 is. And while it has it's place, it's also not eating anyone's lunch money. And also, as weird as it is, 1.5TFlops isn't actually that ridiculous. We had that performance 14 years ago at 150w with desktop GPUs. 14 years to reduce from 150w to what, 5w?, is cool but also honestly pretty par for the course is it not? Especially for a fixed-function block?
- r00fus 4y ago
  Aside from Apple's processors is 1.5TFlops in 5w possible with other archs?
  - kllrnohj 4y ago
    It sounds pretty similar to Qualcomm's hexagon ( https://www.anandtech.com/show/13680/snapdragon-855-going-into-detail/2 https://www.anandtech.com/show/13680/snapdragon-855-going-in... ) and probably whatever Google is doing with their Pixel SoC But otherwise that's less than what mobile GPUs can do, although those take more power they're not in entirely different ballparks either
    - deleted 4y ago
      [deleted]
- sliken 4y ago
  "You'd need a fairly drastic shift in the memory architecture of CPU". You mean like selling laptops (at 400GB/sec) and desktops (at 800GB/sec) with much improved memory systems. I don't want to give up SO-DIMMs for a few mm thinner laptop, but going from the intel/amd standard 70GB/sec to 400GB/sec is a pretty big incentive.
  - kllrnohj 4y ago

thechao 4y ago

That's how we felt when we were writing the software rasterizer for Larrabee! The issue is that that 1.5TFLOP is probably way more power than the M1 GPU's ~2.5TFLOP. The second issue is that a SW rasterizer is going to spend ~50% of its budget emulating fixed function. So, now you're way more power, for 1/4 the perf (best case). Also, you can't run any other apps, and you're probably going to have bandwidth issues to the display controller. GPUs are an optimization to try to use the excess Moore's law we have to get to the ghost of Dennard's law.

bob1029 4y ago
I think a power/latency/perf tradeoff could be agreeable for certain applications. GPUs in the cloud are not exactly cheap. Many gaming experiences do not require nanite-level graphics. Building something that can reliably output reasonable-quality 3d graphics without relying on specific GPU technologies will give you a much broader realm to operate with. I believe something along this path is the solution for streaming gaming. I perceive the failure of Stadia, et. al. as being a consequence of trying to bolt streaming onto existing GPU-based, local gaming solutions. Build something from scratch with streaming/latency as a #1 priority, and you can dramatically expand the operational radius of each datacenter (e.g. ~100km per millisecond saved).
- jvanderbot 4y ago
  Light is 300km/ms in a vacuum. Is it that much slower through switched fiber?
  - rkangel 4y ago
    Speed of light in glass is about 2/3 of the speed of light in a vacuum (refractive index of glass is around 1.5).
  - Someone 4y ago
    Signal speed in fiber is about ⅔ of that in vacuum (https://en.wikipedia.org/wiki/Optical_fiber#Refractive_index https://en.wikipedia.org/wiki/Optical_fiber#Refractive_index), but fiber won’t be straight-line between sender and receiver, light doesn’t move in a straight line inside the fiber, and the switched adds delays. https://www.pingdom.com/blog/theoretical-vs-real-world-speed-limit-of-ping/ https://www.pingdom.com/blog/theoretical-vs-real-world-speed...: “you should probably double the “ideal” response times shown above for a more realistic target to aim at“ So yes, ⅓ of light speed in vacuum seems a decent heuristic.
  - MobiusHorizons 4y ago
    Round trip latency matters here, which would get you down to 150km without any slowdown through fiber.
- thechao 4y ago
  Well; except for the fact that the 1.5TFLOP quoted in the article is because of the AMX part. The actually useful throughput of the big-core is probably more like 35GFLOP peak. This compares to the 1–2TFLOP throughput of the GPU. The CPU is easily going to be 50–100x slower than the GPU. If you're talking full-screen Angry Birds with, say, a 2x average compositing, you're going to be fine on the CPU; but, energy- and jitter- wise you'll still be happier with the GPU, overall.
- dotnet00 4y ago
  I feel like that's a somewhat out-of-touch interpretation, as Stadia failed largely because of Google's terrible reputation and the (completely valid) concerns from gamers about companies intending to turn even single player games into fragmented streaming platforms where the content is entirely dependent on the whims of the company (a fitting example being Google doing its thing and killing Stadia). They had no shortage of GPUs. NVIDIA's streaming service is doing relatively fine in comparison. They simply share a GPU between several users for anything that isn't demanding enough. They also get around some of the concerns about gaming being turned into another streaming style fragmented mess by not actually selling the games. You simply log into your account on Steam/GOG/whatever and play the games you already own as you might on a local PC. Additionally, "building something that can reliably output reasonable-quality 3d graphics without relying on specific GPU technologies" doesn't make much sense to me. If it's an accelerator designed to handle relatively modern 3d graphics, due to the programmability of a modern graphics pipeline it's effectively just a GPU. There aren't any underlying technologies that are required to be used as long as they can produce a similar output (mobile GPUs tend to have a different approach to how they implement the graphics pipeline compared to desktop GPUs for instance).
- kllrnohj 4y ago
  well 2 things: 1) 1.5 TFLOPS is already less than the GPUs in most current phones. Like you're not exactly talking a big number here. You're talking 14 year old graphics (when desktop GPUs crossed the 1TFLOP mark) 2) And, this is the bigger issue, this AMX unit only does matrix multiplies. I'd be fascinated to see someone create a 3d renderer with only matrix multiplication. I'm sure it's possible but this 1.5 tflops ain't remotely comparable to a GPU's 1.5 tflops. Like if you tried to do "traditional" pixel shader rendering code on this unit you'd instantly be cutting that 1.5 tflops to 1/16th the performance - you're now at a theoretical peak of a mere 93 gflops of GPU-comparable performance.
  - brrrrrm 4y ago
    well, AMX does asynchronous outer products from byte-addressable buffers with accumulation. it's not like tensor cores or intel's AMX where the silicon literally only does a matrix multiplication. you can easily do efficient sliding window math from the buffers (convolutions) or sparse data loads (since the instructions run asynchronously from the CPU)
hajile 4y ago
There’s nothing to preclude a CPU from adding fixed function units they can call for certain things. Larabee 2 (one of the knights series) actually included these on the chip (though they were disabled in the final compute only product). You also forgot to mention the usability bit. Branching is bad for GPUs. They either take both branches or in the optimized version, they stall the pipeline until they get a result then attempt to optimize which units need to execute on which branch. The first is simply inefficient. The last is efficient, but impracticality slow on branchy code. On the power side, bloated x86 is hardly a good showcase for what such an architecture could do. I also suspect that an automatic thread manager like most modern GPUs use could also add a lot of value.
- kllrnohj 4y ago
  > Branching is bad for GPUs. Branching is bad for wide SIMD workloads of which GPUs are the most common. But just shoving that SIMD under the control of a CPU instead won't fix your branching issues. You're still probably going to choose the path of just executing both branches & masking off the results.
- thechao 4y ago
  I worked on the LRB through LRB3. I wrote the non-polygon pipeline for omatic (lines, points, billboards); later I switched to the shader compiler (masher) under mattomatic. Larrabee 1 (all of omatic) had fixed function on the turns: texture units, and pointer chasing. The later parts did not have GPU FF; they didn’t even have some of the rasterization opcodes (faddsets & fmad233). I seem to remember that LRB could test & jmp in 4 cycles, if you were careful. Since it was 4x barrel processed, those jmps were “free”. I moved to integrated GPUs, later. x86 is bloated, but LRB was not. Also, the decoder — even a big one like x86 — isn’t a major HW problem. I’d say x86’s memory hierarchy is more of an issue.

jasonwatkinspdx 4y ago

One interesting data point here is the Fugaku supercomputer is based around ARM's scalable vector stuff (basically Cray style variable length vectors vs short vector SIMD like AVX) and no gpu. Using HBM is a key enabler here. I'm not sure GPUs will be displaced, looking at the difficulties Larrabe had on the driver side, but I do think we'll see more flexible alternatives becoming popular.

fnordpiglet 4y ago

A typical goal is 60hz, which is 17k microseconds. My cursory research says as of 10 years ago a typical write/receive latency for an nvidia card, i7, pcie2.0 is 20 microseconds. That gives you a large budget despite the fact SIMD on chip is measured in cycles not microseconds. Inside the GPU you have a huge amount of space and resources to do highly specialized operations in vast concurrency, I.e., bandwidth for compute is huge and specialized. I don’t see how CPU’s or SOC will solve this without vastly increasing die sizes and heat and power consumption to be close to that of a GPU with all its cooling requirements and heavy power needs. That said I think the “good enough” metric is already there and unless you’re doing hardware ray tracing or extreme details at high resolutions you won’t need or care about a GPU any more. Latency though isn’t the issue. The times involved for human perception are long and not getting shorter.

fragmede 4y ago
Things have been "good enough" since 2012. But then VSCode and bigger webpages came along and suddenly a Core2Duo just doesn't cut it anymore. ML models need somewhere to run, locally, and both Apple and Google have dedicated hardware on smartphones for that. Support for bigger and bigger models (read GPU performance) in smaller and smaller packages is just the latest iteration of progress.
- fnordpiglet 4y ago
  Yes I agree. Except I think real time ray tracing really is that much better and shifts the goal posts again.

zozbot234 4y ago

The GPU is more like a slow U-Haul truck, whereas the CPU is a super fast race car. Both have merit in their own domain. And GPU training is pretty solidly in the "slow and steady" camp.

pletnes 4y ago
Training in production, yes. Developing locally is still a thing for many reasons. More importantly, inference is more «sports car» - you want the app to stay interactive!

touisteur 4y ago

The GPU people are also reaching for simd and fixed matmul hw to increase perf. Tensor Cores (int, fp16, tf32 and even fp64 on A100) and the new DPX instructions. RT cores are a different kind of horse but still specialized for BVH traversal and ray-triangle intersection.

theLiminator 4y ago
We're reaching a point where CPUs are increasingly getting more specialized, and GPUs are becoming increasingly generalized. Going at improvements from both sides of the sandwich.

kllrnohj 4y ago

You'd need a fairly drastic shift in the memory architecture of CPUs for that. Not something unheard of, such as Intel's new Xeon Max beast with HBM 2e on the CPU module. But it's definitely not an issue of just throwing some big SIMD blocks onto the die & calling it a day. That is, after all, basically what AVX-512 is. And while it has it's place, it's also not eating anyone's lunch money. And also, as weird as it is, 1.5TFlops isn't actually that ridiculous. We had that performance 14 years ago at 150w with desktop GPUs. 14 years to reduce from 150w to what, 5w?, is cool but also honestly pretty par for the course is it not? Especially for a fixed-function block?

r00fus 4y ago
Aside from Apple's processors is 1.5TFlops in 5w possible with other archs?
- kllrnohj 4y ago
  It sounds pretty similar to Qualcomm's hexagon ( https://www.anandtech.com/show/13680/snapdragon-855-going-into-detail/2 https://www.anandtech.com/show/13680/snapdragon-855-going-in... ) and probably whatever Google is doing with their Pixel SoC But otherwise that's less than what mobile GPUs can do, although those take more power they're not in entirely different ballparks either
  - deleted 4y ago
    [deleted]
sliken 4y ago
"You'd need a fairly drastic shift in the memory architecture of CPU". You mean like selling laptops (at 400GB/sec) and desktops (at 800GB/sec) with much improved memory systems. I don't want to give up SO-DIMMs for a few mm thinner laptop, but going from the intel/amd standard 70GB/sec to 400GB/sec is a pretty big incentive.
- kllrnohj 4y ago