5 ms·

The constant widening of the SIMD registers is an interesting trend to watch. MMX started off as 64-bit, and now AVX-512 is 8 times that. With many cores and a

The constant widening of the SIMD registers is an interesting trend to watch. MMX started off as 64-bit, and now AVX-512 is 8 times that. With many cores and a couple of SIMD units in each core, aren't CPUs becoming ever more suitable to handle graphic workloads that were once reasonable only for GPUs?

deleted 12y ago
[deleted]
vardump 12y ago
GPUs are still about 5x ahead compared to CPUs. CPUs lack texture lookup hardware, that's the biggest catch right now. AVX2 gives gather support (one instruction, up to 8x independent loads to a single vector register), but currently on Haswell there's no performance benefit compared to doing the memory loads individually. GPU memory (GDDR5) has also many times more bandwidth at expense of 10x higher latency compared to CPU. Currently you put the workload that requires low latency on a CPU and embarrassingly parallel "non-branchy" workload on a GPU. Both CPUs and GPUs gain about 20% performance per year per core. I'm no expert on CPU and GPU memory architectures, but both seem to be headed towards integrated 3D-stacked eDRAM memory. This can provide up to several TBps reasonably low latency bandwidth.
Symmetry 12y ago
Everything vardump said is true, but additionally there's the consideration of what isn't there as well as what is there. By not having an out of order execution setup or a broad forwarding network a GPU core is able to be much smaller than a CPU core of equivalent throughput. By "core" here I mean something capable of independently issuing instructions and executing them, not calling each execution unit a "core" the way Nvidia does (though with SIMT that isn't as crazy as it might look at first). That means you can pack more cores on a chip and use less power for a given amount of work, since power consumption is roughly proportional to the number of transistors used. If we eventually start doing graphics on CPUs it will because we move to ray tracing or other sorts of algorithms that GPUs are bad at.
- ANTSANTS 12y ago
  >If we eventually start doing graphics on CPUs it will because we move to ray tracing or other sorts of algorithms that GPUs are bad at. Even for ray tracing, I think you'll get better results on a modern GPU. Each individual processing element may not be anywhere near as good at dealing with the constant branch prediction failures inherent in ray tracing algorithms as modern CPU cores are, but there are thousands of them for every CPU core you have, and millions of completely independent samples to compute in a raytraced image. But I suspect that by the time that realtime raytracting becomes viable, the HSA approach I outlined above will be viable as well, and that will be the way to go.
- Symmetry 12y ago
  One more thing. GPUs don't have to worry about precise interrupts like if you try to load a piece of memory only to find that you have to page it in or such. If they're in a memory protected setup where that's even possible it'll be the CPU who has to deal with the mess, the GPU can just halt while that's taken care of. This makes SIMT easy. SIMT is "Single Intstruction Multiple Threads" which is sort of like SIMD but better. You have an instruction come in which is distributed to multiple execution lanes each of which has a hopper of instructions it can draw from. Each lane then executes those instructions independently and if the lane notices that the instruction has been predicated out it just ignores it. Or maybe if one lane is behind it can give the instruction to its neighbor which isn't. THe fact that you don't have to be able to drop everything in a single cycle and pretend you were executing perfectly in order gives the hardware a lot of flexiblity, and all of this complexity is only O(n) with the number of executions instead of O(n^2) as with a typical OoO setup. The need to have precise exceptions involving SIMD instructions means that this isn't a simple thing for a regular CPU core to add to its SIMD units. The fact that each lane is making decisions about when to execute the instructions it has been issued are why some people refer to the lanes as "cores". I don't, because what they're doing isn't any more complicated than the 8 reservation stations in a typical Tomasulo algorithm OoO CPU core would be doing even if they are smarter than a SIMD lane. With GPUs it makes more sense to break down "cores" by instruction issue, in which case a high end GPU would have dozens rather than thousands of cores.
  - 12y ago
ANTSANTS 12y ago
I had a pretty big post going, but vardump and Symmetry conveyed the gist of what I was going for much more succinctly, so I'll summarize:[1] GPU cores are tiny because the problems they deal with are "embarrassingly parallel", trivially solved by throwing more cores at the problem. You make the cores as simple as possible so you can have thousands of them on a chip. Modern GPUs don't even have SIMD units per core any more; both NVidia and AMD are completely scalar now. You'd think that graphics would be the perfect scenario for SIMD, since shaders spend so much time dealing with 3 and 4D vectors, transformation matrices, colors, and so on, but it worked out that the gain in throughput and instruction density per-core was outweighed by the power, heat, and die cost of having parts of those thousands of SIMD units sitting idle while working on data that doesn't take up a whole SIMD register. And because context switches are much rarer on GPUs than CPUs, they can have extremely deep pipelines that push compute efficiency even further, at the expense of context switch latency. CPU cores are, if I may, "fuckhuge", because by and large they can't solve their problems by throwing more cores at them. They take on problems that are inherently serial and branch heavy, like compiling a program or optimally compressing a large file, and throw bigger cores at them (out-of-order execution, branch prediction, speculative execution, multiple ALUs per core, instruction schedulers that exploit the parallelism hidden in the serial instruction stream, large register files only visible to the microarchitecture, etc) while maintaining a fairly short pipeline so that branch prediction failures and context switches don't take too long to recover from. SIMD fits in well here, because the cost of bigger and bigger ALUs is pretty much insignificant compared to all the other hardware that goes into a high-end CPU core. It can be a pain to optimize for, but it's great for middle-ground tasks that need both heavy parallel and serial/branch-heavy computing resources with little latency between the two, like video compression. So you can't make one as good at solving the tasks of the other without making it worse at the task it's already intended to do. They tried to do this with Larabee, which was, no exaggeration, a few hundred Pentiums on a chip. Very interesting, but unsuccessful in the market, because it was an expensive niche product that was worse than either a CPU or GPU for the majority of their respectful workloads. What's really interesting, however, is the direction that AMD is taking. Rather than pushing SIMD like Intel, they're trying with their "heterogeneous systems architecture" movement to break down the communication barrier between CPUs and GPUs so that sharing data between the two is as easy as passing a pointer. SIMD is still great on CPUs for dealing with higher level primitives like vectors, matrices, or image macroblocks, but I can easily see the aforementioned middle-ground shifting to CPU threads working in close concert with GPU worker threads, as opposed to the mostly hands-off, one way street common in i.e. 3D rendering today. [1] On second thought, this didn't turn out to be much of a summary, did it?
mbell 12y ago
From what I could gather from building and testing a new desktop a couple months ago AVX instruction loads will trip CPU voltage levels to increase over 'normal' and generate a ton of heat as a result, often throttling if you aren't using aftermarket cooling. I would guess that AVX has a long way to go in terms of power efficiency before you'll see any large use of it.

deleted 12y ago

[deleted]

vardump 12y ago

GPUs are still about 5x ahead compared to CPUs. CPUs lack texture lookup hardware, that's the biggest catch right now. AVX2 gives gather support (one instruction, up to 8x independent loads to a single vector register), but currently on Haswell there's no performance benefit compared to doing the memory loads individually. GPU memory (GDDR5) has also many times more bandwidth at expense of 10x higher latency compared to CPU. Currently you put the workload that requires low latency on a CPU and embarrassingly parallel "non-branchy" workload on a GPU. Both CPUs and GPUs gain about 20% performance per year per core. I'm no expert on CPU and GPU memory architectures, but both seem to be headed towards integrated 3D-stacked eDRAM memory. This can provide up to several TBps reasonably low latency bandwidth.

Symmetry 12y ago

Everything vardump said is true, but additionally there's the consideration of what isn't there as well as what is there. By not having an out of order execution setup or a broad forwarding network a GPU core is able to be much smaller than a CPU core of equivalent throughput. By "core" here I mean something capable of independently issuing instructions and executing them, not calling each execution unit a "core" the way Nvidia does (though with SIMT that isn't as crazy as it might look at first). That means you can pack more cores on a chip and use less power for a given amount of work, since power consumption is roughly proportional to the number of transistors used. If we eventually start doing graphics on CPUs it will because we move to ray tracing or other sorts of algorithms that GPUs are bad at.

ANTSANTS 12y ago
>If we eventually start doing graphics on CPUs it will because we move to ray tracing or other sorts of algorithms that GPUs are bad at. Even for ray tracing, I think you'll get better results on a modern GPU. Each individual processing element may not be anywhere near as good at dealing with the constant branch prediction failures inherent in ray tracing algorithms as modern CPU cores are, but there are thousands of them for every CPU core you have, and millions of completely independent samples to compute in a raytraced image. But I suspect that by the time that realtime raytracting becomes viable, the HSA approach I outlined above will be viable as well, and that will be the way to go.
Symmetry 12y ago
One more thing. GPUs don't have to worry about precise interrupts like if you try to load a piece of memory only to find that you have to page it in or such. If they're in a memory protected setup where that's even possible it'll be the CPU who has to deal with the mess, the GPU can just halt while that's taken care of. This makes SIMT easy. SIMT is "Single Intstruction Multiple Threads" which is sort of like SIMD but better. You have an instruction come in which is distributed to multiple execution lanes each of which has a hopper of instructions it can draw from. Each lane then executes those instructions independently and if the lane notices that the instruction has been predicated out it just ignores it. Or maybe if one lane is behind it can give the instruction to its neighbor which isn't. THe fact that you don't have to be able to drop everything in a single cycle and pretend you were executing perfectly in order gives the hardware a lot of flexiblity, and all of this complexity is only O(n) with the number of executions instead of O(n^2) as with a typical OoO setup. The need to have precise exceptions involving SIMD instructions means that this isn't a simple thing for a regular CPU core to add to its SIMD units. The fact that each lane is making decisions about when to execute the instructions it has been issued are why some people refer to the lanes as "cores". I don't, because what they're doing isn't any more complicated than the 8 reservation stations in a typical Tomasulo algorithm OoO CPU core would be doing even if they are smarter than a SIMD lane. With GPUs it makes more sense to break down "cores" by instruction issue, in which case a high end GPU would have dozens rather than thousands of cores.
- 12y ago

ANTSANTS 12y ago

I had a pretty big post going, but vardump and Symmetry conveyed the gist of what I was going for much more succinctly, so I'll summarize:[1] GPU cores are tiny because the problems they deal with are "embarrassingly parallel", trivially solved by throwing more cores at the problem. You make the cores as simple as possible so you can have thousands of them on a chip. Modern GPUs don't even have SIMD units per core any more; both NVidia and AMD are completely scalar now. You'd think that graphics would be the perfect scenario for SIMD, since shaders spend so much time dealing with 3 and 4D vectors, transformation matrices, colors, and so on, but it worked out that the gain in throughput and instruction density per-core was outweighed by the power, heat, and die cost of having parts of those thousands of SIMD units sitting idle while working on data that doesn't take up a whole SIMD register. And because context switches are much rarer on GPUs than CPUs, they can have extremely deep pipelines that push compute efficiency even further, at the expense of context switch latency. CPU cores are, if I may, "fuckhuge", because by and large they can't solve their problems by throwing more cores at them. They take on problems that are inherently serial and branch heavy, like compiling a program or optimally compressing a large file, and throw bigger cores at them (out-of-order execution, branch prediction, speculative execution, multiple ALUs per core, instruction schedulers that exploit the parallelism hidden in the serial instruction stream, large register files only visible to the microarchitecture, etc) while maintaining a fairly short pipeline so that branch prediction failures and context switches don't take too long to recover from. SIMD fits in well here, because the cost of bigger and bigger ALUs is pretty much insignificant compared to all the other hardware that goes into a high-end CPU core. It can be a pain to optimize for, but it's great for middle-ground tasks that need both heavy parallel and serial/branch-heavy computing resources with little latency between the two, like video compression. So you can't make one as good at solving the tasks of the other without making it worse at the task it's already intended to do. They tried to do this with Larabee, which was, no exaggeration, a few hundred Pentiums on a chip. Very interesting, but unsuccessful in the market, because it was an expensive niche product that was worse than either a CPU or GPU for the majority of their respectful workloads. What's really interesting, however, is the direction that AMD is taking. Rather than pushing SIMD like Intel, they're trying with their "heterogeneous systems architecture" movement to break down the communication barrier between CPUs and GPUs so that sharing data between the two is as easy as passing a pointer. SIMD is still great on CPUs for dealing with higher level primitives like vectors, matrices, or image macroblocks, but I can easily see the aforementioned middle-ground shifting to CPU threads working in close concert with GPU worker threads, as opposed to the mostly hands-off, one way street common in i.e. 3D rendering today. [1] On second thought, this didn't turn out to be much of a summary, did it?

mbell 12y ago

From what I could gather from building and testing a new desktop a couple months ago AVX instruction loads will trip CPU voltage levels to increase over 'normal' and generate a ton of heat as a result, often throttling if you aren't using aftermarket cooling. I would guess that AVX has a long way to go in terms of power efficiency before you'll see any large use of it.