4 ms·

> GPUs have turned out to be more generally useful than expected. It's not only raw power. GPU programming comes with builtin asynchronism (waiting on promises

by p0nce 10y ago

> GPUs have turned out to be more generally useful than expected. It's not only raw power. GPU programming comes with builtin asynchronism (waiting on promises of computations, so you can hide latency) and a memory hierarchy (local, block, global, host memory) each one being more local and smaller than the next level. This makes caching much more efficient, since a lot of values don't need to escape the thread/warp/block. The problem on CPUs is that memory is flat and considered equal.

jacquesm 10y ago
> The problem on CPUs is that memory is flat and considered equal. CPUs have a similar memory hierarchy, the difference is that you're a context switch away from losing your preciously cached data. Even in GPUs memory is more often than not a constraint on the speed of computation, it is very easy to starve a GPU by accident. My personal favorite trick to see if I'm using my GPU effectively is to run a benchmark on it to max it out and register the power consumed with an e-meter. As long as I don't hit that power level in my own application there are still improvements to be had (and I almost never manage to reach the maximum but I'm happy with 60 to 80% or so of that). It's one of the dumbest debugging tricks ever but it is surprisingly effective at figuring out if I've laid out memory access patterns properly.
- dom0 10y ago
  Advantage of big, fat CPU cores: They have tons of performance counters :)
- phkahler 10y ago
  >> CPUs have a similar memory hierarchy, the difference is that you're a context switch away from losing your preciously cached data. That's an argument for more cores, each with it's own cache. It also suggests that the OS schedule all of the low CPU load tasks on the same core to prevent context switching on the big workloads. I'm not sure if that's how it's done today, but when my dual core is busy the CPU usage graphs tend to switch places regularly.
  - jacquesm 10y ago
    There are more levels of caching, some shared, some not. For instance, in a one particular quad core CPU all cores would have their own L1 cache, but cores 0 and 1 share the same L1 cache, and cores 2 and 3 another L1 cache. Then finally all cores share the same DRAM. So depending on whether your tasks have been 'pinned' to a certain core you might see a smaller or bigger hit when there is a context switch and a migration of a task that is not pinned to a certain core might flush L1 or both L1 and L2. Of course every architecture has its own quirks so to gage the impact you'd have to know exactly how things are set up in that particular machine. Another set-up (the one I'm writing this on) has 32 K per core L1 cache, 256K L2 cache and a 15 MB shared L3 cache. Migrating from one core to another on this machine is cheaper than on the four core example above.
- paulmd 10y ago
  That's pretty inaccurate as a rule. TDPs are specified for a standard graphics workload. Compute tasks can easily draw between 50% and 150% of the rated TDP. A given task may bottleneck on bandwidth or some other limit long before it hits a high level of TDP (which may fall into the generic 'unsuited for GPU' category and can never really be optimized). So apart from a generic inspiration to keep going further, this really does nothing for you. In fact you may still have further performance gains possible after this heuristic tells you to stop.
  - jacquesm 10y ago
    I usually have a pretty good idea how much utilization I should see, it's just a quick way to make sure I get what I'm aiming for and it has definitely helped me on a number of occasions when performance was disappointing to point out the GPU was simply starved. Memory access for a GPU is everything, even one detail wrong and you could easily get a 50% or more performance penalty so it is handy to have a quick confirmation that you're in the ballpark. It's not suitable as a detailed profiler.
- p0nce 10y ago
  What I wanted to say is ith the GPU, memory access have a known latency (you are writing to the L1 equivalent) without a statistical component. Also the nature of tasks submitted on the GPU (reified) makes it easier to optimize imho.

by p0nce 10y ago

jacquesm 10y ago

> The problem on CPUs is that memory is flat and considered equal. CPUs have a similar memory hierarchy, the difference is that you're a context switch away from losing your preciously cached data. Even in GPUs memory is more often than not a constraint on the speed of computation, it is very easy to starve a GPU by accident. My personal favorite trick to see if I'm using my GPU effectively is to run a benchmark on it to max it out and register the power consumed with an e-meter. As long as I don't hit that power level in my own application there are still improvements to be had (and I almost never manage to reach the maximum but I'm happy with 60 to 80% or so of that). It's one of the dumbest debugging tricks ever but it is surprisingly effective at figuring out if I've laid out memory access patterns properly.

dom0 10y ago
Advantage of big, fat CPU cores: They have tons of performance counters :)
phkahler 10y ago
>> CPUs have a similar memory hierarchy, the difference is that you're a context switch away from losing your preciously cached data. That's an argument for more cores, each with it's own cache. It also suggests that the OS schedule all of the low CPU load tasks on the same core to prevent context switching on the big workloads. I'm not sure if that's how it's done today, but when my dual core is busy the CPU usage graphs tend to switch places regularly.
- jacquesm 10y ago
  There are more levels of caching, some shared, some not. For instance, in a one particular quad core CPU all cores would have their own L1 cache, but cores 0 and 1 share the same L1 cache, and cores 2 and 3 another L1 cache. Then finally all cores share the same DRAM. So depending on whether your tasks have been 'pinned' to a certain core you might see a smaller or bigger hit when there is a context switch and a migration of a task that is not pinned to a certain core might flush L1 or both L1 and L2. Of course every architecture has its own quirks so to gage the impact you'd have to know exactly how things are set up in that particular machine. Another set-up (the one I'm writing this on) has 32 K per core L1 cache, 256K L2 cache and a 15 MB shared L3 cache. Migrating from one core to another on this machine is cheaper than on the four core example above.
paulmd 10y ago
That's pretty inaccurate as a rule. TDPs are specified for a standard graphics workload. Compute tasks can easily draw between 50% and 150% of the rated TDP. A given task may bottleneck on bandwidth or some other limit long before it hits a high level of TDP (which may fall into the generic 'unsuited for GPU' category and can never really be optimized). So apart from a generic inspiration to keep going further, this really does nothing for you. In fact you may still have further performance gains possible after this heuristic tells you to stop.
- jacquesm 10y ago
  I usually have a pretty good idea how much utilization I should see, it's just a quick way to make sure I get what I'm aiming for and it has definitely helped me on a number of occasions when performance was disappointing to point out the GPU was simply starved. Memory access for a GPU is everything, even one detail wrong and you could easily get a 50% or more performance penalty so it is handy to have a quick confirmation that you're in the ballpark. It's not suitable as a detailed profiler.
p0nce 10y ago
What I wanted to say is ith the GPU, memory access have a known latency (you are writing to the L1 equivalent) without a statistical component. Also the nature of tasks submitted on the GPU (reified) makes it easier to optimize imho.