5 ms·

Forgive a semi-relevant ramble: From a GPUs point of view, this isn't latency hiding, this is latency reduction. In GPU terminology, using pre-fetching and ca

Forgive a semi-relevant ramble: From a GPUs point of view, this isn't latency hiding, this is latency reduction. In GPU terminology, using pre-fetching and caching to reduce the latency for a read is called latency reduction. This approach is used heavily in CPUs. This is in contrast to the main approach used in GPUs, where you just switch threads if you're waiting on a read, dubbed latency hiding. GPUs are able to do this terribly quickly, so there's little worry about thread-switching overhead. It's analogous to 'simultaneous multithreading'/HyperThreading in CPUs. Latency-hiding is vital in GPUs, which handle massively parallel workloads and use graphics RAM, which has high throughput but poor latency. If you give a GPU a single-threaded workload, you can expect it to stall on reads far more than if you had a CPU handle the equivalent workload. (Disclaimer: my GPU knowledge is rather stale.)

jacquesm 6y ago
About your last paragraph: the RAM used in GPUs has pretty good latency when compared to that used in typical CPU applications, it's just that between that RAM and a CPU there are multiple layers of caching to make it appear as though the RAM is faster than it really is. CPUs that max out their caches crawl to a halt whereas GPUs crunching on datasets that are carefully laid out to maximize RAM bandwidth can process vast amounts of data, much more than a CPU if it had the equivalent workload. Giving a GPU a single threaded workload would be like having a sausage factory start up to make a single sausage, it's inappropriate use of the tool, which will always give poor results.
- MaxBarraclough 6y ago
  > RAM used in GPUs has pretty good latency when compared to that used in typical CPU applications I see [0] that what I should have said is this (taken from page 7): High throughput is obtained through the use of wide memory buses and specialized GDDR (graphics double data rate) memories that operate most efficiently when memory access granularities are large. Thus, GPU memory controllers must buffer, reorder, and then coalesce large numbers of memory requests to synthesize large operations that make efficient use of the memory system. Which roughly translates to higher latency from the point of view of any particular wavefront/warp, compared to the cache-heavy CPU approach. Of course, as the GPU's cores can easily switch between wavefronts, the latency from the wavefront's point of view isn't all that important; it's the overall throughput that counts. > Giving a GPU a single threaded workload would be like having a sausage factory start up to make a single sausage, it's inappropriate use of the tool, which will always give poor results. I don't like the analogy, as the single-threaded job could be enormous. Offloading a very small job onto a GPU is a bad idea because of orchestration overheads. Offloading a large single-threaded job onto a GPU is a bad idea because of poor core utilisation (you're using just one of the lanes of just one of the cores). Offloading a fixed-point-intensive job onto a GPU may be a bad idea because of inappropriate instruction-set. There's the matter of wavefront-coherent flow control. And if the job has chaotic memory access patterns, we're back to our original topic, and you'll get bitten by the wide memory bus. [0] https://www.cs.cmu.edu/afs/cs.cmu.edu/academic/class/15869-f11/www/readings/fatahalian08_cacm.pdf https://www.cs.cmu.edu/afs/cs.cmu.edu/academic/class/15869-f...
  - jacquesm 6y ago
    > I don't like the analogy, as the single-threaded job could be enormous. Anything that isn't an embarrassingly parallel workload is a bad match for a GPU. Use tools appropriately.
    - MaxBarraclough 6y ago
      That's a necessary condition, but not a sufficient one, for GPU-friendliness. An embarrassingly parallel workload on 64 bit integers, without wavefront flow-control coherence, and with highly chaotic memory-access patterns, might not fare well on GPU.
      - jacquesm 6y ago
        This is what makes GPU programming hard. Finding ways to shoehorn a sizeable fraction of a problem into a pattern that matches GPU capabilities is quite a bit of work. Very small code improvements can have a huge effect on throughput, and very small mistakes can just as fast tank your performance. Great care is required and a lot of knowledge of how the data access patterns translate into hits on the memory. I actually like that kind of cycle squeezing, much more than I like web development or other 'high level' plumbing.
        rbanffy 6y ago
        > I actually like that kind of cycle squeezing, much more than I like web development or other 'high level' plumbing. Fun problems are always the rarest.
derefr 6y ago
This terminology makes a lot of sense! Especially if you interpret "latency" as explicitly meaning "time the system as a whole does nothing, due to execution-units' pipelines being stalled, due to said units waiting on synchronous memory fetches." "Latency reduction", then, would mean "reducing pipeline stalls per execution unit", while "latency hiding" would mean "hiding the fact that some units' pipelines have stalled, by finding other work for their ALUs-and-such to do, while waiting for them to unstall."
rbanffy 6y ago
> GPUs are able to do this terribly quickly I wonder if now would be a proper time to revisit barrel processors.

jacquesm 6y ago

About your last paragraph: the RAM used in GPUs has pretty good latency when compared to that used in typical CPU applications, it's just that between that RAM and a CPU there are multiple layers of caching to make it appear as though the RAM is faster than it really is. CPUs that max out their caches crawl to a halt whereas GPUs crunching on datasets that are carefully laid out to maximize RAM bandwidth can process vast amounts of data, much more than a CPU if it had the equivalent workload. Giving a GPU a single threaded workload would be like having a sausage factory start up to make a single sausage, it's inappropriate use of the tool, which will always give poor results.

MaxBarraclough 6y ago
> RAM used in GPUs has pretty good latency when compared to that used in typical CPU applications I see [0] that what I should have said is this (taken from page 7): High throughput is obtained through the use of wide memory buses and specialized GDDR (graphics double data rate) memories that operate most efficiently when memory access granularities are large. Thus, GPU memory controllers must buffer, reorder, and then coalesce large numbers of memory requests to synthesize large operations that make efficient use of the memory system. Which roughly translates to higher latency from the point of view of any particular wavefront/warp, compared to the cache-heavy CPU approach. Of course, as the GPU's cores can easily switch between wavefronts, the latency from the wavefront's point of view isn't all that important; it's the overall throughput that counts. > Giving a GPU a single threaded workload would be like having a sausage factory start up to make a single sausage, it's inappropriate use of the tool, which will always give poor results. I don't like the analogy, as the single-threaded job could be enormous. Offloading a very small job onto a GPU is a bad idea because of orchestration overheads. Offloading a large single-threaded job onto a GPU is a bad idea because of poor core utilisation (you're using just one of the lanes of just one of the cores). Offloading a fixed-point-intensive job onto a GPU may be a bad idea because of inappropriate instruction-set. There's the matter of wavefront-coherent flow control. And if the job has chaotic memory access patterns, we're back to our original topic, and you'll get bitten by the wide memory bus. [0] https://www.cs.cmu.edu/afs/cs.cmu.edu/academic/class/15869-f11/www/readings/fatahalian08_cacm.pdf https://www.cs.cmu.edu/afs/cs.cmu.edu/academic/class/15869-f...
- jacquesm 6y ago
  > I don't like the analogy, as the single-threaded job could be enormous. Anything that isn't an embarrassingly parallel workload is a bad match for a GPU. Use tools appropriately.
  - MaxBarraclough 6y ago
    That's a necessary condition, but not a sufficient one, for GPU-friendliness. An embarrassingly parallel workload on 64 bit integers, without wavefront flow-control coherence, and with highly chaotic memory-access patterns, might not fare well on GPU.
    - jacquesm 6y ago
      This is what makes GPU programming hard. Finding ways to shoehorn a sizeable fraction of a problem into a pattern that matches GPU capabilities is quite a bit of work. Very small code improvements can have a huge effect on throughput, and very small mistakes can just as fast tank your performance. Great care is required and a lot of knowledge of how the data access patterns translate into hits on the memory. I actually like that kind of cycle squeezing, much more than I like web development or other 'high level' plumbing.
      - rbanffy 6y ago
        > I actually like that kind of cycle squeezing, much more than I like web development or other 'high level' plumbing. Fun problems are always the rarest.

derefr 6y ago

This terminology makes a lot of sense! Especially if you interpret "latency" as explicitly meaning "time the system as a whole does nothing, due to execution-units' pipelines being stalled, due to said units waiting on synchronous memory fetches." "Latency reduction", then, would mean "reducing pipeline stalls per execution unit", while "latency hiding" would mean "hiding the fact that some units' pipelines have stalled, by finding other work for their ALUs-and-such to do, while waiting for them to unstall."

rbanffy 6y ago

> GPUs are able to do this terribly quickly I wonder if now would be a proper time to revisit barrel processors.