The "third killer" is parallelism. Not all workloads have available parallelism to make the GPU effective - and some have the parallelism only in theory. I work

The "third killer" is parallelism. Not all workloads have available parallelism to make the GPU effective - and some have the parallelism only in theory. I work in regular expression acceleration (obplug: Hyperscan) and have done regex-on-GPU in the past. One of the problems was that the easiest way to get parallelism for regex was to throw a lot of data at the problem, so you could have your 16K CUDA threads. However, hitting peak throughput performance with 2MB input buffers is a bit of a problem if you typical workload is 64-1500 bytes and you want the results as quickly as possible. The launch overhead issue is also a real thing, despite some of the protestations in responses that it's not. The problem is that some computationally expensive operations are expensive and launched frequently from something non-GPU. So just because you 'have a lot of accelerable work to do in aggregate' doesn't mean that it will be structured nicely to accelerate. We had that problem as well; you might have tons of overall work that all parallelizes quite nicely in aggregate, but it's dribbling in from the network card or CPU in little units.