There's two killers, compatibility and launch overhead. CUDA is the most commonly-used compute language but it's NVIDIA-only. OpenCL is "universal" (sort of)

There's two killers, compatibility and launch overhead. CUDA is the most commonly-used compute language but it's NVIDIA-only. OpenCL is "universal" (sort of) but - being a general-purpose framework for heterogeneous compute rather than a task-specific GPGPU language - it's much more complex and lacks the libraries/etc that are available in CUDA. NVIDIA's OpenCL performance is all over the board, sometimes good sometimes fairly terrible (half of CUDA performance or less). Other users have nothing at all. So inherently - any GPU acceleration targets only a subset of your users. The other killer is launch overhead. Marshalling data into an appropriate structure, copying it over to the GPU, and copying it back takes a significant amount of time in many cases, independent of computation. This is really only viable when there is a significant amount of computation to be done, which inherently limits the number of applicable tasks. In fact ideally you would have data living right on the GPU all the time and you just make queries against it. But then you run into memory limitations - most GPUs have ~4 GB of VRAM and that's only enough VRAM for a couple applications at most (remember, we need to do a lot of work which means a fairly large quantity of data). And also since you want memory allocations to be contiguous you tend to allocate aggressively, which really means only 1-2 applications can use it. So practically speaking you now have a subset of your subset of users, the people who don't have another application running on their GPU at the same time. You can see how all this pushes it towards very niche tasks, being performed one at a time. The other approach is having an iGPU which can work in the same memory space as the CPU, like AMD does with HSA. That would lower the overhead quite a bit but right now that's a very limited subset of hardware. Practically speaking, right now the most viable general approach for vector processing is SSE/AVX, and you can see how poor the uptake of AVX has been. There's also major security concerns: GPUs aren't very conscious about things like memory segmentation or zeroing before re-allocation. As long as what you are doing is not blatantly illegal (writing to memory addresses that do not exist, etc) then it pretty much is just a single flat memory space that you can do anything with, and on non-ECC devices memory is only zeroed on a cold startup. Think "microcontroller" here, not memory like you'd get in desktop-land. https://arxiv.org/pdf/1305.7383.pdf https://arxiv.org/pdf/1305.7383.pdf https://petsymposium.org/2017/papers/issue2/paper10-2017-2-source.pdf https://petsymposium.org/2017/papers/issue2/paper10-2017-2-s... https://cryptome.org/2013/09/gpu-keylogger.pdf https://cryptome.org/2013/09/gpu-keylogger.pdf