Am I correct in my intuition that we are missing a lot of progress in computing due to the fact that we only have one GPGPU supplier with closed source software

Am I correct in my intuition that we are missing a lot of progress in computing due to the fact that we only have one GPGPU supplier with closed source software (CUDA). Aren't we at the point that quite a big part of what is computed by the CPU could and should move to the GPU? Or are gaming and BLAS-applications the 'only' ones that can easily be massively parallel? We have monster CPUs, GPUs and NVMe-drives all quite affordable but I don't feel it's all working together at speed. And anytime I'm looking into solutions that use say the GPU for statistics I'm quite far outside my comfort zone with regards to maturity of the software.