I agree. It's a huge issue for software dev companies as it is related to a non-negligible effort to keep your software working efficiently with a large array o

I agree. It's a huge issue for software dev companies as it is related to a non-negligible effort to keep your software working efficiently with a large array of GPUs. One potential workaround are libraries such as KOKKOS [0]. Of course it's never going to be as fast as pure optimized CUDA, but if it gets you half the way there, you'll still be faster than using the CPU. Particularly when considering that some of the new CPUs seem to have scaling issues themselves [1] [0] https://github.com/kokkos/kokkos https://github.com/kokkos/kokkos [1] https://www.pugetsystems.com/labs/hpc/Threadripper-3990x-64-core-Parallel-Scaling-1680/ https://www.pugetsystems.com/labs/hpc/Threadripper-3990x-64-...