The problem I see with trying to emulate NumPy with a GPU accelerated version is that the communication overhead to the GPU is so high that you are losing a lot

The problem I see with trying to emulate NumPy with a GPU accelerated version is that the communication overhead to the GPU is so high that you are losing a lot of performance as opposed to something like TensorFlow. It takes a couple of microseconds just to start a kernel much less the time it takes to transfer data back and forth.