PCIe can be a limitation, but there are a lot of ways to amortize the latency of copying data to/from the GPU. Overlapping transfers with kernels, direct load/s

PCIe can be a limitation, but there are a lot of ways to amortize the latency of copying data to/from the GPU. Overlapping transfers with kernels, direct load/store from system memory to a kernel, multiple kernels running on the chip at the same time--there are a lot of things you can do. But in general, you're looking at a runtime of at least a few hundred microseconds before you're going to be able to get a benefit from the GPU.