I don't think the transfer to GPU memory is so much a bottleneck as it involves some latency. The PCIe bus provides a ton of bandwidth, it just takes longer to

I don't think the transfer to GPU memory is so much a bottleneck as it involves some latency. The PCIe bus provides a ton of bandwidth, it just takes longer to get stuff there from main memory than it does to get stuff to the CPU from main memory. So, wasn't trying to pick nits, just trying to say that if it's a latency issue, the penalty is roughly the same for small amounts of data as it is for big amounts, and if you have a huge amount of data that takes non-trivial amount of processing, the latency penalty is, as you said, worth it.