> This will have a dramatic effect on performance. Agreed, but the effect is bigger on latency than throughput. If the problem set is bigger than available GPU

> This will have a dramatic effect on performance. Agreed, but the effect is bigger on latency than throughput. If the problem set is bigger than available GPU memory, the latency is at least partially hidden by streaming as the GPU can be kept busy while DMA is still transferring. The same thing applies if you will be doing more computations on the same data in the GPU so it won't have to be transferred back and forth.