Nvidia actually does support this via unified memory. It’s actually an amazing performance trick since you can avoid launching a bunch of kernels just to ferry

Nvidia actually does support this via unified memory. It’s actually an amazing performance trick since you can avoid launching a bunch of kernels just to ferry data back and forth. I did this on a a GTX 1080 for a school project to speed up random shuffling of large amounts of data. However, even without this feature, you can implement this sort of thing manually in most cases and you’re already being careful on a GPU to respect the cache (only working with one contiguous set of data of memory at a time). Really, we just need some good systems devs working on running these huge models.