> CUDA/OpenCL have both something that's too simple(SVM / Unified Memory) It’s sadly the only choice around for maximum code reuse… with then manually optimisi

> CUDA/OpenCL have both something that's too simple(SVM / Unified Memory) It’s sadly the only choice around for maximum code reuse… with then manually optimising the parts that need to be later through profiling, as developer time is quite limited. On systems where the CPU and GPU are coherent such SoCs w/ CPU and GPU on the same die, Infinity Fabric-linked GPUs or NVlink linked ones (Power9/Grace), that option is also quite attractive. The memory manager measures the frequency of access from the CPU and GPU to (try to) locate where a buffer should be stored or moved dynamically. However, on PCIe linked GPUs, reliance on heuristics is more necessary to reach good perf levels. You can also ask to have your memory allocation migrated to the GPU on-demand with cudaMemPrefetchAsync (finally implemented in ROCm 4.5 too, which has unified memory on AMD cards) before doing a kernel launch for the data that you’ll use. (See https://on-demand.gputechconf.com/gtc/2017/presentation/s7285-nikolay-sakharnykh-unified-memory-on-pascal-and-volta.pdf https://on-demand.gputechconf.com/gtc/2017/presentation/s728... for the changes that happened in Volta on that front) The goal is to make the use of regular cudaMalloc/hipMalloc not necessary for good performance in a wide variety of scenarios. The explicit storage location APIs will stay available for when more optimisation is necessary.