Yes, GPUs are able to directly pass messages and bulk data to each other these days. It's completely supported in CUDA, OpenCL, Vulkan, DirectX 12, etc. DirectX

Yes, GPUs are able to directly pass messages and bulk data to each other these days. It's completely supported in CUDA, OpenCL, Vulkan, DirectX 12, etc. DirectX calls it "Explicit Multi GPU". It's been around since GPUs transitioned to PCIe, it's just that both AMD and NVIDIA just locked it to the "professional" GPUs, or only used it under the hood of some other feature. AMD GPUs have used PCIe peer to peer copies for CrossFireX since the GCN Radeon GPUs (R9 290X and friends), that's why they don't need an external bridge. There's DirectGMA (AMD) and GPUDirect (NVIDIA) which allows PCIe drivers for other devices to directly copy to and from GPU VRAM. Other than driver support, nothing prevents PCs from having GPUs communicate directly with storage devices. The GPU would need to expose some of it's VRAM over PCIe (using a BAR window). The OS and the GPU driver would need to create some NVMe submission and completion queues either in system or VRAM that the GPU can be considered to own. You'd then issue NVMe operations where the backing pages point into the VRAM. The resizable bar support of the current Radeon and GeForce GPUs is a major step in this direction, although again, resizable bars were part of the original PCIe specification and has been available on server hardware since ~2008.