Because VRAM is faster than main memory, and a lot faster than ferrying data over PCIe. Nvidia supports reading data directly from main memory. They introduced

Because VRAM is faster than main memory, and a lot faster than ferrying data over PCIe. Nvidia supports reading data directly from main memory. They introduced it around 2013 as Unified Memory, and it occasionally gets updates that make it more usable. It's probably faster than loading each layer into VRAM, computing it, discarding the weights and loading the next layer; but it's still a lot slower than having the weights in VRAM and often slower than just doing the work on the CPU.