Well if you don't have enough VRAM to hold the whole model, you have to swap out to main RAM for every iteration, which for most home/local setups means once fo

Well if you don't have enough VRAM to hold the whole model, you have to swap out to main RAM for every iteration, which for most home/local setups means once for every single token generated. So having enough is kind of a minimum.