Haha, for posterity, if anyone was interested in the things I mentioned. My specific frustrations came from leaving out torch.no_grad when running inference. I

Haha, for posterity, if anyone was interested in the things I mentioned. My specific frustrations came from leaving out torch.no_grad when running inference. I used copilot to refactor that, and it just didn’t put no_grad back, and I did not notice. I was uselessly recalculating all of my weights to /dev/null and waste heat. I daemonized my process to control it more tightly, but I still see way above expected vram allocation. Just keep in mind. It’s not like the models are going to get smaller, past the quantization limit. What, is there a quantization of retrievable information to 0 bits? ;)