3 ms·

Ok, well now that we’ve downvoted me below the visibility threshold, I was being sincere. And Altman did say that. I am not a hater. So. Maybe we could help ot

by b33j0r 3y ago

Ok, well now that we’ve downvoted me below the visibility threshold, I was being sincere. And Altman did say that. I am not a hater. So. Maybe we could help other people figure out why VRAM is maxing out. I think it has to do with various new platforms leaking memory. In my case, I suspect ollama and diffusers are not actually evicting VRAM. nvidia-smi shows it in one case, but I haven’t figured it out yet. Hey, my point remains. The models are going to get too expensive for me, personally, to run locally. I suspect we’ll default into subscriptions to APIs because the upgrade slope is too steep.

b33j0r 3y ago
Haha, for posterity, if anyone was interested in the things I mentioned. My specific frustrations came from leaving out torch.no_grad when running inference. I used copilot to refactor that, and it just didn’t put no_grad back, and I did not notice. I was uselessly recalculating all of my weights to /dev/null and waste heat. I daemonized my process to control it more tightly, but I still see way above expected vram allocation. Just keep in mind. It’s not like the models are going to get smaller, past the quantization limit. What, is there a quantization of retrievable information to 0 bits? ;)