3 ms·
How much system RAM are you running with? And I'm guessing it wouldn't hurt to have a fast SSD for disk offloading?
by lxe 4y ago
How much system RAM are you running with? And I'm guessing it wouldn't hurt to have a fast SSD for disk offloading?
- Miraste 4y ago128GB, but by turning on compression I managed to fit the whole thing on the GPU. I did try it off a mix of RAM and SSD as well, and it was slower but still usable. Presumably disk speed matters a lot.
- lxe 4y agoWell just got some more sticks. While I wait for RAM to arrive, will try with compress_weight and compress_cache. If you're in any discord or any other space where people are tinkering with this, would love to join!
- lxe 4y agoWith compression, was able to get 30b to run on 3090 with '100 0'! Let me see if I can tweak the prompt a bit and make it come alive...