3 ms·
Very interested to hear a similar answer, but if anyone has tried running it locally on high-end consumer grade hardware, eg an Nvidia 4090?
by rybosome 3y ago
Very interested to hear a similar answer, but if anyone has tried running it locally on high-end consumer grade hardware, eg an Nvidia 4090?
- mdcurrent 3y agostumbled on this thread looking for tips- so far i haven't had success. i can load the model into memory by lowering the precision, I was OOMing on my 128GB RAM, now im OOMing on loading the model into the 4090 on the 17th shard at 4 bit qtization so i would imagine theres another knob or two im missing to get this to run
- rybosome 3y agoThanks for your insight. Sounds like it’s quite a challenge, 128GB of RAM and a 4090 are pretty beefy for a consumer PC.