stumbled on this thread looking for tips- so far i haven't had success. i can load the model into memory by lowering the precision, I was OOMing on my 128GB RAM

stumbled on this thread looking for tips- so far i haven't had success. i can load the model into memory by lowering the precision, I was OOMing on my 128GB RAM, now im OOMing on loading the model into the 4090 on the 17th shard at 4 bit qtization so i would imagine theres another knob or two im missing to get this to run