3 ms·
As someone with a basement rig of 6x 3090s, not really. It's quite slow, as with that many params (685B) it's offloading basically all of it into system RAM. I
by halyconWays 11mo ago
As someone with a basement rig of 6x 3090s, not really. It's quite slow, as with that many params (685B) it's offloading basically all of it into system RAM. I limit myself to models with <144B params, then it's quite an enjoyable experience. GLM 4.5 Air has been great in particular
- lostmsu 11mo agoDid you find it better than GPT-OSS 120B? The public rankings are contradictory.
- halyconWays 10mo agoI haven't used GPT-OSS 120B, or other GPT-OSS models, and I mostly go on personal recommendations rather than benchmarks directly.