4 ms·
It can't run the latest models today - GLM-5.2 class models already need 1TB+ of RAM. ... but, the models that WILL run on 128GB (or 64GB or even 32GB) models
by simonw 4mo ago
It can't run the latest models today - GLM-5.2 class models already need 1TB+ of RAM.
... but, the models that WILL run on 128GB (or 64GB or even 32GB) models today are a huge improvement on the best models that would run in the same amount of memory six months ago.
- godwinsonsucks 4mo ago[dead]
- johndough 4mo ago> GLM-5.2 class models already need 1TB+ of RAM. If you quantize GLM-5.2 to 4 bit, you can do it in less than 500GB: https://huggingface.co/unsloth/GLM-5.2-GGUF https://huggingface.co/unsloth/GLM-5.2-GGUF (table on the right) If you find three finds that also have a 128GB MacBook, you can chain them together (the MacBooks, not your friends) and make it work. You could also run GLM-5.2 on a single MacBook if you stream the active parameters from disk, but even with speculative decoding, you'd probably only get in the order of 1 token per second, so this is not really practical for most applications.