3 ms·
Smaller quantizations are possible [1], but I think you're right in that you wouldn't want to run anything substantially smaller than 128 GB. Single-GPU on 1x
by daemonologist 1y ago
Smaller quantizations are possible [1], but I think you're right in that you wouldn't want to run anything substantially smaller than 128 GB. Single-GPU on 1x H200 (141 GB) might be feasible though (if you have some of those lying around...)
[1] - https://huggingface.co/unsloth/Qwen3-235B-A22B-GGUF/tree/main https://huggingface.co/unsloth/Qwen3-235B-A22B-GGUF/tree/mai...