4 ms·

The large MoE could be the DeepSeek V3 for people with just 128gb of (V)RAM.

• rahimnathwani 1y ago
  The smallest quantized version of the large MoE model on ollama is 143GB: https://ollama.com/library/qwen3:235b-a22b-q4_K_M https://ollama.com/library/qwen3:235b-a22b-q4_K_M Is there a smaller one?
  • whbrown 1y ago
    Running the 3 bit quant of https://huggingface.co/unsloth/Qwen3-235B-A22B-GGUF https://huggingface.co/unsloth/Qwen3-235B-A22B-GGUF now on a 128GB macbook.
    • Ecko123 1y ago
      [dead]
  • daemonologist 1y ago
    Smaller quantizations are possible [1], but I think you're right in that you wouldn't want to run anything substantially smaller than 128 GB. Single-GPU on 1x H200 (141 GB) might be feasible though (if you have some of those lying around...) [1] - https://huggingface.co/unsloth/Qwen3-235B-A22B-GGUF/tree/main https://huggingface.co/unsloth/Qwen3-235B-A22B-GGUF/tree/mai...