3 ms·

Yes I usually run Unsloth models, however you are linking to the big model now (355B-A32B), which I can't run on my consumer hardware. The flash model in this

Yes I usually run Unsloth models, however you are linking to the big model now (355B-A32B), which I can't run on my consumer hardware. The flash model in this thread is more than 10x smaller (30B).

latchkey 9mo ago
There are a bunch of 4bit quants in the GGUF link and the 0xSero has some smaller stuff too. Might still be too big and you'll need to ungpu poor yourself.
- disiplus 9mo ago
  yeah there is no way to run 4.7 on a 32g vram this flash is something that im also waiting to try later tonight
  - omneity 9mo ago
    Why not? Run it with vLLM latest and enable 4bit quantization with bnb, and it will quantize the original safetensors on the fly and fit your vram.
    - disiplus 9mo ago
      because how huge glm 4.7 is https://huggingface.co/zai-org/GLM-4.7 https://huggingface.co/zai-org/GLM-4.7
      - omneity 9mo ago
        Except this is GLM 4.7 Flash which has 32B total params, 3B active. It should fit with a decent context window of 40k or so in 20GB of ram at 4b weights quantization and you can save even more by quantizing the activations and KV cache to 8bit.
        disiplus 9mo ago
        yes, but the parrent link was to the big glm 4.7 that had a bunch of ggufs, the new one at the point of posting did not, nor does it now. im waiting for unsloth guys for the 4.7 flash
a_e_k 9mo ago
When the Unsloth quant of the flash model does appear, it should show up as unsloth/... on this page: https://huggingface.co/models?other=base_model:quantized:zai-org/GLM-4.7-Flash https://huggingface.co/models?other=base_model:quantized:zai... Probably as: https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF
- homarp 9mo ago
  it'a a new architecture. Not yet implemented in llama.cpp issue to follow: https://github.com/ggml-org/llama.cpp/issues/18931 https://github.com/ggml-org/llama.cpp/issues/18931
- dumbmrblah 9mo ago
  One thing to consider is that this version is a new architecture, so it’ll take time for Llama CPP to get updated. Similar to how it was with Qwen Next.
  - cristoperb 9mo ago
    Apparently it is the same as the DeepseekV3 architecture and already supported by llama.cpp once the new name is added. Here's the PR: https://github.com/ggml-org/llama.cpp/pull/18936 https://github.com/ggml-org/llama.cpp/pull/18936
    - khimaros 9mo ago
      has been merged

Yes I usually run Unsloth models, however you are linking to the big model now (355B-A32B), which I can't run on my consumer hardware. The flash model in this thread is more than 10x smaller (30B).

latchkey 9mo ago

There are a bunch of 4bit quants in the GGUF link and the 0xSero has some smaller stuff too. Might still be too big and you'll need to ungpu poor yourself.

disiplus 9mo ago
yeah there is no way to run 4.7 on a 32g vram this flash is something that im also waiting to try later tonight
- omneity 9mo ago
  Why not? Run it with vLLM latest and enable 4bit quantization with bnb, and it will quantize the original safetensors on the fly and fit your vram.
  - disiplus 9mo ago
    because how huge glm 4.7 is https://huggingface.co/zai-org/GLM-4.7 https://huggingface.co/zai-org/GLM-4.7
    - omneity 9mo ago
      Except this is GLM 4.7 Flash which has 32B total params, 3B active. It should fit with a decent context window of 40k or so in 20GB of ram at 4b weights quantization and you can save even more by quantizing the activations and KV cache to 8bit.
      - disiplus 9mo ago
        yes, but the parrent link was to the big glm 4.7 that had a bunch of ggufs, the new one at the point of posting did not, nor does it now. im waiting for unsloth guys for the 4.7 flash

a_e_k 9mo ago

When the Unsloth quant of the flash model does appear, it should show up as unsloth/... on this page: https://huggingface.co/models?other=base_model:quantized:zai-org/GLM-4.7-Flash https://huggingface.co/models?other=base_model:quantized:zai... Probably as: https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF

homarp 9mo ago
it'a a new architecture. Not yet implemented in llama.cpp issue to follow: https://github.com/ggml-org/llama.cpp/issues/18931 https://github.com/ggml-org/llama.cpp/issues/18931
dumbmrblah 9mo ago
One thing to consider is that this version is a new architecture, so it’ll take time for Llama CPP to get updated. Similar to how it was with Qwen Next.
- cristoperb 9mo ago
  Apparently it is the same as the DeepseekV3 architecture and already supported by llama.cpp once the new name is added. Here's the PR: https://github.com/ggml-org/llama.cpp/pull/18936 https://github.com/ggml-org/llama.cpp/pull/18936
  - khimaros 9mo ago
    has been merged