4 ms·

The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models https://huggingface.co/prism-ml/models I've tried a couple in

by simonw 3mo ago

The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models https://huggingface.co/prism-ml/models I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.

trollbridge 3mo ago
Didn't work for me in Unsloth, but it will probably be fixed in a day or two when the next batch of updates comes out.
- dofm 3mo ago
  Have Prism-ML upstreamed their forked code?
  - lta 3mo ago
    Yeah it's a llama.cpp fork and it's linked everywhere on their repo
    - dofm 3mo ago
      Of course — I compiled it and ran it yesterday. Not aware if they have a PR in place with the main project, though as I recall I ran the last Bonsai model without needing it so I guess there might be plans.
bansaltushar 3mo ago
Depending on which model you're running, you might need to use the custom forks. Details are here -> https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README.md#upstream-status-for-ternary https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README....
- motbus3 3mo ago
  I spent quite sometime trying to install their tools and nothing really worked. I used these repos you shared but the dependencies all fail on mac
  - PrismML 3mo ago
    If you can share details on where it is failing, we'd love to help fix. You can also join Discord to communicate with us directly http://discord.gg/prismml http://discord.gg/prismml
    - motbus3 3mo ago
      I did not know you guys would be watching. For sure. Let me do that tomorrow when I turn it on again :) I am happy to see the message. Thanks!
SwellJoe 3mo ago
I got their previous model working in their custom fork of llama.cpp (https://github.com/PrismML-Eng/llama.cpp https://github.com/PrismML-Eng/llama.cpp). I haven't tried this one yet, but will find some time to benchmark it sometime this week. Though this says mainline llama.cpp has their patches for Metal and CPU backends, so maybe it's simply "use current llama.cpp" if you have a Mac or fast enough CPU/memor to use the CPU backend.
dofm 3mo ago
Not sure if there's any way to run Prism's fork of llama.cpp inside LM Studio. The fork runs fine for me. The model gets very notably stuck in a reasoning loop on one of my simple tests, though it might be that it has the same issues with setting reasoning effort high. On my M1 Max I still think the MoE Qwen 3.6 and Gemma 4 models are the best options. And I am far from convinced that the 35B is actually worse; it gets stuck in reasoning loops much less often than 27B in my experience.
try-working 3mo ago
I downloaded two of the official ones in LM Studio, both 3.6gb, and neither loaded.
anon373839 3mo ago
I can report that it's working in oMLX. I've been experimenting with the ternary one; it is quite an impressive model! I've been grilling it on some deep learning/computer vision stuff and it's aced everything so far. Responses are thorough, accurate, sophisticated. General knowledge outside of CS doesn't seem as robust, which I expected. Honestly, I don't think the examples in the blog post do it justice.
- ch_sm 3mo ago
  Nice, I tried it too with oMLX — agreed, it seems very capable for coding! Was slightly underwhelmed by performance though. I got about ~24 t/s on the ternary version on my M2 Max 64. That’s quite a bit slower than Qwen A3B 35B (4 bit unsloth). How was perf for you?
  - anon373839 3mo ago
    I had roughly the same performance on M3 Max 64: ~30 tokens/sec. Which is not terrible, but with the latest Lightning MTP optimization I am getting ~100 tokens/sec with Qwen 3.6 35B-A3B.
    - ch_sm 3mo ago
      WHAT. that‘s amazing. thanks so much for sharing, i‘ve been looking for ways to speed a3b up for days. It‘s 11pm here but i‘ll try this right now
      - anon373839 3mo ago
        I’m using the new oQe quants in case that matters.
        ch_sm 3mo ago
        i see! would you mind sharing your model string?

by simonw 3mo ago

trollbridge 3mo ago

Didn't work for me in Unsloth, but it will probably be fixed in a day or two when the next batch of updates comes out.

dofm 3mo ago
Have Prism-ML upstreamed their forked code?
- lta 3mo ago
  Yeah it's a llama.cpp fork and it's linked everywhere on their repo
  - dofm 3mo ago
    Of course — I compiled it and ran it yesterday. Not aware if they have a PR in place with the main project, though as I recall I ran the last Bonsai model without needing it so I guess there might be plans.

bansaltushar 3mo ago

Depending on which model you're running, you might need to use the custom forks. Details are here -> https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README.md#upstream-status-for-ternary https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README....

motbus3 3mo ago
I spent quite sometime trying to install their tools and nothing really worked. I used these repos you shared but the dependencies all fail on mac
- PrismML 3mo ago
  If you can share details on where it is failing, we'd love to help fix. You can also join Discord to communicate with us directly http://discord.gg/prismml http://discord.gg/prismml
  - motbus3 3mo ago
    I did not know you guys would be watching. For sure. Let me do that tomorrow when I turn it on again :) I am happy to see the message. Thanks!

SwellJoe 3mo ago

I got their previous model working in their custom fork of llama.cpp (https://github.com/PrismML-Eng/llama.cpp https://github.com/PrismML-Eng/llama.cpp). I haven't tried this one yet, but will find some time to benchmark it sometime this week. Though this says mainline llama.cpp has their patches for Metal and CPU backends, so maybe it's simply "use current llama.cpp" if you have a Mac or fast enough CPU/memor to use the CPU backend.

dofm 3mo ago

Not sure if there's any way to run Prism's fork of llama.cpp inside LM Studio. The fork runs fine for me. The model gets very notably stuck in a reasoning loop on one of my simple tests, though it might be that it has the same issues with setting reasoning effort high. On my M1 Max I still think the MoE Qwen 3.6 and Gemma 4 models are the best options. And I am far from convinced that the 35B is actually worse; it gets stuck in reasoning loops much less often than 27B in my experience.

try-working 3mo ago

I downloaded two of the official ones in LM Studio, both 3.6gb, and neither loaded.

anon373839 3mo ago

I can report that it's working in oMLX. I've been experimenting with the ternary one; it is quite an impressive model! I've been grilling it on some deep learning/computer vision stuff and it's aced everything so far. Responses are thorough, accurate, sophisticated. General knowledge outside of CS doesn't seem as robust, which I expected. Honestly, I don't think the examples in the blog post do it justice.

ch_sm 3mo ago
Nice, I tried it too with oMLX — agreed, it seems very capable for coding! Was slightly underwhelmed by performance though. I got about ~24 t/s on the ternary version on my M2 Max 64. That’s quite a bit slower than Qwen A3B 35B (4 bit unsloth). How was perf for you?
- anon373839 3mo ago
  I had roughly the same performance on M3 Max 64: ~30 tokens/sec. Which is not terrible, but with the latest Lightning MTP optimization I am getting ~100 tokens/sec with Qwen 3.6 35B-A3B.
  - ch_sm 3mo ago
    WHAT. that‘s amazing. thanks so much for sharing, i‘ve been looking for ways to speed a3b up for days. It‘s 11pm here but i‘ll try this right now
    - anon373839 3mo ago
      I’m using the new oQe quants in case that matters.
      - ch_sm 3mo ago
        i see! would you mind sharing your model string?