2 ms·

Probably won't have to wait that long. Prism released Bonsai 27B (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit https://huggingface.co/prism-ml/Te

by smeeth 2mo ago

Probably won't have to wait that long. Prism released Bonsai 27B (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one. It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.

dwa3592 2mo ago
agreed!! in my heart i really wanted to say by the end of 2026 but wanted to add some wiggle room in case they start to ban open source AI development.
- dTal 2mo ago
  >in case they start to ban open source AI development Is that likely, do you think?
  - dwa3592 2mo ago
    dario has been saying open source models are dangerous. who knows who is listening to him.
    - mdp2021 2mo ago
      The one time in which I saw Juergen Schmidhuber in perfect nervous control, "coolness" they may say westward, was when he replied to one member of the audience, "The same observation was made when they invented fire: oh, it's dangerous. But in the end, now it's here (shrugh)". There is a proposal in the USA to restrict LLM access. This will only have us depend more and more on open source models and their providers. And cause a drain of research in those areas in which it will be impeded.
  - submain 2mo ago
    NVidia seems to want them. They have a lot of lobbying power.
acchow 2mo ago
We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?
- nkozyra 2mo ago
  No. But do you need to run every small problem through a 10B-30B model? We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code. Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for the end user.
  - edg5000 2mo ago
    I do about 10 google search queries for every 1 opus/gpt prompt. For google, I don't actually open pages anymore 9 out 10 times; I rely on the AI summary. It's fast and accurate; the trick is that you learn where the boundary is of what you can ask it. Querying information the small model is great at. Then there might be slow, batch tasks. I can see myself getting 1T of slow RAM one day (in a few years?) and having a slow onsite GLM5.2 doing batch jobs that would be wasteful of my subscription limits, plus sensitive but boring things, such as bookeeping and general admin. I'd like to to read all my email and al quarterly reporting. But that would have to be a good local model, probably a model simmilar to whatever google search uses, which seems just correct unless you throw serious challenges at it.
  - mdp2021 2mo ago
    > do you need to run every small problem through a 10B-30B model? ... We're asking frontier Opus/Fable models to classify text Actually probably yes: text analysis (magazine articles) by LLMs in the ~30b .. ~120b range failed miserably (and also randomly - the rare cases of proper interpretation occurred among the failure cases) with the main public models of around one year ago, tried extensively. So, yes, you can employ an ~80IQ only if you will expect the related quality.
  - ubercore 2mo ago
flockonus 2mo ago
There is also the 1bit version @ 3.9 GB that retains 90% of the intelligence - quite a feat! https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit

by smeeth 2mo ago

dwa3592 2mo ago

agreed!! in my heart i really wanted to say by the end of 2026 but wanted to add some wiggle room in case they start to ban open source AI development.

dTal 2mo ago
>in case they start to ban open source AI development Is that likely, do you think?
- dwa3592 2mo ago
  dario has been saying open source models are dangerous. who knows who is listening to him.
  - mdp2021 2mo ago
    The one time in which I saw Juergen Schmidhuber in perfect nervous control, "coolness" they may say westward, was when he replied to one member of the audience, "The same observation was made when they invented fire: oh, it's dangerous. But in the end, now it's here (shrugh)". There is a proposal in the USA to restrict LLM access. This will only have us depend more and more on open source models and their providers. And cause a drain of research in those areas in which it will be impeded.
- submain 2mo ago
  NVidia seems to want them. They have a lot of lobbying power.

acchow 2mo ago

We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?

nkozyra 2mo ago
No. But do you need to run every small problem through a 10B-30B model? We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code. Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for the end user.
- edg5000 2mo ago
  I do about 10 google search queries for every 1 opus/gpt prompt. For google, I don't actually open pages anymore 9 out 10 times; I rely on the AI summary. It's fast and accurate; the trick is that you learn where the boundary is of what you can ask it. Querying information the small model is great at. Then there might be slow, batch tasks. I can see myself getting 1T of slow RAM one day (in a few years?) and having a slow onsite GLM5.2 doing batch jobs that would be wasteful of my subscription limits, plus sensitive but boring things, such as bookeeping and general admin. I'd like to to read all my email and al quarterly reporting. But that would have to be a good local model, probably a model simmilar to whatever google search uses, which seems just correct unless you throw serious challenges at it.
- mdp2021 2mo ago
  > do you need to run every small problem through a 10B-30B model? ... We're asking frontier Opus/Fable models to classify text Actually probably yes: text analysis (magazine articles) by LLMs in the ~30b .. ~120b range failed miserably (and also randomly - the rare cases of proper interpretation occurred among the failure cases) with the main public models of around one year ago, tried extensively. So, yes, you can employ an ~80IQ only if you will expect the related quality.
- ubercore 2mo ago

flockonus 2mo ago

There is also the 1bit version @ 3.9 GB that retains 90% of the intelligence - quite a feat! https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit