4 ms·

Any details on the latest approach to distillation would also be very interesting.

Schlagbohrer 9d ago
I am surprised at the lack of open-weights models in the >35B, but <200B range. I keep thinking about devices like the NVIDIA Spark and AMD Ryzen Halo, which have their 128GB of combined memory, but there are so few models made for that range. Nearly all the open weights distillations are for larger customer bases with <24GB VRAM.
- MaKey 9d ago
  The market is too small.
  - ElectricalUnion 9d ago
    The only (still in prototype stage!) "competitor" for those GB10/Ryzen Al Max+ 395 (in my region, borderline unobtainable) systems seems to be the Xiaomi AI Cube.
  - Schlagbohrer 8d ago
    Yes exactly, I am hoping that when the AMD Ryzen Halo gets wider release and more consumers have these 128GB devices, the market will justify a wider range of quantized model sizes. And then when I win the lottery and can buy one I'll have lots of nice options!
- bitexploder 9d ago
  Qwen Flash Next 3.8 … even at 3 bit quant it is very solid.
- Catloafdev 9d ago
  Qwen3.8 Flash Next just released which hits that range. Also, Deepseek V4 Flash can be run relatively well in hybrid 2-bit quantization on 128gb devices, with way better results than you'd expect for a typical 2-bit quant. Those are currently the 'smartest' options for that memory level.
alightsoul 9d ago
American exceptionalism states that America is special and unique so everyone else must be a copycat. American ai labs don't need this kind of optimization and fable will outright refuse to do it.
- ElectricalUnion 9d ago
  Isn't Fable intentionally trained and system prompted to act maliciously and attempt to sabotage third party attempts to use it to train or improve other LLMs?
  - alightsoul 9d ago
    Right i forgot. That's even worse, it's a form of data poisoning but it poisons humans and not ai.
  - CamperBob2 8d ago
    It was, as insane as it sounds. They walked that back AFAIK. Fable will still dump you back into Opus at the drop of a hat, but at least it will tell you when it happens.

I am surprised at the lack of open-weights models in the >35B, but <200B range. I keep thinking about devices like the NVIDIA Spark and AMD Ryzen Halo, which have their 128GB of combined memory, but there are so few models made for that range. Nearly all the open weights distillations are for larger customer bases with <24GB VRAM.

MaKey 9d ago
The market is too small.
- ElectricalUnion 9d ago
  The only (still in prototype stage!) "competitor" for those GB10/Ryzen Al Max+ 395 (in my region, borderline unobtainable) systems seems to be the Xiaomi AI Cube.
- Schlagbohrer 8d ago
  Yes exactly, I am hoping that when the AMD Ryzen Halo gets wider release and more consumers have these 128GB devices, the market will justify a wider range of quantized model sizes. And then when I win the lottery and can buy one I'll have lots of nice options!
bitexploder 9d ago
Qwen Flash Next 3.8 … even at 3 bit quant it is very solid.
Catloafdev 9d ago
Qwen3.8 Flash Next just released which hits that range. Also, Deepseek V4 Flash can be run relatively well in hybrid 2-bit quantization on 128gb devices, with way better results than you'd expect for a typical 2-bit quant. Those are currently the 'smartest' options for that memory level.

alightsoul 9d ago

American exceptionalism states that America is special and unique so everyone else must be a copycat. American ai labs don't need this kind of optimization and fable will outright refuse to do it.

ElectricalUnion 9d ago
Isn't Fable intentionally trained and system prompted to act maliciously and attempt to sabotage third party attempts to use it to train or improve other LLMs?
- alightsoul 9d ago
  Right i forgot. That's even worse, it's a form of data poisoning but it poisons humans and not ai.
- CamperBob2 8d ago
  It was, as insane as it sounds. They walked that back AFAIK. Fable will still dump you back into Opus at the drop of a hat, but at least it will tell you when it happens.