4 ms·
Aged like milk https://news.ycombinator.com/item?id=49746618 https://news.ycombinator.com/item?id=49746618
by sheo 15d ago
Aged like milk
https://news.ycombinator.com/item?id=49746618 https://news.ycombinator.com/item?id=49746618
- Mashimo 15d agoIn the comments it reads like bonsei falls apart on longer running tasks.
- txrx0000 15d agoNot really. The largest IQ4_XS quant here is still worth it because Bonsai doesn't offer larger quants. They could beat it if they made a quaternary variant though, I don't know why they're stopping at ternary.
- rguiscard 15d agoI wonder the same thing for Bonsai 2. ByteShape offers 5 models from IQ2_XXS-2.56bpw (8.8GB), IQ3_XXS-2.88bpw (9.9GB), IQ3_XS-3.01bpw (10.4GB), IQ3_S-3.23bpw (11.0GB) to IQ4_XS-3.84bpw (13.1GB). Their benchmarks show gradual improvement with size and users can pick one to fit theirs need. Bonsai-2-27B now is about 8.6GB. It might be good to have a quaternary version around 10-11GB to fit a computer with 16-24GB RAM.
- deleted 15d ago[deleted]
- dingdingdang 15d agoAgree here, as it is ByteShape wins practicality wise if the goal is doing actual work with these quants!
- electroglyph 15d agoprismml's title is very misleading. in their own paper the model is at 75% of coding scores.