3 ms·
Just use llama.ccp and run the quantized model like yi-34b-chat.Q4_K_M.gguf (you will need a 32GB Macbook, likely).
by antirez 3y ago
Just use llama.ccp and run the quantized model like yi-34b-chat.Q4_K_M.gguf (you will need a 32GB Macbook, likely).
- numpad0 3y agoNote also that 2x32GB DDR4 UDIMM kits are ~$120. It's still an overkill for most purposes, but a cheap one.
- Der_Einzige 3y agoHighly quantized models are crippled versions of the original model. Don't evaluate a model for a production use case based on it's quantized versions.
- deleted 3y ago[deleted]
- antirez 3y agoThis depends on the quantization level. Sure, they lose part of the performance, but are still quite useful and cheap to run.
- eigenvalue 3y agoI find the larger Q5 versions from theBloke to be pretty good and allow you to run a larger model than you’d be able to otherwise. But there is definitely an impact, although it’s hard to quantify. It’s sort of like how you get less effective at coding when you’ve only slept 5 hours.
- nacs 3y ago4_k_m and 5_k_m are very good quant levels that don't sacrifice much.