3 ms·
This is false. I have tested Q8s over f16 and the raw weights and pretty much seen no difference. I strictly run Q8s, I only keep the raw weights if I plan t
by segmondy 2y ago
This is false. I have tested Q8s over f16 and the raw weights and pretty much seen no difference. I strictly run Q8s, I only keep the raw weights if I plan to do some fine tunes.
- staticman2 2y agoThis matches the general consensus at the LocalLLama subreddit that Q8 is basically indistinguishable from the full model.