3 ms·
This is basically a GPT-4 level model that runs (quantized) on a 32gb ram laptop. Yes it doesn't recall facts from training material as well but with tool use
by tosh 1y ago
This is basically a GPT-4 level model that runs (quantized) on a 32gb ram laptop.
Yes it doesn't recall facts from training material as well but with tool use (e.g. wikipedia lookup) that's not a problem and even preferable to a larger model.
- anyg 1y ago>basically a GPT-4 level model Can you share more insights on this? Going by @simonw's testing, the quantized model doesn't seem close to GPT-4 level.
- simonw 1y agoI think calling it "GPT-4 level" is justified if we are talking about original GPT-4 from March 2023.
- andygeorge 1y agoin my limited testing, qwen3:30b-a3b-instruct-2507-q4_K_M is fast but far less accurate/helpful than gemma3:27b-it-q4_K_M