2 ms·
You're welcome! Should have specified the reason I stick with LLama 3.2 Vision 8B Q4_K_M is that the Q8 doesn't fully fit in memory and so, as mentioned, is sig
by magicalhippo 2y ago
You're welcome! Should have specified the reason I stick with LLama 3.2 Vision 8B Q4_K_M is that the Q8 doesn't fully fit in memory and so, as mentioned, is significantly slower.
But I do like to compare. Open WebUI for example makes it very easy where you can load up multiple models and it'll send the same prompt to each one in turn, and show the answers side by side.