2 ms·
I've just tested the 1B and 3B at Q8, some interesting bits: - The 1B is extremely coherent (feels something like maybe Mistral 7B at 4 bits), and with flash a
by moffkalast 2y ago
I've just tested the 1B and 3B at Q8, some interesting bits:
- The 1B is extremely coherent (feels something like maybe Mistral 7B at 4 bits), and with flash attention and 4 bit KV cache it only uses about 4.2 GB of VRAM for 128k context
- A Pi 5 runs the 1B at 8.4 tok/s, haven't tested the 3B yet but it might need a lower quant to fit it and with 9T training tokens it'll probably degrade pretty badly
- The 3B is a certified Gemma-2-2B killer
Given that llama.cpp doesn't support any multimodality (they removed the old implementation), it might be a while before the 11B and 90B become runnable. Doesn't seem like they outperform Qwen-2-VL at vision benchmarks though.
- Patrick_Devine 2y agoHoping to get this out soon w/ Ollama. Just working out a couple of last kinks. The 11b model is legit good though, particularly for tasks like OCR. It can actually read my cursive handwriting.
- jsarv 2y agoNaah, Qwen2-VL-7b still is much much better than 11b model for handwritten OCR from what i have tested. The 11b model hallucinates in case of handwritten OCR.
- sumedh 2y agoWhere can I try it out. The playground on their homepage is very slow. I am willing to pay for it as well if the OCR is good.
- Ey7NFZ3P0nzAe 2y agoOpenrouter.ai