3 ms·

I tried running those locally via Ollama on my 3080 (latency doesn't matter) and the performance is abysmal. They're almost random token generators, literally u

I tried running those locally via Ollama on my 3080 (latency doesn't matter) and the performance is abysmal. They're almost random token generators, literally unusable. My hallucination rate was near 100% for basic tasks like data extraction from markdown.

dist-epoch 1y ago
Have you tried Qwen-32b-8bit? It's pretty solid. Anything lower, yeah, it's kind of garbage.