I’m very skeptical of LLM benchmarks because I’ve been testing models through Ollama and none of the open source 7b models are at even close to GPT4 (or Open AI

I’m very skeptical of LLM benchmarks because I’ve been testing models through Ollama and none of the open source 7b models are at even close to GPT4 (or Open AI’s GPT 3.5 implementation) in daily usage, despite what benchmarks say. LLM results are actually really hard to measure objectively because it depends on what kinds of questions you ask and what answers you expect. I’ve just been finding lots of little errors and reasoning mistakes, and even in terms of producing useful results, they fall short of GPT4.