5 ms·
Open Source 7B models have already caught up with OpenAI. The current top of the open model 7B leaderboard beats GPT-3.5 and Bard while running on a laptop, sm
by MacsHeadroom 3y ago
Open Source 7B models have already caught up with OpenAI.
The current top of the open model 7B leaderboard beats GPT-3.5 and Bard while running on a laptop, smartphone, raspberry pi, or 10 year old graphics card.
Tim Dettmers just released code to get Mixtral 8x7b running in 4GB of RAM, the same amount required by Mistral 7b.
We are quite clearly a matter of weeks from Open Source being on par with GPT-4, likely before Google's Gemini Ultra is even released.
This is all to say nothing of the multi-model LLama-3 120B coming in 2 months according to insider leaks.
- wenc 3y agoI’m very skeptical of LLM benchmarks because I’ve been testing models through Ollama and none of the open source 7b models are at even close to GPT4 (or Open AI’s GPT 3.5 implementation) in daily usage, despite what benchmarks say. LLM results are actually really hard to measure objectively because it depends on what kinds of questions you ask and what answers you expect. I’ve just been finding lots of little errors and reasoning mistakes, and even in terms of producing useful results, they fall short of GPT4.