6 ms·
Llama 3 70B tied with GPT-4 for first place on LMSYS chatbot arena leaderboard
- tmaly 3y agocomments on the X post point out that 8B models is pretty good. What type of VRAM would you need to run 8B locally?
- mertbio 3y agoI can run the 8B with my M2 Pro Mac mini with 32GB via Ollama. It takes around one second to get a response from the API.
- zamalek 3y agoI have enough VRAM on my 6900XT (16GB) for 13B quantized models alongside desktop/hyprland usage.
- CobaltFire 3y agoMy understanding is that as a general rule: ( Q / 8 ) * B = GB RAM, where Q is the Quant level and B is the model size. So a Q4 7B model is ( 4 / 8 ) * 7 = 3.5GB RAM (or VRAM). A non-Q model is 16, so 2 * B. I believe this is before context, which adds a bit.
- SushiHippie 3y agoDefinitely does not seem too far off: https://ollama.com/library/llama3/tags https://ollama.com/library/llama3/tags Llama3 8B fp16 is 16GB, q8 is 8.5GB and q4 is between 4.3 and 4.9GB. Llama3 70B fp16 is 141GB, q8 75GB and q4 is between 40 and 43GB.
- tmaly 3y agobased on your formulas, I am definitely going to need a larger card.
- MyFirstSass 3y agoI run 8b on my Macbook air m2 with 16gb's of vram at decent speed of 12 t/s (mlx can probably get 15 t/s). I use the Q5_K_M GGUF, which is >99% the same as the original. I've seen tests that there is close to no divergence with these quantisations, but it rises steeply going lower: https://www.reddit.com/r/LocalLLaMA/comments/1816h1x/how_much_does_quantization_actually_impact_models/ https://www.reddit.com/r/LocalLLaMA/comments/1816h1x/how_muc...
- tosh 3y ago~8GB RAM for a quantized version of 8B
- modeless 3y agoNote that this is the English-only leaderboard, which you have to manually select when you visit the site. Also the uncertainty on the score is still high, so it may change. But this is undoubtedly the strongest open weights model of any size so far, by far. People seem to be saying that Llama 3's safety tuning is much less severe than before so my speculation is that this is due to reduced refusal of prompts rather than superior knowledge or reasoning, given the eval scores. But still, a real and useful improvement! At this rate, the 400B is practically guaranteed to dominate.
- zone411 3y agoOnce again, I must ask everyone not to place too much emphasis on this benchmark. Another post I see right now on the HN homepage is this: https://www.tbray.org/ongoing/When/202x/2024/04/18/Meta-AI-oh-my https://www.tbray.org/ongoing/When/202x/2024/04/18/Meta-AI-o.... As it points out, Llama 3 gave a plausible, smart-sounding answer and people would rate it highly on the LMSYS leaderboard, yet it might be totally incorrect. It's best to think of the LMSYS ranking as something akin to the Turing Test, with all its flaws. That said, all other benchmarks so far (including my NYT Connections benchmark) show that both Llama 3 models are exceptionally strong for their sizes.
- futureshock 3y agoPeople love this benchmark. And it has a solid idea behind it, but it also seems like it would be trivial to game since it relies on public “blind” rankings. All the models need for an advantage is just to sound more personal and confident and that’ll heavily bias most users asking generic questions or having a chat. It’ll be a terrible measure for any kind of intelligence aside from sounding human-ish. It’s also good to keep in mind that the ELO scores compute the probability of winning, so a 5 point Elo difference essentially still means it’s a 49-51 chance of user preference. Not much better than a coin flip between the first 4 models, especially considering the confidence interval.
- tibbar 3y agoI thought that surely the ELO ratings were on a different scale than the usual chess ones, but nope, it uses the same factor of 400: https://colab.research.google.com/drive/1RAWb22-PFNI-X1gPVzc927SGUdfr6nsR?usp=sharing#scrollTo=hytEb0aXfcwm https://colab.research.google.com/drive/1RAWb22-PFNI-X1gPVzc.... That means we truly do have a long way to go. However, it's also possible that user questions don't have enough 'difficulty' to distinguish -- eg, if the proportion of relatively easy user questions is high, then there are fewer opportunities for a better model to distinguish itself.
- tosh 3y agothis.
- futureshock 3y agoTotally. And it’s interesting to consider what Elo a human would have, if there was some way to account for the extra latency without unblinding the test. I suspect a human control would not be a lot higher ranked. For the most part top LLMs already sound human if you prompt or fine tune them to emulate conversational style. A determined Turing tester would tell the difference most of the time I believe, but most people submitting arena ratings are not determined.
- modeless 3y ago
- londons_explore 3y agoThe lmsys leaderboard has 700k votes. Who did all those votes? I did like 5 of them... I'm slightly suspicious someone might be gaming the votes, because billions of dollars of company valuations ride on who has the best AI...
- zooq_ai 3y agoIt's mostly gamed by open source zealots, who have plenty of time in their hands to send "OSS, BEST" on the internet
- refulgentis 3y agoThis is incorrect. This is the first time there's been a non-private model at the top, and I'd wager top 3. This isn't OSS either. Not sure where the jaundiced view comes from. I generally believe the free models tend to be overrated, but I don't think it's a good idea to pretend there's widespread gaming of this. There's a strong self-peasantization streak in LLM discussions, and truth is, no one thing may be "correct", but we can't exclude all data without making the very act of discussion meaningless.
- zooq_ai 3y agoYou have to be seriously delusional to think that a 70B model will beat Claude Opus. But go ahead and use it to prove my point. I know the power of Zealotry and Cult in the age of internet and this is just one example of that
- refulgentis 3y agoI'm at the front of the line, I maintain an app with integrations with all 4 major providers and llama.cpp with 4 built-in models. Yesterday, I was one of the, if not the, first to get it integrated and flag to model makers they had the wrong chat template embedded. That's a lengthy way of saying, contra what you imply, I have practical experience & understand this stuff intimately. I agree with your opinion re: it's unlikely LLaMA 3 70B beats all private models, but think your way of relaying it, claiming there's widespread gaming of it by advocates for open models, is obviously incorrect, and adds more confusion rather than reducing it.
- davej 3y agoLlama 3 is tuned very nicely for English answers. What is most surprising to me is that the 8B model is performing similarly to Mistral's large model and the original GPT4 model (in English answers). Easily the most efficient model currently available.
- swalsh 3y agoParameter count seems to only matter for range of skills, but these smaller models can be tuned to be more than competitive with far larger models. I suspect the future is going to be owned by lots of smaller more specific models, possibly trained by much larger models. These smaller models have the advantage of faster and cheaper inference.
- theLiminator 3y agoProbably why MoE models are so competitive now. Basically that idea within a single model.
- elevaet 3y agoPlease, what is an MoE model?
- CuriouslyC 3y agoI don't think MoE is the way forward. The bottleneck is memory, and MoE trades MORE memory consumption for lower inference times at a given performance level. Before too long we're going to see architectures where a model decomposes a prompt into a DAG of LLM calls based on expertise, fans out sub-prompts then reconstitutes the answer from the embeddings they return.
- hackerlight 3y agoConfidence intervals too wide to draw that conclusion.
- oersted 3y agoNot really, right now it's within +14/-15 ELO with 95% probability, which barely changes the ranking. It's well ahead of Gemini Pro and squarely within the top 5 with Claude 3 Opus and a few versions of GPT-4-Turbo. More impressively even, Llama 3 8B is approximately tied with GPT-4 (depending on the version), as well as Mistral-Large and Mixtral 8x22B, which is mad for the size.
- refulgentis 3y agoI'm very surprised to see this considering the models aren't chat-tuned and shouldn't be used for chat, at least, by the disclaimer Meta provided. I wish they didn't include the disclaimer just for the sake of having one, they provided a chat format, so they must have trained _some_ chat, and we can see here its good enough at chat to not act like a pure LLM, or even close. Labelling it with a disclaimer it doesn't need causes general confusion in scenarios like Phi-2, which had no chat tuning but great scores, so people couldn't understand "why it didn't work".
- arnaudsm 3y agoTitle ignores the low statistic significance. It lost 23 points since the screenshot was taken.
- modeless 3y agoNo, you're just looking at the multi language results instead of the English results. Llama 3 isn't as good in other languages.
- deleted 3y ago[deleted]
- random_i 3y agoI asked the console "How can I visualize a decision tree trained using PySpark?" and it very confidently gave me 4 detailed, step-by-step answers (including code!) that are utterly false and useless. :)
- oersted 3y agoThe ranking has changed significantly after 6 hours: Llama 3 70B Instruct (ELO 1198) is now behind even Claude 3 Sonnet (ELO 1202) and well behind the latest GPT-4-Turbo (ELO 1259), but still above the last GPT-4 (ELO 1189). EDIT: My bad, I was looking at the Overall leaderboard instead of the English one as other commenters. Still quite impressive.
- vitorgrs 3y agoOnly in English. At least in Brazilian Portuguese, Llama 3 is awful. You literally ask in Portuguese and it answers in English, etc. It's a very much English First model, which is totally different from even GPT 3.5 that works amazing in Brazilian Portuguese.