5 ms·
I was curious how the numbers compare to GPT-4 in the paid ChatGPT Plus, since they don't compare directly themselves. Llama 3 8B Llama 3 70B GPT-4
by modeless 3y ago
I was curious how the numbers compare to GPT-4 in the paid ChatGPT Plus, since they don't compare directly themselves.
Llama 3 8B Llama 3 70B GPT-4
MMLU 68.4 82.0 86.5
GPQA 34.2 39.5 49.1
MATH 30.0 50.4 72.2
HumanEval 62.2 81.7 87.6
DROP 58.4 79.7 85.4
Note that the free version of ChatGPT that most people use is based on GPT-3.5 which is much worse than GPT-4. I haven't found comprehensive eval numbers for the latest GPT-3.5, however I believe Llama 3 70B handily beats it and even the 8B is close. It's very exciting to have models this good that you can run locally and modify!
GPT-4 numbers from from https://github.com/openai/simple-evals https://github.com/openai/simple-evals gpt-4-turbo-2024-04-09 (chatgpt)
- gliched_robot 3y agoWild considering, GPT-4 is 1.8T.
- deleted 3y ago[deleted]
- oersted 3y agoWhere did you find this number? Not doubting it, just want to get a better idea of how precise the estimate may be.
- cjbprime 3y agoIt's a very plausible rumor, but it is misleading in this context, because the rumor also states that it's a mixture of experts model with 8 experts, suggesting that most (perhaps as many as 7/8) of those weights are unused by any particular inference pass. That might suggest that GPT-4 should be thought of as something like a 250B model. But there's also some selection for the remaining 1/8 of weights that are used by the chosen expert as being the "most useful" weights for that pass (as chosen/defined by the mixture routing), so now it feels like 250B is undercounting the parameter size, whereas 1.8T was overcounting it. I think it's not really defined how to compare parameter counts with a MoE model.
- wongarsu 3y agoBut from an output quality standpoint the total parameter count still seems more relevant. For example 8x7B Mixtral only executes 13B parameters per token, but it behaves comparable to 34B and 70B models, which tracks with its total size of ~45B parameters. You get some of the training and inference advantages of a 13B model, with the strength of a 45B model. Similarly, if GPT-4 is really 1.8T you would expect it to produce output of similar quality to a comparable 1.8T model without MoE architecture.
- staticman2 3y ago"For example 8x7B Mixtral only executes 13B parameters per token, but it behaves comparable to 34B and 70B models" Are you sure about that? I'm pretty sure Miqu (the leaked Mistral 70b model) is generally thought to be smarter than Mixtral 8x7b.
- chessgecko 3y agoI think its almost certainly using at least two experts per token. It helps a lot during training to have two experts to contrast when putting losses on the expert router.
- worldsayshi 3y agoWhat is the reason for settling on 7/8 experts for mixture of experts? Has there been any serious evaluation of what would be a good MoE split?
- nycdatasci 3y agoIt's not always 7-8. From Databricks: "DBRX has 16 experts and chooses 4, while Mixtral and Grok-1 have 8 experts and choose 2. This provides 65x more possible combinations of experts and we found that this improves model quality. DBRX uses rotary position encodings (RoPE), gated linear units (GLU), and grouped query attention (GQA). It uses the GPT-4 tokenizer as provided in the tiktoken repository. We made these choices based on exhaustive evaluation and scaling experiments." https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm https://www.databricks.com/blog/introducing-dbrx-new-state-a...
- huijzer 3y agoProbably from Nvidia's GTC keynote: https://www.youtube.com/live/USlE2huSI_w?t=2995 https://www.youtube.com/live/USlE2huSI_w?t=2995. In the keynote, Jensen uses 1.8T in an example and suggests that this is roughly the size of GPT-4 (if I remember correctly).
- sputknick 3y agoI'm not OP, but George Hotz said in his lex friedman podcast a while back that it was an MoE of 8 250B. subtract out duplication of attention nodes, and you get something right around 1.8T
- qeternity 3y agoI'm pretty sure he suggested it was a 16 way 110 MoE
- brandall10 3y agoThe exact quote: "Sam Altman won’t tell you that GPT 4 has 220 billion parameters and is a 16 way mixture model with eight sets of weights."
- refulgentis 3y agoIt's a really funny story that I comment about at least once a week because it drives me nuts. 1. After ChatGPT release, Twitter spam from influencers about chatGPT is one billion and GPT-4 is 1 trillion. 2. Semianalysis publishes a blog post claiming 1.8T sourced from insiders. 3. The way info diffusion works these days, everyone heard from someone else other than Semianalysis. 4. Up until about a month ago, you could confidently say "hey its just that one blog post" and work through it with people to trace their initial hearing of it back to the post. 5. nVidia press conference some time in the last month used the rumors as an example with "apparently" attached, and now people will tell you NVidia confirmed 1.8 trillion. my $0.02: I'd bet my life GPT-4 isn't 1.8T, and I very much doubt its over 1 trillion. Like, lightning striking the same person 3 times in the same week.
- cjbprime 3y agoYou're ignoring geohot, who is a credible source (is an active researcher himself, is very well-connected) and gave more details (MoE with 8 experts, when no-one else was doing production MoE yet) than the Twitter spam.
- anoncareer0212 3y agoGeohot? I know enough people at OpenAI to know 4 people's reaction at the time he started claiming 1T based on timing latency in the ChatGPT webui per token. In general, not someone you wanna be citing with lengthy platitudes, he's an influencer who speaks engineer, he's burned out of every community he's been in, acrimonously.
- andy99 3y agoOnce benchmarks exist for a while, they become meaningless - even if it's not specifically training on the test set, actions (what used to be called "graduate student descent") end up optimizing new models towards overfitting on benchmark tasks.
- acchow 3y agoAlso, the technological leader focuses less on the benchmarks
- bilbo0s 3y ago"graduate student descent" Ahhh that takes me back!
- karmasimida 3y agoEven random seed could cause bad big shift in human eval performance if you know you know. It is perfectly illegal to choose one ckpt that looks best on those benchmarks and move along HumanEval is meaningless regardless, those 164 problems have been overfit to the tea. Hook this up to LLM arena we will get a better picture regarding how powerful they really are
- anvuong 3y agoI actually can't wrap my head around this number, even though I have been working on and off with deep learning for a few years. The biggest models we've ever deployed on production still have less than 1B parameters, and the latency is already pretty hard to manage during rush hours. I have no idea how they deploy (multiple?) 1.8T models that serve tens of millions of users a day.
- Simon321 3y agoIt's a mixture of experts model. Only a small part of those parameters are active at any given time. I believe it's 16x110B
- qeternity 3y agoThe original GPT4 may have been around that size (16x 110B). But it's pretty clear GPT4 Turbo is a smaller and heavily quantized model.
- IceHegel 3y agoYeah, it’s not even close to doing inference on 1.8T weights for turbo queries.
- sp332 3y agoThe bottom of https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/ has in-progress results for the 400B model as well. Looks like it's not quite there yet. Llama 3 400B Base / Instruct MMLU 84.8 86.1 GPQA - 48.0 MATH - 57.8 HumanEval - 84.1 DROP 83.5 -
- YetAnotherNick 3y agoFor the still training 400B: Llama 3 GPT 4(Published) BBH 85.3 83.1 MMLU 86.1 86.4 DROP 83.5 80.9 GSM8K 94.1 92.0 MATH 57.8 52.9 HumEv 84.1 74.4 Although it should be noted that the API numbers were generally better than published numbers for GPT4. [1]: https://deepmind.google/technologies/gemini/ https://deepmind.google/technologies/gemini/
- oliwary 3y agoWild! So if this indeed holds up, it looks like OpenAI were about a year ahead when GPT-4 was released, compared to the open source world. However, given the timespan between matching GPT-3.5 (Mixtral perhaps?) and matching GPT-4 has just been a few weeks, I am wondering if the open source models have more momentum. That said, I am very curious what OpenAI has in their labs... Are they actually barely ahead? Or do they have something much better that is not yet public? Perhaps they were waiting for Llama 3 to show it? Exciting times ahead either way!
- ChildOfChaos 3y agoYou've also got to consider that we don't really know where OpenAI are though, what they have released in the past year have been tweaks to GPT4, while I am sure the real work is going into GPT5 or whatever it gets called. While all the others are catching up and in some cases being slightly better, I wouldn't be surprised to see a rather large leap back into the lead from OpenAI pretty soon and then a scrabble for some time for others to get close again. We will really see who has the momentum soon, when we see OpenAI's next full release.
- cjbprime 3y agoHas anyone prepared a comparison to Mixtral 8x22B? (Life sure moves fast.)
- Davidzheng 3y agoit's in the official post the comparison with Mixtral 8x22B
- cjbprime 3y agoWhere? I only see comparisons to Mistral 7B and Mistral Medium, which are totally different models.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- gs17 3y agohttps://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/ has it about a third of the way down. It's a little bit better on every benchmark than Mixtral 8x22B (according to Meta).
- cjbprime 3y agoOh cool! But at the cost of twice the VRAM and only having 1/8th of the context, I suppose?
- modeless 3y agoLlama 3 70B takes half the VRAM as Mixtral 8x22B. But it does need almost twice the FLOPS/bandwidth. Yes, Llama's context is smaller although that should be fixable in the near future. Another thing is that Llama is English-focused while Mixtral is more multilingual.
- eru 3y agoVia Microsoft Copilot (and perhaps Bing?) you can get access to GPT-4 for free.
- geepytee 3y agoI was particularly excited for the high HumanEval score, and this is before the 400B model and the CodeLlama tune! I just added Llama 3 70B to our coding copilot https://www.double.bot https://www.double.bot if anyone wants to try it for coding within their IDE
- 3abiton 3y agoBut I'm waiting for the finetunedz/merged models. Many devs produced great models based on Llama 2, that outperformed the vanilla one, so I expect similar treatment for the new version. Exciting nonetheless!
- deleted 3y ago[deleted]