3 ms·
This article's credibility suffers a little from the way it talks about GPT-4o mini: "just in front of GPT-4o-mini, which is, according to itself, a model with
by simonw 2y ago
This article's credibility suffers a little from the way it talks about GPT-4o mini:
"just in front of GPT-4o-mini, which is, according to itself, a model with 1.3B or 1.5B or 1.7B parameters, depending on when you ask."
Then later:
"On the Artificial Analysis benchmark Scout achieved the same score as GPT 4o mini. A 109B model vs a 1.5B model (allegedly). This is ABYSMAL."
Asking models how many parameters they have doesn't make sense.
There is absolutely no way GPT-4o mini is 1.5B. I can run a 3B model on my iPhone, but it's a fraction of the utility of GPT-4o mini.
- fancyfredbot 2y agoIt's strange because there's no need to make this assumption about GPT-4o in order to demonstrate their point.
- gliptic 2y agoIt's strange that someone from FastML can be confused about this, unless it's supposed to be a bad joke.
- PunchTornado 2y agowe are in for a lot of pain if seemingly intelligent people make mistakes like this. grabbing the number of params from what gpt gives you. how can you do that?
- jychang 2y agoCorrect, in that models know nothing about themselves other than what they are told. Deepseek R1 will tell you that it's created by OpenAI. GPT-4o mini is supposed to be ~8b params from estimates.
- simonw 2y agoThe best source I've seen for the 8B number is this TechCrunch article: https://techcrunch.com/2024/07/18/openai-unveils-gpt-4o-mini-a-small-ai-model-powering-chatgpt/ https://techcrunch.com/2024/07/18/openai-unveils-gpt-4o-mini... > OpenAI would not disclose exactly how large GPT-4o mini is, but said it’s roughly in the same tier as other small AI models, such as Llama 3 8b, Claude Haiku and Gemini 1.5 Flash. As far as I can tell all of the 8B rumors were seeded by that loosely sourced comparison to Llama 3 8B. I know for a fact that Gemini 1.5 Flash is NOT an 8B model, because a separate model called "Gemini 1.5 Flash 8B" came out after that article was published - the first Gemini model with a documented parameter count. Flash 8B is priced at half the cost of regular Flash. There's also this paper that mentions 8B but provides no source for that at all, which makes me suspect their initial source may have been that TechCrunch rumor: https://arxiv.org/pdf/2412.19260 https://arxiv.org/pdf/2412.19260
- jychang 2y ago8b params is a reasonable estimate. As a sanity check, we can look at scores for how it performs. On livebench, GPT-4o-mini scores 37.63, right next to Gemini 1.5 Flash 8B at 34.37, and above Qwen2.5 7B Instruct Turbo/Gemma 3 4B at 29.22/30.13. And it's below Phi-4 14b at 40.68, and Gemma 3 12B at 41.25.
- gwd 2y agoThe comment at the top says it's a draft. It's not unreasonable to ask for random values from a GPT for "filler" for the draft (or even just make them up), just to stay in the flow, and then track down the real numbers later.
- NitpickLawyer 2y agoI flagged it for these reasons as well. It's just a bad article. Shows very poor understanding of the basics of LLM workings, and the field in general. Lingers on the "cheated" benchmark (lmsys) but never mentions all the other 3rd party benchmarks performed after the inference fixes, which are in line with what Meta originally published. To be clear, submitting a different fine-tuned model to one arena and releasing the untuned model without clearly mentioning this, is bad. But conflating the "human prefference" bench with the others and not mentioning the models capabilities on other benchmarks is also bad writing. The MoE paragraphs are bad, and the writer never explains why the copy 17B vs VRAM size is bad, they just leave it there unexplained. Poor form, I was expecting better from someone working in this field.