6 ms·

Meta got caught gaming AI benchmarks

347 points by pseudolus 2y ago

dvcky_db 2y ago
This should surprise no one..Also Goodhart's law strikes again
arresin 2y ago
I believe this was designed to flatter the prompter more / be more ingratiating. Which is a worry if true (what it says about the people doing the comparing).
- add-sub-mul-div 2y ago
  There's no end to the possible vectors of human manipulation with this "open-weight" black box.
deckar01 2y ago
The top of that leaderboard is filled with closed weight experimental models.
etamponi 2y ago
Meta got caught _first_.
- sumeno 2y ago
  Not even first, OpenAI got caught a while back
  - Mond_ 2y ago
    Do you have a source for this? That's interesting (if true).
    - sumeno 2y ago
      They got the dataset from Epoch AI for one of the benchmarks and pinky swore that they wouldn't train on it https://techcrunch.com/2025/01/19/ai-benchmarking-organization-criticized-for-waiting-to-disclose-funding-from-openai/ https://techcrunch.com/2025/01/19/ai-benchmarking-organizati...
      - tananaev 2y ago
        I don't see anything in the article about being caught. Maybe I missed something?
        tedsanders 2y ago
        [flagged]
        suddenlybananas 2y ago
        >They got the dataset from Epoch AI for one of the benchmarks and pinky swore that they wouldn't train on it Is anything here actually false or do you not like the conclusions that people may draw from it?
        tedsanders 2y ago
        The false statement is “OpenAI got caught [gaming FrontierMath] a while back.” From a primary source: "OpenAI did not use FrontierMath data to guide the development of o1 or o3, at all.... we only downloaded FrontierMath for our evals long after the training data was frozen, and only looked at o3 FrontierMath results after the final announcement checkpoint was already picked." https://x.com/__nmca__/status/1882563755806281986 https://x.com/__nmca__/status/1882563755806281986
        suddenlybananas 2y ago
        Well, if we trust you. But you had the extra dataset and no one else did.
  - hooloovoo_zoo 2y ago
    People have been gaming ML benchmarks as long as there have been ML benchmarks. That's why it's better to see if other researchers are incorporating a technique into their actual models rather than 'is this paper the bold entry in a benchmark table'. But it takes longer.
    - jandrese 2y ago
      When a measure becomes a target it is no longer a good measure. These ML benchmarks were never going to last very long. There is far too much pressure to game them, even unintentionally.
- mkolodny 2y ago
  “Got caught” is a misleading way to present what happened. According to the article, Meta publicly stated, right below the benchmark comparison, that the version of Llama on LMArena was the experimental chat version: > According to Meta’s own materials, it deployed an “experimental chat version” of Maverick to LMArena that was specifically “optimized for conversationality” The AI benchmark in question, LMArena, compares Llama 4 experimental to closed models like ChatGPT 4o latest, and Llama performs better (https://lmarena.ai/?leaderboard https://lmarena.ai/?leaderboard).
JKCalhoun 2y ago
Is LMArena junk now? I thought there was an aspect where you run two models on the same user-supplied query. Surely this can't be gamed? > “optimized for conversationality” I don't understand what that means - how it gives it an LMArena advantage.
- light_hue_1 2y ago
  LMArena was always junk. I work in this space and while the media takes it seriously most scientists don't. Random people ask random stuff and then it measures how good they feel. This is only a worthwhile evaluation if you're Google or Meta or OpenAI and you need to make a chartbot that keeps people coming back. It doesn't measure anything else useful.
  - HPsquared 2y ago
    Conversation is a two-way street. A good conversation mechanic could elicit better interaction from the users and result in better answers. Stands to reason, anyway.
  - genewitch 2y ago
    I hear AI news from time to time from the M5M in the US - and the only place I've ever seen "LMArena" is on HN and in the LM studio discord. At a ratio of 5:1 at least.
    - brandall10 2y ago
      It's mentioned quite a bit in the LLM related subreddits.
  - cma 2y ago
    Llama 1 derived models on it were beating gpt 3.5 by have less refusals.
- gwern 2y ago
  It can be easily gamed. The users are self-selected, and they have zero incentive to be honest or rigorous or provide good responses. Some have incentives the opposite way. (There was a report of a prediction market user who said they had won a market on Gemini models by manipulating the votes; LMArena swore furiously there had definitely been no manipulation but was conspicuously silent on any details.) And the release of more LMArena responses has shown that a lot of the user ratings are blatantly wrong: either they're basically fraudulent, or LMArena's current users are people whose ratings you should be optimizing against because they are so ignorant, lazy, and superficial. At this point, when I look at the output from my Gemini-2.5-pro sessions, they are so high quality, and take so long to read, and check, and have an informed opinion on, I just can't trust the slapdash approach of LMArena in assuming that careless driveby maybe-didn't-even-read-the-responses-ain't-no-one-got-time-for-that-nerd-shit ratings mean much of anything. There have been red flags in the past and I've been taking them ever less seriously even as one of many benchmarks since early last year, but maybe this is the biggest backfire yet. And it's only going to get worse. At this rate, without major overhaul, you should take being #1 on LMArena seriously as useful and important news - as a reason to not use a model. It's past time for LMArena people to sit down and have some thorough reflection on whether it is still worth running at all, and at what point they are doing more harm than good. No benchmark lives forever and it is normal and healthy to shut them down at some point after having been saturated, but some manage to live a lot longer than they should have...
Mond_ 2y ago
The Llama 4 launch looks like a real debacle for Meta. The model doesn't look great. All the coverage I've seen has been negative. This is about what I expected, but it makes you wonder what they're going to do next. At this point it looks like they are falling behind the other open models, and made an ambitious bet on MoEs, without this paying off. Did Zuck push for the release? I'm sure they knew it wasn't ready yet.
- bko 2y ago
  I remember reading that they were in panic mode when the DeepSeek model came out so they must have scrambled and had to re-work a lot of things since DeepSeek was so competitive and open source as well
  - blueboo 2y ago
    Fear of R2 looms large as well. I suspect they succumbed to the nuance collapse along the lines of “Is double checking results worth it if DeepSeek eats our lunch?”
- nightski 2y ago
  Do you know that they made a bet on MoE? Meaning they abandonded dense models? I doubt that is the case. Just releasing MoE Llama 4 does not constitute a "bet" without further information. Also from what I can tell this performs better than models with parameter counts equal to one expert, and worse than fully dense models equal to total parameter count. Isn't that kind of what we'd expect? in what way is that a failure? Maybe I am missing some details. But it feels like you have an axe to grind.
  - genewitch 2y ago
    A 4x8 MOE performs better than an 8B but worse than a 32B, is your statement? My response would be, "so why bother with MOE?" However deepseek r1 is MOE from my understanding, but the "E" are all =>32B parameters. There's > 20 experts. I could be misinformed; however, even so, I'd say a MOE with 32B or even 70B experts will outperform (define this!) Models with equal parameter counts, because deepseek outperforms (define?) ChatGPT et al.
seydor 2y ago
tech companies competing over something that is losing them money is the most bizarre spectacle yet.
- abc-1 2y ago
  Is it really losing them money if investors throw fistfuls of cash at them for it
  - deleted 2y ago
    [deleted]
- FL33TW00D 2y ago
  Plot big tech stock valuations with markers for successful OS model releases.
- fullshark 2y ago
  Come on, you can do the critical thinking here to understand why these companies would want the best in class (open/closed) weight LLMs.
  - seydor 2y ago
    then why would they cheat?
    - fullshark 2y ago
      Well we'll see if they suffer consequences of this and they cheated too hard, but being perceived as best in class is arguably worth even more than being the best in class, especially if differences in performance are hard to perceive anecdotally. The goal is long term control over a technology's marketshare, as winner take all dynamics are in play here.
    - baby 2y ago
      they're all cheating, see grok
gessha 2y ago
Next on Matt Levine’s newsletter: Is Meta fudging with stock evaluation-correlated metrics? Is this securities fraud?
- myrmidon 2y ago
  If that's what it takes to get some honesty out of corporate, is it such a bad idea? Why?
- benhill70 2y ago
  Sarcasm doesn't translate well in text. Please, elaborate.
  - camjw 2y ago
    Matt Levine has a common refrain which is that basically everything is securities fraud. If you were an investor who invested on the premise that Meta was good at AI and Zuck knowingly put out a bad model, is that securities fraud? Matt Levine will probably argue that it could be in a future edition of Money Stuff (his very good newsletter).
    - NickC25 2y ago
      is it securities fraud? sort of. If Mark, both through Meta and through his own resources, has the capital to hire and retain the best AI researchers / teams, and claims he's doing so, but puts out a model that sucks, he's liable. It's probably not directly fraud, but if he claims he's trying to compete with Google or Microsoft or Apple or whoever, yet doesn't adequately deploy a comparable amount of resources, capital, people, whatever, and doesn't explain why, it could (stretch) be securities fraud....I think.
      - genewitch 2y ago
        And the fine for that? Probably 0.001% of revenue. If that.
    - jjani 2y ago
goldchainposse 2y ago
I wonder how much the current work environment contributed to this. There's immense pressure to deliver, so it's not surprising to see this.
ekojs 2y ago
I think it's most illustrative to see the sample battles (H2H) that LMArena released [1]. The outputs of Meta's model is too verbose and too 'yappy' IMO. And looking at the verdicts, it's no wonder by people are discounting LMArena rankings. [1]: https://huggingface.co/spaces/lmarena-ai/Llama-4-Maverick-03-26-Experimental_battles https://huggingface.co/spaces/lmarena-ai/Llama-4-Maverick-03...
- smeeth 2y ago
  In fairness, 4o was like this until very recently. I suspect it comes from training on COT data from larger models.
- ed 2y ago
  Yep, it’s clear that many wins are due to Llama 4’s lowered refusal rate which is an effective form of elo hacking.
goldchainposse 2y ago
In other news, the head of AI research just left https://www.cnbc.com/2025/04/01/metas-head-of-ai-research-announces-departure.html https://www.cnbc.com/2025/04/01/metas-head-of-ai-research-an...
- kridsdale3 2y ago
  I would have thought that title would belong to Yann.
  - yodsanklai 2y ago
    TBH I'm very surprised Yann Le Cun is still there. He looks to me like a free thinker and an independent person. I don't think he buys into the Trump agenda and US nationalistic anti-Europe speech like Zuck does. He may be giving Zuck the benefit of the doubt, and probably is grateful that Zuck gave him a chance when nobody else did.
    - lenerdenator 2y ago
      > TBH I'm very surprised Yann Le Cun is still there. He looks to me like a free thinker and an independent person. I don't think he buys into the Trump agenda and US nationalistic anti-Europe speech like Zuck does. He may be giving Zuck the benefit of the doubt, and probably is grateful that Zuck gave him a chance when nobody else did. Zuck doesn't buy it, either. He just knows what's good for business right now. In an example of the worst person you know making a great point, Josh Hawley said "What really struck me is that they can read an election return." [0]. Though it's worth remembering, it's very difficult to accumulate the volume of data necessary to do the current kind of AI training while sticking to the strictest interpretations of EU privacy law. Social media companies aren't just feeding the user data into marketing algorithms, they're feeding them into AI models. If you're a leading researcher in that field - Like Le Cun - and the current state-of-the-art means getting as much data as possible, you might not appreciate the regulatory environment of the EU. [0] https://www.npr.org/2025/02/27/nx-s1-5302712/senator-josh-hawley-data-privacy-censorship-doge https://www.npr.org/2025/02/27/nx-s1-5302712/senator-josh-ha...
ignoramous 2y ago
Ahmad al-Dahle, who leads "Gen AI" at Meta, wrote this on Twitter: ... We're also hearing some reports of mixed quality across different services ... We've also heard claims that we trained on test sets -- that's simply not true and we would never do that. Our best understanding is that the variable quality people are seeing is due to needing to stabilize implementations. We believe the Llama 4 models are a significant advancement and we're looking forward to working with the community to unlock their value. https://x.com/Ahmad_Al_Dahle/status/1909302532306092107 https://x.com/Ahmad_Al_Dahle/status/1909302532306092107 / https://archive.vn/JzONp https://archive.vn/JzONp
- SubiculumCode 2y ago
  There seems to be a lot of haloo, accusations, and rumors, but little meat to any of them. Maybe they rushed the release, were unsure of which one to go with, and some moderate rule bending in terms of which tune got sent to the arena, but I have seen no real hard evidence of real hard underhandedness.
labrador 2y ago
Meta does themselves a disservice by having such a crappy public facing AI for people to try (meta.ai). I regularly use the web versions for GPT 4o, Deepseek, Grok, and Google Gemeni 2.5. Meta is always the worst so I don't even bother anymore.
alittletooraph2 2y ago
I tried to make Studio Ghibli inspired images using presumably their new models. It was ass.
- echelon 2y ago
  GPT 4o images is the future of all image gen. Every other player: Black Forest Labs' Flux, Stability.ai's Stable Diffusion, and even closed models like Ideogram and Midjourney, are all on the path to extinction. Image generation and editing must be multimodal. Full stop. Google Imagen will probably be the first model to match the capabilities of 4o. I'm hoping one of the open weights labs or Chinese AI giants will release a model that demonstrates similar capabilities soon. That'll keep the race neck and neck.
  - minimaxir 2y ago
    One very important distinction between image models is the implementation: 4o is autogressive, slow, and extremely expensive. Although the Ghibli trend is market validation, I suspect that competitors may not want to copy it just yet.
    - echelon 2y ago
      > 4o is autogressive, slow, and extremely expensive. If you factor in the amount of time wasted with prompting and inpainting, it's extremely well worth it.
    - JamesBarney 2y ago
      Extremely expensive in what since? In that it costs $.03 instead of $.00003c? Yeah it's relatively far more expensive than other solutions, but from an absolute standpoint still very cheap for the vast majority of use cases. And it's a LOT better.
      - svachalek 2y ago
        Dall-E is already 4-8 cents per image. Afaik this is not in the API yet but I wouldn't be surprised if it's $1 or more.
arcastroe 2y ago
https://archive.is/Ec6V1 https://archive.is/Ec6V1
codingwagie 2y ago
The truth is that the vast majority of FAANG engineers making high six figures are only good at deterministic work. They cant produce new things, and so meta and google are struggling to compete when actual merit matters, and they cant just brute force the solutions. Inside these companies, the massive tech systems built, are actually generally terrible, but they pile on legions of engineers to fix the problems. This is the culture of META hurting them, they are paying "AI VPs" millions of dollars to go to status meetings to get dates for when these models will be done. Meanwhile, deepseek r1 has a flat hierarchy with engineers that actually understand low level computing Its making a mockery of big tech, and is why startups exist. Big company employees rise the ranks by building skill sets other than producing true economic value
- kylebyte 2y ago
  The problem is less that those high level engineers are only good at deterministic work and more that they're only rewarded for deterministic work. There is no system to pitch an idea as opening new frontiers - all ideas must be able to optimize some number that leadership has already been tricked into believing is important.
- jjani 2y ago
  > They cant produce new things, and so meta and google are struggling to compete when actual merit matters, and they cant just brute force the solutions. You haven't been keeping up. Less than 2 weeks ago, Google released a model that has crushed the competition, clearly being SotA while currently effectively free for personal use. Gemini 2.0 was already good, people just weren't paying attention. In fact 1.5 pro was already good, and ironically remains the #1 model at certain very specific tasks, despite being set for deprecation in September. Google just suffered from their completely botched initial launch way back when (remember Bard?), rushed before the product was anywhere near ready, making them look lile a bunch of clowns compared to e.g. OpenAI. That left a lasting impression on those who don't devote significant time to keeping up with newer releases.
  - codingwagie 2y ago
    gemini 2.5 pro isnt good, and if you think it is, you arent using LLMs correctly. The model gets crushed by o1 pro and sonnet 3.7 thinking. Build a large contextual prompt ( > 50k tokens) with a ton of code, and see how bad it is. I cancelled my gemini subscription
HunterX1 2y ago
Impressive results from Meta's Llama adapting to various benchmarks. However, gaming performance seems lackluster compared to specialized models like Alpaca. It raises questions about the viability of large language models for complex, interactive tasks like gaming without more targeted fine-tuning. Exciting progress nonetheless!
deleted 2y ago
[deleted]
LetsGetTechnicl 2y ago
I'm just shocked that the companies who stole all kinds of copyrighted material would again do something unethical to keep the bubble and gravy train going...
- antonkar 2y ago
  Yes, their worst fear is people figuring out that an AI chatbot is a strict librarian that spits out quotes but doesn't let you enter the library (the AI model itself). Because with 3D game-like UIs people can enter the library and see all their stolen personal photos (if they were ever online), all kind of monsters. It'll be all over YouTube. Imagine this but you remove the noise and can walk like in an art gallery (it's a diffusion model but LLMs can be loosely converted into 3D maps with objects, too): https://writings.stephenwolfram.com/2023/07/generative-ai-space-and-the-mental-imagery-of-alien-minds/ https://writings.stephenwolfram.com/2023/07/generative-ai-sp...
  - djmips 2y ago
    Do you really think facebook's model weights contain all the facebook personal photos?
    - antonkar 2y ago
      With more than 50% probability. Instagram has a clause that they can use your photos for ads and endorsements, basically for profit. Many sites repost those photos, I doubt Meta will care to meticulously remove them. If the model can generate photo-like content of people - it had photos of people put inside. I doubt they only trained on public domain photos of people. If they started doing shady stuff, why would they ever stop?
moralestapia 2y ago
LeCun making up results ... well he comes from Academia, so ...
kittikitti 2y ago
For me at least, the 10M context window is a big deal and as long as it's decent, I'm going to use it instead. I'm running Scout locally and my chat history can get very long. I'm very frustrated when the context window runs out. I haven't been able to fully test the context length but at least that one isn't fudged.
jerrygoyal 2y ago
lmarena has lost credibility for me. Are there any better alternatives out there?
casey2 2y ago
This feels like AI deniers grasping at straws. You have to prove the claim that "friendly" models do better in head to head user ratings. You also have to adequately define your claim which hasn't been done. And then you have to prove that by whatever definition of "friendliness" you've constructed made a significant difference in the benchmark. What I suspect will happen is that someone will just tell the LLM to "be curt" that may cause it's score to drop and people will unthinkingly take that to mean the above nonsensical claims are true.
badmonster 2y ago
wow, is that real?
TheJCDenton 2y ago
Last LLMs launches from top US AI labs were rather disappointing. Products seems under baked and mainly reactive... They oversold the capabilities of their future models when Deepseek landed to stay in the news at the time, but now that we look at the result it's not what everyone expected. And people are starting to question the hype, which is healthy but also very dangerous for these companies needing very large capital. It seems that some labs like Meta are unto something, but it's more research material for now than a short term product. Interesting times.
openplatypus 2y ago
Meta doing something dodgy OR unethical OR criminal ... and nobody is surprised.
floppiplopp 2y ago
"I tested myself on a subset of my training data and found that I am the greatest!"

347 points by pseudolus 2y ago

This should surprise no one..Also Goodhart's law strikes again

I believe this was designed to flatter the prompter more / be more ingratiating. Which is a worry if true (what it says about the people doing the comparing).

add-sub-mul-div 2y ago
There's no end to the possible vectors of human manipulation with this "open-weight" black box.

deckar01 2y ago

The top of that leaderboard is filled with closed weight experimental models.

etamponi 2y ago

Meta got caught _first_.

sumeno 2y ago
Not even first, OpenAI got caught a while back
- Mond_ 2y ago
  Do you have a source for this? That's interesting (if true).
  - sumeno 2y ago
    They got the dataset from Epoch AI for one of the benchmarks and pinky swore that they wouldn't train on it https://techcrunch.com/2025/01/19/ai-benchmarking-organization-criticized-for-waiting-to-disclose-funding-from-openai/ https://techcrunch.com/2025/01/19/ai-benchmarking-organizati...
    - tananaev 2y ago
      I don't see anything in the article about being caught. Maybe I missed something?
      - tedsanders 2y ago
        [flagged]
        suddenlybananas 2y ago
        >They got the dataset from Epoch AI for one of the benchmarks and pinky swore that they wouldn't train on it Is anything here actually false or do you not like the conclusions that people may draw from it?
        tedsanders 2y ago
        The false statement is “OpenAI got caught [gaming FrontierMath] a while back.” From a primary source: "OpenAI did not use FrontierMath data to guide the development of o1 or o3, at all.... we only downloaded FrontierMath for our evals long after the training data was frozen, and only looked at o3 FrontierMath results after the final announcement checkpoint was already picked." https://x.com/__nmca__/status/1882563755806281986 https://x.com/__nmca__/status/1882563755806281986
        suddenlybananas 2y ago
        Well, if we trust you. But you had the extra dataset and no one else did.
- hooloovoo_zoo 2y ago
  People have been gaming ML benchmarks as long as there have been ML benchmarks. That's why it's better to see if other researchers are incorporating a technique into their actual models rather than 'is this paper the bold entry in a benchmark table'. But it takes longer.
  - jandrese 2y ago
    When a measure becomes a target it is no longer a good measure. These ML benchmarks were never going to last very long. There is far too much pressure to game them, even unintentionally.
mkolodny 2y ago
“Got caught” is a misleading way to present what happened. According to the article, Meta publicly stated, right below the benchmark comparison, that the version of Llama on LMArena was the experimental chat version: > According to Meta’s own materials, it deployed an “experimental chat version” of Maverick to LMArena that was specifically “optimized for conversationality” The AI benchmark in question, LMArena, compares Llama 4 experimental to closed models like ChatGPT 4o latest, and Llama performs better (https://lmarena.ai/?leaderboard https://lmarena.ai/?leaderboard).

JKCalhoun 2y ago

Is LMArena junk now? I thought there was an aspect where you run two models on the same user-supplied query. Surely this can't be gamed? > “optimized for conversationality” I don't understand what that means - how it gives it an LMArena advantage.

light_hue_1 2y ago
LMArena was always junk. I work in this space and while the media takes it seriously most scientists don't. Random people ask random stuff and then it measures how good they feel. This is only a worthwhile evaluation if you're Google or Meta or OpenAI and you need to make a chartbot that keeps people coming back. It doesn't measure anything else useful.
- HPsquared 2y ago
  Conversation is a two-way street. A good conversation mechanic could elicit better interaction from the users and result in better answers. Stands to reason, anyway.
- genewitch 2y ago
  I hear AI news from time to time from the M5M in the US - and the only place I've ever seen "LMArena" is on HN and in the LM studio discord. At a ratio of 5:1 at least.
  - brandall10 2y ago
    It's mentioned quite a bit in the LLM related subreddits.
- cma 2y ago
  Llama 1 derived models on it were beating gpt 3.5 by have less refusals.
gwern 2y ago
It can be easily gamed. The users are self-selected, and they have zero incentive to be honest or rigorous or provide good responses. Some have incentives the opposite way. (There was a report of a prediction market user who said they had won a market on Gemini models by manipulating the votes; LMArena swore furiously there had definitely been no manipulation but was conspicuously silent on any details.) And the release of more LMArena responses has shown that a lot of the user ratings are blatantly wrong: either they're basically fraudulent, or LMArena's current users are people whose ratings you should be optimizing against because they are so ignorant, lazy, and superficial. At this point, when I look at the output from my Gemini-2.5-pro sessions, they are so high quality, and take so long to read, and check, and have an informed opinion on, I just can't trust the slapdash approach of LMArena in assuming that careless driveby maybe-didn't-even-read-the-responses-ain't-no-one-got-time-for-that-nerd-shit ratings mean much of anything. There have been red flags in the past and I've been taking them ever less seriously even as one of many benchmarks since early last year, but maybe this is the biggest backfire yet. And it's only going to get worse. At this rate, without major overhaul, you should take being #1 on LMArena seriously as useful and important news - as a reason to not use a model. It's past time for LMArena people to sit down and have some thorough reflection on whether it is still worth running at all, and at what point they are doing more harm than good. No benchmark lives forever and it is normal and healthy to shut them down at some point after having been saturated, but some manage to live a lot longer than they should have...

Mond_ 2y ago

The Llama 4 launch looks like a real debacle for Meta. The model doesn't look great. All the coverage I've seen has been negative. This is about what I expected, but it makes you wonder what they're going to do next. At this point it looks like they are falling behind the other open models, and made an ambitious bet on MoEs, without this paying off. Did Zuck push for the release? I'm sure they knew it wasn't ready yet.

bko 2y ago
I remember reading that they were in panic mode when the DeepSeek model came out so they must have scrambled and had to re-work a lot of things since DeepSeek was so competitive and open source as well
- blueboo 2y ago
  Fear of R2 looms large as well. I suspect they succumbed to the nuance collapse along the lines of “Is double checking results worth it if DeepSeek eats our lunch?”
nightski 2y ago
Do you know that they made a bet on MoE? Meaning they abandonded dense models? I doubt that is the case. Just releasing MoE Llama 4 does not constitute a "bet" without further information. Also from what I can tell this performs better than models with parameter counts equal to one expert, and worse than fully dense models equal to total parameter count. Isn't that kind of what we'd expect? in what way is that a failure? Maybe I am missing some details. But it feels like you have an axe to grind.
- genewitch 2y ago
  A 4x8 MOE performs better than an 8B but worse than a 32B, is your statement? My response would be, "so why bother with MOE?" However deepseek r1 is MOE from my understanding, but the "E" are all =>32B parameters. There's > 20 experts. I could be misinformed; however, even so, I'd say a MOE with 32B or even 70B experts will outperform (define this!) Models with equal parameter counts, because deepseek outperforms (define?) ChatGPT et al.

seydor 2y ago

tech companies competing over something that is losing them money is the most bizarre spectacle yet.

abc-1 2y ago
Is it really losing them money if investors throw fistfuls of cash at them for it
- deleted 2y ago
  [deleted]
FL33TW00D 2y ago
Plot big tech stock valuations with markers for successful OS model releases.
fullshark 2y ago
Come on, you can do the critical thinking here to understand why these companies would want the best in class (open/closed) weight LLMs.
- seydor 2y ago
  then why would they cheat?
  - fullshark 2y ago
    Well we'll see if they suffer consequences of this and they cheated too hard, but being perceived as best in class is arguably worth even more than being the best in class, especially if differences in performance are hard to perceive anecdotally. The goal is long term control over a technology's marketshare, as winner take all dynamics are in play here.
  - baby 2y ago
    they're all cheating, see grok

gessha 2y ago

Next on Matt Levine’s newsletter: Is Meta fudging with stock evaluation-correlated metrics? Is this securities fraud?

myrmidon 2y ago
If that's what it takes to get some honesty out of corporate, is it such a bad idea? Why?
benhill70 2y ago
Sarcasm doesn't translate well in text. Please, elaborate.
- camjw 2y ago
  Matt Levine has a common refrain which is that basically everything is securities fraud. If you were an investor who invested on the premise that Meta was good at AI and Zuck knowingly put out a bad model, is that securities fraud? Matt Levine will probably argue that it could be in a future edition of Money Stuff (his very good newsletter).
  - NickC25 2y ago
    is it securities fraud? sort of. If Mark, both through Meta and through his own resources, has the capital to hire and retain the best AI researchers / teams, and claims he's doing so, but puts out a model that sucks, he's liable. It's probably not directly fraud, but if he claims he's trying to compete with Google or Microsoft or Apple or whoever, yet doesn't adequately deploy a comparable amount of resources, capital, people, whatever, and doesn't explain why, it could (stretch) be securities fraud....I think.
    - genewitch 2y ago
      And the fine for that? Probably 0.001% of revenue. If that.
  - jjani 2y ago

goldchainposse 2y ago

I wonder how much the current work environment contributed to this. There's immense pressure to deliver, so it's not surprising to see this.

ekojs 2y ago

I think it's most illustrative to see the sample battles (H2H) that LMArena released [1]. The outputs of Meta's model is too verbose and too 'yappy' IMO. And looking at the verdicts, it's no wonder by people are discounting LMArena rankings. [1]: https://huggingface.co/spaces/lmarena-ai/Llama-4-Maverick-03-26-Experimental_battles https://huggingface.co/spaces/lmarena-ai/Llama-4-Maverick-03...

smeeth 2y ago
In fairness, 4o was like this until very recently. I suspect it comes from training on COT data from larger models.
ed 2y ago
Yep, it’s clear that many wins are due to Llama 4’s lowered refusal rate which is an effective form of elo hacking.

goldchainposse 2y ago

In other news, the head of AI research just left https://www.cnbc.com/2025/04/01/metas-head-of-ai-research-announces-departure.html https://www.cnbc.com/2025/04/01/metas-head-of-ai-research-an...

kridsdale3 2y ago
I would have thought that title would belong to Yann.
- yodsanklai 2y ago
  TBH I'm very surprised Yann Le Cun is still there. He looks to me like a free thinker and an independent person. I don't think he buys into the Trump agenda and US nationalistic anti-Europe speech like Zuck does. He may be giving Zuck the benefit of the doubt, and probably is grateful that Zuck gave him a chance when nobody else did.
  - lenerdenator 2y ago
    > TBH I'm very surprised Yann Le Cun is still there. He looks to me like a free thinker and an independent person. I don't think he buys into the Trump agenda and US nationalistic anti-Europe speech like Zuck does. He may be giving Zuck the benefit of the doubt, and probably is grateful that Zuck gave him a chance when nobody else did. Zuck doesn't buy it, either. He just knows what's good for business right now. In an example of the worst person you know making a great point, Josh Hawley said "What really struck me is that they can read an election return." [0]. Though it's worth remembering, it's very difficult to accumulate the volume of data necessary to do the current kind of AI training while sticking to the strictest interpretations of EU privacy law. Social media companies aren't just feeding the user data into marketing algorithms, they're feeding them into AI models. If you're a leading researcher in that field - Like Le Cun - and the current state-of-the-art means getting as much data as possible, you might not appreciate the regulatory environment of the EU. [0] https://www.npr.org/2025/02/27/nx-s1-5302712/senator-josh-hawley-data-privacy-censorship-doge https://www.npr.org/2025/02/27/nx-s1-5302712/senator-josh-ha...

ignoramous 2y ago

Ahmad al-Dahle, who leads "Gen AI" at Meta, wrote this on Twitter: ... We're also hearing some reports of mixed quality across different services ... We've also heard claims that we trained on test sets -- that's simply not true and we would never do that. Our best understanding is that the variable quality people are seeing is due to needing to stabilize implementations. We believe the Llama 4 models are a significant advancement and we're looking forward to working with the community to unlock their value. https://x.com/Ahmad_Al_Dahle/status/1909302532306092107 https://x.com/Ahmad_Al_Dahle/status/1909302532306092107 / https://archive.vn/JzONp https://archive.vn/JzONp

SubiculumCode 2y ago
There seems to be a lot of haloo, accusations, and rumors, but little meat to any of them. Maybe they rushed the release, were unsure of which one to go with, and some moderate rule bending in terms of which tune got sent to the arena, but I have seen no real hard evidence of real hard underhandedness.

labrador 2y ago

Meta does themselves a disservice by having such a crappy public facing AI for people to try (meta.ai). I regularly use the web versions for GPT 4o, Deepseek, Grok, and Google Gemeni 2.5. Meta is always the worst so I don't even bother anymore.

alittletooraph2 2y ago

I tried to make Studio Ghibli inspired images using presumably their new models. It was ass.

echelon 2y ago
GPT 4o images is the future of all image gen. Every other player: Black Forest Labs' Flux, Stability.ai's Stable Diffusion, and even closed models like Ideogram and Midjourney, are all on the path to extinction. Image generation and editing must be multimodal. Full stop. Google Imagen will probably be the first model to match the capabilities of 4o. I'm hoping one of the open weights labs or Chinese AI giants will release a model that demonstrates similar capabilities soon. That'll keep the race neck and neck.
- minimaxir 2y ago
  One very important distinction between image models is the implementation: 4o is autogressive, slow, and extremely expensive. Although the Ghibli trend is market validation, I suspect that competitors may not want to copy it just yet.
  - echelon 2y ago
    > 4o is autogressive, slow, and extremely expensive. If you factor in the amount of time wasted with prompting and inpainting, it's extremely well worth it.
  - JamesBarney 2y ago
    Extremely expensive in what since? In that it costs $.03 instead of $.00003c? Yeah it's relatively far more expensive than other solutions, but from an absolute standpoint still very cheap for the vast majority of use cases. And it's a LOT better.
    - svachalek 2y ago
      Dall-E is already 4-8 cents per image. Afaik this is not in the API yet but I wouldn't be surprised if it's $1 or more.

arcastroe 2y ago

https://archive.is/Ec6V1 https://archive.is/Ec6V1

codingwagie 2y ago

The truth is that the vast majority of FAANG engineers making high six figures are only good at deterministic work. They cant produce new things, and so meta and google are struggling to compete when actual merit matters, and they cant just brute force the solutions. Inside these companies, the massive tech systems built, are actually generally terrible, but they pile on legions of engineers to fix the problems. This is the culture of META hurting them, they are paying "AI VPs" millions of dollars to go to status meetings to get dates for when these models will be done. Meanwhile, deepseek r1 has a flat hierarchy with engineers that actually understand low level computing Its making a mockery of big tech, and is why startups exist. Big company employees rise the ranks by building skill sets other than producing true economic value

kylebyte 2y ago
The problem is less that those high level engineers are only good at deterministic work and more that they're only rewarded for deterministic work. There is no system to pitch an idea as opening new frontiers - all ideas must be able to optimize some number that leadership has already been tricked into believing is important.
jjani 2y ago
> They cant produce new things, and so meta and google are struggling to compete when actual merit matters, and they cant just brute force the solutions. You haven't been keeping up. Less than 2 weeks ago, Google released a model that has crushed the competition, clearly being SotA while currently effectively free for personal use. Gemini 2.0 was already good, people just weren't paying attention. In fact 1.5 pro was already good, and ironically remains the #1 model at certain very specific tasks, despite being set for deprecation in September. Google just suffered from their completely botched initial launch way back when (remember Bard?), rushed before the product was anywhere near ready, making them look lile a bunch of clowns compared to e.g. OpenAI. That left a lasting impression on those who don't devote significant time to keeping up with newer releases.
- codingwagie 2y ago
  gemini 2.5 pro isnt good, and if you think it is, you arent using LLMs correctly. The model gets crushed by o1 pro and sonnet 3.7 thinking. Build a large contextual prompt ( > 50k tokens) with a ton of code, and see how bad it is. I cancelled my gemini subscription

HunterX1 2y ago

Impressive results from Meta's Llama adapting to various benchmarks. However, gaming performance seems lackluster compared to specialized models like Alpaca. It raises questions about the viability of large language models for complex, interactive tasks like gaming without more targeted fine-tuning. Exciting progress nonetheless!

deleted 2y ago

[deleted]

LetsGetTechnicl 2y ago

I'm just shocked that the companies who stole all kinds of copyrighted material would again do something unethical to keep the bubble and gravy train going...

antonkar 2y ago
Yes, their worst fear is people figuring out that an AI chatbot is a strict librarian that spits out quotes but doesn't let you enter the library (the AI model itself). Because with 3D game-like UIs people can enter the library and see all their stolen personal photos (if they were ever online), all kind of monsters. It'll be all over YouTube. Imagine this but you remove the noise and can walk like in an art gallery (it's a diffusion model but LLMs can be loosely converted into 3D maps with objects, too): https://writings.stephenwolfram.com/2023/07/generative-ai-space-and-the-mental-imagery-of-alien-minds/ https://writings.stephenwolfram.com/2023/07/generative-ai-sp...
- djmips 2y ago
  Do you really think facebook's model weights contain all the facebook personal photos?
  - antonkar 2y ago
    With more than 50% probability. Instagram has a clause that they can use your photos for ads and endorsements, basically for profit. Many sites repost those photos, I doubt Meta will care to meticulously remove them. If the model can generate photo-like content of people - it had photos of people put inside. I doubt they only trained on public domain photos of people. If they started doing shady stuff, why would they ever stop?

moralestapia 2y ago

LeCun making up results ... well he comes from Academia, so ...

kittikitti 2y ago

For me at least, the 10M context window is a big deal and as long as it's decent, I'm going to use it instead. I'm running Scout locally and my chat history can get very long. I'm very frustrated when the context window runs out. I haven't been able to fully test the context length but at least that one isn't fudged.

jerrygoyal 2y ago

lmarena has lost credibility for me. Are there any better alternatives out there?

casey2 2y ago

This feels like AI deniers grasping at straws. You have to prove the claim that "friendly" models do better in head to head user ratings. You also have to adequately define your claim which hasn't been done. And then you have to prove that by whatever definition of "friendliness" you've constructed made a significant difference in the benchmark. What I suspect will happen is that someone will just tell the LLM to "be curt" that may cause it's score to drop and people will unthinkingly take that to mean the above nonsensical claims are true.

badmonster 2y ago

wow, is that real?

TheJCDenton 2y ago

Last LLMs launches from top US AI labs were rather disappointing. Products seems under baked and mainly reactive... They oversold the capabilities of their future models when Deepseek landed to stay in the news at the time, but now that we look at the result it's not what everyone expected. And people are starting to question the hype, which is healthy but also very dangerous for these companies needing very large capital. It seems that some labs like Meta are unto something, but it's more research material for now than a short term product. Interesting times.

openplatypus 2y ago

Meta doing something dodgy OR unethical OR criminal ... and nobody is surprised.

floppiplopp 2y ago

"I tested myself on a subset of my training data and found that I am the greatest!"