12 ms·
Grok 3: Another win for the bitter lesson
- deleted 2y ago[deleted]
- vasco 2y agoThat's not what "the exception that proves the rule" means.
- deleted 2y ago[deleted]
- huijzer 2y agoIn general from a formal logic perspective the whole idea of “an exception that proves the rule” is flawed. If the statement was “an exception that disproves the rule”, then I would agree.
- OccamsMirror 2y ago"The exception that proves the rule" does not mean that an exception confirms a rule in a logical sense. Instead, it originates from legal and linguistic contexts where an explicit exception implies the existence of a general rule. E.g. a sign that says "No parking on Sundays" implies that the rule is that parking is fine on other days.
- huijzer 2y agoFor years I didn't know. Finally. Thanks!
- throw310822 2y agoI always thought it meant "the exception (to the rule-of-thumb) proves the (hard, correct) rule".
- AndrewDucker 2y agoIt's used in multiple senses, so it's almost impossible to tell what the person using it means. https://en.wikipedia.org/wiki/Exception_that_proves_the_rule?wprov=sfla1 https://en.wikipedia.org/wiki/Exception_that_proves_the_rule...
- vasco 2y agoIt's only flawed because you are also using it wrong!
- searealist 2y ago[flagged]
- GaggiX 2y agoThe bitter lesson is about the fact that general methods that leverage computation are ultimately the most effective. Grok 3 is not more general than DeepSeek or OpenAI models so mentioning the bitter lesson here doesn't make much sense, it's just the scaling law.
- aqueueaqueue 2y agoHow bitter is the bitter lesson when throwing more compute js costing billions. Maybe the bitter lesson is more about money now than the hardware. You are scaling up investments not just relying on moores law. But I think there is a path for the less power consuming models that people can run affordably without VC money.
- tankenmate 2y agoDon't forget it isn't just money; there's a whole heap of infrastructure that already exists to make this cheaper (standards for electricity, production scale for said standards, etc) but also socio-economic infrastructure (rule of law, etc). Imagine trying to build the same scale of infrastructure in a country with rampant corruption or in a war zone. In particular from a strategic level consider all the benefits of Mahan's "Influence of Sea Power"[0] and the benefit nations with suitable sea power / access to global markets has on being able to build the necessary infrastructure at scale quickly in order to support such endeavours. And this isn't just about raw "Sea Power" but all the concomitant requirements to achieve this, in particular the "Number of population", "Character of the people", and "Character of the government". [0] Alfred Thayer Mahan; "The Influence of Sea Power on History"
- ttoinou 2y agoI agree with you, it’s never really about the money. By definition, money is the cheapest good in the economic equation
- az226 2y agoThe bitter lesson is all about more data/compute. Doesn’t matter if it costs more because it’s not as efficient, the scale is what matters at the end of the day. Sure you’d reach farther with both scale and optimization, but raw scale still takes the cake.
- smy20011 2y agoDid they? Deepseek spent about 17 months achieving SOTA results with a significantly smaller budget. While xAI's model isn't a substantial leap beyond Deepseek R1, it utilizes 100 times more compute. If you had $3 billion, xAI would choose to invest $2.5 billion in GPUs and $0.5 billion in talent. Deepseek, would invest $1 billion in GPUs and $2 billion in talent. I would argue that the latter approach (Deepseek's) is more scalable. It's extremely difficult to increase compute by 100 times, but with sufficient investment in talent, achieving a 10x increase in compute is more feasible.
- stpedgwdgfhgdd 2y agoLarge amounts of teams are very hard to scale. There is a reason why startups innovate and large companies follow.
- sigmoid10 2y ago>It's extremely difficult to increase compute by 100 times, but with sufficient investment in talent, achieving a 10x increase in compute is more feasible. The article explains how in reality the opposite is true. Especially when you look at it long term. Compute power grows exponentially, humans do not.
- smy20011 2y agoHuman do write code that scalable with compute. The performance is always raw performance * software efficiency. You can use shitty software and waste all these FLOPs.
- llm_trw 2y agoIf the bitter lesson were true we'd be getting sota results out of two layer neural networks using tanh as activation functions. It's a lazy blog post that should be thrown out after a minute of thought by anyone in the field.
- sigmoid10 2y agoThat's not how the economics work. There has been a lot of research that showed how deeper nets are more efficient. So if you spend a ton of compute money on a model, you'll want the best output - even though you could just as well build something shallow that may well be state of the art for its depth, but can't hold up with the competition on real tasks.
- ArtTimeInvestor 2y agoIt looks like the USA is bringing all technology in-house that is needed to build AI. TSMC has a factory in the USA now, ASML too. OpenAI, Google, xAI and Nvidia are natively in the USA. While no other country is even close to build AI on their own. Is the USA going to "own" the world by becoming the keeper of AI? Or is there an alternative future that has a probability > 0?
- OccamsMirror 2y agoAre LLMs really going to own the world?
- ArtTimeInvestor 2y agoIt looks like neural network based software is to surpass humans in intelligence in every task in the forseeable future. If one country moves along this direction faster than the others, no country will stand a chance to compete with them militarily or economically.
- viraptor 2y ago> no country will stand a chance to compete with them militarily or economically. It really depends on how they go about it. It can easily instead end up with lots of people without work, no social security and disillusioned with the country. Instead of being economically great, the country may end up fighting uprisings and sabotage.
- hagbarth 2y agoHow so? First of all, assuming ASI is developed, as it stands now, it will be owned by a private corporation, not a nation state. ASI also will not be magic. Like what exactly would it be doing that enables the country to subject the others? Develop new weapons? We already have the capability to destroy earth. Actually come to think of it, if ASI is an existential threat to other nations, maybe the rational action would be to nuke whichever country develops it first. To safe the world. You see what I am saying? There is such a thing as the real world with real constraints.
- petesergeant 2y agoThe author has come up with their own, _unusual_ definition of the bitter lesson. In fact, as a direct quote, the original Is: > Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation. eg: “the study of linguistics doesn’t help you build an LLM” or “you don’t need to know about chicken physiology to make a vision system that tells you how old a chicken is” The author then uses a narrow and _unusual_ definition of what computation _means_, by saying it simply means access to fast chips, rather than the work you can perform on them, which would obviously include how efficiently you use them. In short, this article misuses two terms to more simply say “looks like the scaling laws still work”.
- s1mplicissimus 2y agooh what a surprise, a new model performs better on barcharts than the old models. yaawn
- user14159265 2y agoIt will be interesting to see how talent acquisition evolves. Many great engineers were put off by strong DEI-focused PR, and even more oppose the sudden opportunistic shift to the right. Will Muslims continue to want to work for Google? Will Europeans work for X? Some may have previously avoided close relations with China for ethical reasons—will the same soon apply to the US?
- sigmoid10 2y agoThanks to a particular culture in the US, those companies have a very strong argument against morals: money. If you can make $440k per year as an engineer at xAi, do you really care what your boss says in public? How many people sit in jobs right now getting paid much less and still have to endure similar shit from their execs?
- giladvdn 2y agoIf you're an engineer capable of holding a $440k/year job then you should care where your talent goes and who benefits from it. There are plenty of places that will pay you a good salary where the boss isn't trying to badly play at world domination.
- NitpickLawyer 2y ago> There are plenty of places that will pay you a good salary where the boss isn't trying to badly play at world domination. Ah, yes. Google? Meta? Amazon? Microsoft? Hahaha. You're right, they aren't doing it in the open, and some certainly aren't doing a bad job about it. But they are all playing at world domination.
- crimsoneer 2y agoThis isn't really true though... The US tech sector owns big salaries right now.
- lynx97 2y ago[dead]
- rfoo 2y agoI'm pretty skeptical of that 75% on GPQA Diamond for a non-reasoning model. Hope that xAI can make Grok 3 API available next week so I can run it against some private evaluations to see if it's really this good. Another nit-pick: I don't think DeepSeek had 50k Hopper GPUs. Maybe they have 50k now after getting the world's attention and having national-sponsored grey market back them, but that 50k number is certainly dreamed-up. During the past year DeepSeek's intern recruitment ads always just mentioned "unlimited access to 10k A100s", suggesting that they may have very limited H100/H800s, and most of their research ideas were validated on smaller models on an Ampere cluster. The 10k A100 number matches with a cluster their parent hedge fund company announced a few years ago. All in all my estimation is they had more (maybe 20k) A100s, and single-digit thousands of H800s.
- kgwgk 2y ago> my estimation is they had more (maybe 20k) A100s, and single-digit thousands of H800s. Their technical report on DeepSeek-V3 says that it "is trained on a cluster equipped with 2048 NVIDIA H800 GPUs." If they had even high-single-digit thousands of H800s they would have probably used more computing power instead of waiting almost two months.
- riku_iki 2y ago> I'm pretty skeptical of that 75% on GPQA Diamond for a non-reasoning model. could that benchmark be simply leaked to training data as many others?
- viraptor 2y agoThis is a weird takeaway from the recent changes. Right now companies can scale because there's stupid amount of stupid money flowing into the AI craze, but that's going to end. Companies are already discovering the issues with monetising those systems. Sure, they can "let go" and burn the available cash, but the investors will come knocking finally. Since everyone figures out similar tech anyway, it's the people with most tech improvement experience that will be in the best position long term, while openai will be stuck trying to squeeze adverts and monitoring into their chat for cash flow.
- az226 2y agoUntil we see progress slowing down, I don’t see venture capital disappearing in the race to ASI and beyond.
- podgorniy 2y agoAdjustment: not progress, the hype. People belive that LLM progress will become foundation of the future economy expansion the same way as microelectronics did. But for now there are few signs of that economic benefit from AI/LLM stuff. If one do math of what productivity increase tech should give in order to have positive ROI, one would be surprised how reality is far from feasibility of investments https://www.bruegel.org/working-paper/tension-between-exploding-ai-investment-costs-and-slow-productivity-growth https://www.bruegel.org/working-paper/tension-between-explod.... Yes, anecdotally people tell stories how they can code twice/trice/ten times faster, or how they atumated their whole workflow or replaced support with LLM. But that's far not enough for AI investment feasibility in existing businesses (AI startups will flurish for a while on venture money). Also anecdotally there are many failed attempts to replace people with LLMs (like mcdonalds ordering which resulted in crazy orders). So what we have is a hype on top of beliefs in progress as continious phenomena. But progress itself has slowed greately. Where are all breakthroughs which change our way of living? Pocket computers and consumer electronics (which is not a discovery rather an optimisation) and internet (also more about scaling than inventing) were the last. 3d printing, cancer treatment, robotics thought to be the new factors. Till AI/LLM. Now AIs/LLMs are the last resourt for believers in progress and technooptimists like Musk.
- Amekedl 2y agoanother ai hype blog entry. Not even a mention of the differently colored bars on the benchmark result. For me, grok-3 does not prove/disprove scaling laws in any meaningful capacity.
- readthenotes1 2y agoI had to ask Grok 3 what the bitter lesson was. It gave a plausible answer (compute scale beats human cleverness)
- bambax 2y agoThis article is weak and just general speculation. Many people doubt the actual performance of Grok 3 and suspect it has been specifically trained on the benchmarks. And Sabine Hossenfelder says this: > Asked Grok 3 to explain Bell's theorem. It gets it wrong just like all other LLMs I have asked because it just repeats confused stuff that has been written elsewhere rather than looking at the actual theorem. https://x.com/skdh/status/1892432032644354192 https://x.com/skdh/status/1892432032644354192 Which shows that "massive scaling", even enormous, gigantic scaling, doesn't improve intelligence one bit; it improves scope, maybe, or flexibility, or coverage, or something, but not "intelligence".
- melodyogonna 2y agoHow can it be specifically trained on benchmarks when it is leading on blind chatbot tests? The post you quoted is not a Grok problem if other LLMs are also failing, it seems, to me, to be a fundamental failure in the current approach to AI model development.
- bearjaws 2y agoAny LLM that is uncensored does well on Chatbot tests because a refusal is an automatic loss. And since 30% of people using Chatbots are Gooning it up theres far more refusals...
- pyinstallwoes 2y agoGooning?
- bearjaws 2y agohttps://www.urbandictionary.com/define.php?term=gooning https://www.urbandictionary.com/define.php?term=gooning
- nycdatasci 2y agoI think a more plausible path to gaming benchmarks would be to use watermarks in text output to identify your model, then unleash bots to consistently rank your model over opponents.
- sylware 2y agoIs the next step ML-inference fusion? aka artificial small brain?
- _giorgio_ 2y agoGrok is the best LLM on https://lmarena.ai/ https://lmarena.ai/. --- No benchmarks involved, just user preference. Rank* (UB)Rank (StyleCtrl)ModelArena Score95% CIVotesOrganizationLicense 1 1 chocolate (Early Grok-3) 1402 +7/-6 7829 xAI Proprietary 2 4 Gemini-2.0-Flash-Thinking-Exp-01-21 1385 +5/-5 13336 Google Proprietary 2 2 Gemini-2.0-Pro-Exp-02-05 1379 +5/-6 11197 GoogleProprietary
- nickfromseattle 2y agoSide question, let's say Grok is comparable in intelligence to other leading models. Will any serious business switch their default AI capabilities to Grok?
- ssijak 2y agoWhy not?
- chank 2y agoBecause they already have something that works. Why switch if theres no advantage.
- tucnak 2y agoThe API is compatible, and even if it weren't, it wouldn't matter anyway; everybody has been writing OpenAI API-compatible proxies, including Google, for months now. The only thing that matters is availability, throughput, cost per token (Google is ahead of everyone here: Vertex API is insanely cheap for what it does, Batch API at 50% discount, Prompt Caching at 75%, fully multimodal, better performance in multilingual tasks so actually useful outside the U.S. etc etc etc)
- cowpig 2y agoBecause people want to protect their businesses? I don't even think it makes sense to use the closed, API-access models for core business functions. Seems like an existential risk to be handing over that kind of information to a third party in the age of AI. But handing it over to the guy who is simultaneously running for-profit companies that contract with the government and seizing control of the Treasury? To a guy who has a long history of SEC issues, and who was caught cheating at a video game just to convince everyone he's a genius? That just seems incredibly short-sighted.
- dmix 2y ago? there's tons of businesses and startups building on closed models already the risk is mitigated by competition in the space and commoditization of LLMs over time. This isn't like building an app solely on Facebook platform or something. It's very easy to swap models. There's platforms that let you use multiple models with the same interface, etc.
- dubeye 2y agoI use chat gpt for general brain dumping I've compared my last week's queries and prefer Grok 3
- cowpig 2y agoI haven't seen grok3 on any benchmark leaderboard other than lm arena. Has anyone else?
- Rochus 2y agoInteresting, but I think the article’s argument for the "bitter lesson" relies on logical fallacies. First, it misrepresents critics of scaling as dismissing compute entirely, then frames scaling and optimization as mutually exclusive strategies (which creates a false dilemma), ignoring their synergy. E.g. DeepSeek’s algorithmic innovations under export constraints augmented - and not replaced - the scaling efforts. The article also overgeneralizes from limited cases, asserting that compute will dominate the "post-training era" while overlooking potential disruptors like efficient architectures. The CEO's statements are barely suited to support its claims. A balanced view aligning with the "bitter lesson" should recognize that scaling general methods (e.g. learning algorithms) inherently requires both compute and innovation.
- bccdee 2y agoThe creation of a model which is "co-state-of-the-art" (assuming it wasn't trained on the benchmarks directly) is not a win for scaling laws. I could just as easily claim out that xAI's failure to significantly outperform existing models despite "throwing more compute at Grok 3 than even OpenAI could" is further evidence that hyper-scaling is a dead end which will only yield incremental improvements. Obviously more computing power makes the computer better. That is a completely banal observation. The rest of this 2000-word article is groping around for a way to take an insight based on the difference between '70s symbolic AI and the neural networks of the 2010s and apply it to the difference between GPT-4 and Grok 3 off the back of a single set of benchmarks. It's a bad article.
- horsawlarway 2y agoI agree. There's a lot of attention being paid to metrics that often don't align all that well with actual production use-cases, and frankly the metrics are good but hardly breath-taking. They have an absolutely insane outlay of additional compute, which appears to have given them a relatively paltry increase in capabilities. 15 times the compute for 5-15% better performance is basically the exact opposite of the bitter lesson. Hell - it genuinely seems like the author didn't even read the actual bitter lesson. The lesson is not "scale always wins" the lesson was "We have to learn the bitter lesson that building in how we think we think does not work in the long run." And somewhat ironically - the latest advances seem to genuinely undermine the lesson. It turns out that building in reasoning/thinking (a heuristic that copies human behavior) is the biggest performance jump we've seen in the last year. Does that mean we won't scale out of the current capabilities? No, we definitely might. But we also definitely might not. The diminishing returns we're seeing for scale hint strongly that just throwing more compute at the problem is not enough by itself. Possibly still required, but definitely not sufficient.
- starspangled 2y ago> The creation of a model which is "co-state-of-the-art" (assuming it wasn't trained on the benchmarks directly) is not a win for scaling laws. Just based on the comparisons linked in the article, it's not "co-state-of-the-art", it's the clear leader. You might argue those numbers are wrong or not representative, but you can't accept them then claim it's not outperforming existing models.
- PaulHoule 2y agoInference cost rules everything around me.
- graycat 2y ago> Grok 3 performs at a level comparable to, and in some cases even exceeding, models from more mature labs like OpenAI, Google DeepMind, and Anthropic. It tops all categories in the LMSys arena and the reasoning version shows strong results—o3-level—in math,.... "Math"? Fields Medal level? Tenure? Ph.D.? ... high school plane geometry??? As in 'Grok 3 AI and Some Plane Geometry' at https://news.ycombinator.com/item?id=43113949 https://news.ycombinator.com/item?id=43113949 Grok 3 failed at a plane geometry exercise.
- thatgerhard 2y agoI've been using Grok3 with deep think for 2 days now and the things it built has been waaaaay passed any other LLM I've tried
- greyjoyduck 2y agoYour reasoning is extremely vague and you praise the hell out of musk and xai for some weird reason...