5 ms·
"Orion’s problems signaled to some at OpenAI that the more-is-more strategy, which had driven much of its earlier success, was running out of steam." So LLMs f
by Animats 2y ago
"Orion’s problems signaled to some at OpenAI that the more-is-more strategy, which had driven much of its earlier success, was running out of steam."
So LLMs finally hit the wall. For a long time, more data, bigger models, and more compute to drive them worked. But that's apparently not enough any more.
Now someone has to have a new idea. There's plenty of money available if someone has one.
The current level of LLM would be far more useful if someone could get a conservative confidence metric out of the internals of the model. This technology desperately needs to output "Don't know" or "Not sure about this, but ..." when appropriate.
- thrwthsnw 2y agoSeriously? All they do is produce a “confidence metric”
- emtel 2y agoBut how do they do that?
- whoisthemachine 2y agoUnfortunately, the best they can do is "This is my confidence on what someone would say given the prior context".
- sooheon 2y agoWhat someone from the past would have said.
- synapsomorphy 2y agoThe new idea is already here and it's reasoning / chain of thought. Anecdotally Claude is pretty good at knowing the bounds of its knowledge.
- threeseed 2y agoAnecdotally Claude is just as bad as every other LLM. Step into more niche areas e.g. I am trying to use it with Scala macros and at least 90% of the time it is giving code that either (a) fails to compile or (b) is just complete gibberish. And at no point ever has it said it didn't know something.
- mrbungie 2y agoYep, get into any sufficiently deep niche (i.e. actually almost any non-trivial app) and the LLM magic fades off. Yeah sure you can make a pong clone in html/js and that's mainly because there the internet is full of pong clone demos. Ask how to constraint a statsmodels lineal model in some non-standard way? It will gaslight how it is possible and make you loss time in the process.
- IAmGraydon 2y agoMaking a pong clone by telling the LLM to make a pong clone is a cute trick that sometimes works, but that's not the way anyone who understands how to properly use these tools is using them. You don't describe and app and hope the LLM builds it correctly. You have to know how to architect an application and you use the LLM to build small pieces of code. For example, you tell it to build a function that does x, takes the inputs a, b, and c and returns z. LLMs don't turn non-coders into coders. It gives actual coders superpowers.
- mrbungie 2y agoNo true scottsman fallacy. I know how to use them, but using them "correctly" still produces many errors. They suck at non-trivial code outside of standard library usage and boilerplate coding: I gave an example and parent did as well. In that regard would at least change your phrase from "actual coders" to "actual senior coders", as any junior receiving bad advice (in eternal loops as LLMs normally like to do it) is only going to make them waste time and tokens.
- briga 2y agoWhat wall? Not a week has gone by in recent years without an LLM breaking new benchmarks. There is little evidence to suggest it will all come to a halt in 2025.
- jrm4 2y agoSure, but "benchmarks" here seems roughly as useful as "benchmarks" for GPUs or CPUs, which don't much translate to what the makers of GPT need, which is 'money making use cases.'
- deleted 2y ago[deleted]
- peepeepoopoo98 2y agoO3 has demonstrated that OpenAI needs 1,000,000% more inference time compute to score 50% higher on benchmarks. If O3-High costs about $350k an hour to operate, that would mean making O4 score 50% higher would cost $3.5B (!!!) an hour. That scaling wall.
- Kuinox 2y agoWait a few month and they will have a distilled model with the same performance and 1% of the run cost.
- peepeepoopoo98 2y ago100X efficiency improvement (doubtful) still means that costs grow 200X faster than benchmark performance.
- deleted 2y ago[deleted]
- achierius 2y agoEven assuming that past rates of inference cost scaling hold up, we would only expect a 2 OoM decrease after about a year or so. And 1% of 3.5b is still a very large number.
- simonw 2y agoThe new idea is inference-time scaling, as seen in o1 (and o3 and Qwen's QwQ and DeepSeek's DeepSeek-R1-Lite-Preview and Google's gemini-2.0-flash-thinking-exp). I suggest reading these two pieces about that: - https://www.aisnakeoil.com/p/is-ai-progress-slowing-down https://www.aisnakeoil.com/p/is-ai-progress-slowing-down - best explanation I've seen of inference scaling anywhere - https://arcprize.org/blog/oai-o3-pub-breakthrough https://arcprize.org/blog/oai-o3-pub-breakthrough - François Chollet's deep dive into o3 I've been tracking it on this tag on my blog: https://simonwillison.net/tags/inference-scaling/ https://simonwillison.net/tags/inference-scaling/
- exhaze 2y agoI think the wildest thing is actually Meta’s latest paper where they show a method for LLMs reasoning not in English, but in latent space https://arxiv.org/pdf/2412.06769 https://arxiv.org/pdf/2412.06769 I’ve done research myself adjacent to this (mapping parts of a latent space onto a manifold), but this is a bit eerie, even to me.
- asadalt 2y agokinda how we do it. language is just an io interface(but also neural obv) on top of our reasoning engine.
- oceanparkway 2y agoIt’s not just a protocol buffer for concepts though (weak wharf Sapir, lakoff’s ubiquitous metaphors). Language itself is also a concept layer and plasticity and concept development is bidirectional. But (I’m not very versed in the language here re ‘latent space’) I would imagine the forward pass through layers converges towards near-token-matches before output, so you have very similar reason to token/language reasoning even in latent/conceptual reasoning? Like the neurons that nearly only respond to a single token for ex.
- ynniv 2y agoIs it "eerie"? LeCun has been talking about it for some time, and may also be OpenAI's rumored q-star, mentioned shortly after Noam Brown (diplomacybot) joining OpenAI. You can't hill climb tokens, but you can climb manifolds.
- knapcio 2y agoI’m wondering whether O3 can be used to explore its own improvement or optimization ideas, or if it hasn’t reached that point yet.
- Yizahi 2y agoTo output "don't know" a system needs to "know" too. Random token generator can't know. It can guess better and better, maybe it can even guess 99.99% of time, but it can't know, it can't decide or reason (not even o1 can "reason").
- aleph_minus_one 2y ago> Now someone has to have a new idea. There's plenty of money available if someone has one. I honestly do claim to have some ideas where I see evidence that they might work (and I do attempt to work privately on a prototype if only out of curiosity and to see whether I am right). The bad news: these ideas very likely won't be helpful for these LLM companies because they are not useful for their agenda, and follow a very different approach. So no money for me. :-( Let me put it this way: Have you ever talked to a person whose intelligence is miles above yours? It can easily become very exhausting. Thus an "insanely intelligent" AI would not be of much use for most people - it would think "too different" from such people. There do exist tasks in commerce for which an insane amount of intelligence would make a huge difference (in the sense of being positive regarding some important KPIs), but these are rare. I can imagine some applications of such (fictional) "super-intelligent" AIs in finance and companies doing some bleeding-edge scientific research - but these are niche applications (though potentially very lucrative ones). If OpenAI, Anthropic & Co were really attempting to develop some "super-smart" AI, they were working on such very lucrative niche applications where an insane amount of intelligence would make a huge difference, and where you can assume and train the AI operator to have a "Fields-medal level" intelligence.
- mnk47 2y ago> So LLMs finally hit the wall Not really. Throwing a bunch of unfiltered garbage at the pretraining dataset, throwing in RLHF of questionable quality during post-training, and other current hacks - none of that was expected to last forever. There is so much low-hanging fruit that OpenAI left untouched and I'm sure they're still experimenting with the best pre-training and post-training setups. One thing researchers are seeing is resistance to post-training alignment in larger models, but that's almost the opposite of a wall, they're figuring it out as well. > Now someone has to have a new idea OpenAI already has a few, namely the o* series in which they discovered a way to bake Chain of Thought into the model via RL. Now we have reasoning models that destroy benchmarks that they previously couldn't touch. Anthropic has a post-training technique, RLAIF, which supplants RLHF,and it works amazingly well. Combined with countless other tricks we don't know about in their training pipeline, they've managed to squeeze so much performance out of Sonnet 3.5 for general tasks. Gemini is showing a lot of promise with their new Flash 2.0 and Flash 2.0-Thinking models. They're the first models to beat Sonnet at many benchmarks since April. The new Gemini Pro (or Ultra? whatever they call it now) is probably coming out in January. > The current level of LLM would be far more useful if someone could get a conservative confidence metric out of the internals of the model. This technology desperately needs to output "Don't know" or "Not sure about this, but ..." when appropriate. You would probably enjoy this talk [0], it's by an independent researcher who IIRC is a former employee of Deepmind or some other lab. They're exploring this exact idea. It's actually not hard to tell when a model is "confused" (just look at the probability distribution of likely tokens), the challenge is in steering the model to either get back to the right track or give up and say "you know what, idk" [0] https://www.youtube.com/watch?v=4toIHSsZs1c https://www.youtube.com/watch?v=4toIHSsZs1c
- NitpickLawyer 2y ago> Not really. Throwing a bunch of unfiltered garbage at the pretraining dataset, throwing in RLHF of questionable quality during post-training, and other current hacks - none of that was expected to last forever. There is so much low-hanging fruit that OpenAI left untouched and I'm sure they're still experimenting with the best pre-training and post-training setups. Exactly! LLama3 and their .x iterations have shown that, at least for now, the idea of using the previous models to filter out the pre-training datasets and use a small amount of seeds to create synthetic datasets for post-training still holds. We'll see with L4 if it continues to hold.
- atleastoptimal 2y agothe new idea is the o series and clearly OpenAI’s main focus now. It’s advancing much faster than the GPT series
- Jean-Papoulos 2y ago> So LLMs finally hit the wall. For a long time, more data, bigger models, and more compute to drive them worked We can't say whether there is a wall, since we don't have anymore data to train on.
- az226 2y agoThe problem is data. GPT-3 was trained on 4:1 ratio of data to parameters. And for GPT-4 the ratio was 10:1. So to scale this out, GPT-5 should be 25:1. The parameter count jumped from 175B to 1.3T, which means GPT-5 should be 10T parameters and 250T training tokens. There is zero chance OpenAI has a training set of high quality data that is 250T tokens. If I had to guess, they trained a model that was maybe 3-4T in size and used 30-50T high quality tokens and maybe 10-30 medium and low quality ones. There is only one company in the world that stores the data that could get us past the wall. The training cost of the above scaled GPT-5 is 150x GPT-4, which was 25k A100 for 90 days, which poor MFU. Let’s assume they double MFU, it would mean 1M H100s. But let’s say they made algorithmic improvements, so maybe it’s only 250-500k H100s. While the training cluster size was 100k and then grew to 150k, this cluster is suggestive of a smaller model and less data. But ultimately data is the bottleneck.