5 ms·
Seeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learnin
by 10xDev 3mo ago
Seeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learning.
We are probably going to need a lot more GPUs.
- vatsachak 3mo agoI mean, theoretically you can solve every finitary problem with a brute force solution... Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far
- deleted 3mo ago[deleted]
- altcognito 3mo agoWhile I think this is true, remember as we get more efficient we just decide to scale even bigger. So more GPUs, and more efficient. I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)
- Razengan 3mo ago> We are probably going to need a lot more GPUs. Or a breakthrough in algorithms etc. The human brain, heck all bio brains, are proof that you don't need a lot of power or size for intelligence.
- altcognito 3mo ago20 watts for inference AND training!
- aeyes 3mo agoFor intelligence, I expect the next breakthrough to be colocation of memory and compute in the same chip. And we'll need much more of this memory, probably a few petabytes.
- ryandvm 3mo agoThe human brain has 80 billion neurons and a 100 trillion synapses. I think you're underselling the processing power of that warm chunk of meat. The real message of the last 15 years has actually been the opposite: if you throw enough processing power at it, intelligence emerges.
- thunky 3mo agoI think you're helping GPs point: there is a lot of efficiency gains to be made to match the processing power of the brain, given it's size and power draw.
- dbspin 3mo agoMoreover we've known for quite a while now that glial cells also participate in cognition and moderate learning (e.g.: [1]). When you take those connections into account the numbers get really staggering. 85 billion glial cells with trillions of protein channels facilitating communication between the glial syncytium [2]. [1] https://www.sciencedirect.com/science/article/pii/S1934590913000076 https://www.sciencedirect.com/science/article/pii/S193459091... [2] https://pmc.ncbi.nlm.nih.gov/articles/PMC5063692/ https://pmc.ncbi.nlm.nih.gov/articles/PMC5063692/
- deleted 3mo ago[deleted]
- orbital-decay 3mo agoThe real question is not how many "weights" the human brain has (neurons+synapses may or may not translate into "weights", and brain might be also inefficient for what it is), but rather how much evolutionary and social "compute" was necessary to pack everything into that capacity.
- baq 3mo agoIt’s still 20W. We have living proof what is possible within 20W. The message has always been clear - try to get silicon computer to be as power efficient as the brain is as it is obviously possible.
- 3mo ago
- emp17344 3mo agoThis isn’t really how it works anymore. Agents rely heavily on tool use and the agentic harness to perform tasks. Pre-training is no longer very effective.
- HDThoreaun 3mo agoI thought models werent allowed tools on arc-agi?
- bevekspldnw 3mo agoThese aren’t raw base models they are the result of a ton of RLHF and various adjustments. Bitter lesson wildly overstated in this context.
- froh 3mo agorlhf = reinforcement learning from human feedback (had to look it up)
- visarga 3mo agoI think it's more RLVR (reinforcement learning from verified rewards). The RLHF is just to align models to human preferences, meaning to behave nice.
- redanddead 3mo agoWhat makes you say that
- versteegen 3mo agoMore accurate to say RLHF aligns models to human preferences, most significantly to be helpful.
- fastball 3mo agoMore RLHF is in fact scaling.
- bevekspldnw 3mo agoYes, but not in the “dump another chunk of all written language in the bucket and stir”-sense which is what bitter lesson became synonymous with. That may not be the intent of the original article, but over the past few years that’s what the phrase turned into.
- avarun 3mo ago
- Salgat 3mo agoKind of refreshing though that the "throw more processing at it" scaling we saw in the 90s has returned in a different way. For a while we were really bottlenecked in our advances by relatively low levels of parallelism (most software used by your average user doesn't scale cleanly with more than a few threads).
- dyauspitr 3mo agoAnd dozens of data centers in every state so tokens are dirt cheap.
- energy123 3mo ago> Dramatic difference Isn't this just the difference between getting 0 right and getting 1 right?
- navigate8310 3mo agoThere goes my plan to buy a PC for the next decade
- echelon 3mo agoThe whole of knowledge work is being automated. We've barely begun to see the GPU build out. This is just the start. I'd imagine they're going to 10x this, maybe 100x this.
- y1n0 3mo agoyeah, spacex's planned million satellite datacenter-in-space constellation doesn't seem as absurd.
- Schlagbohrer 3mo agoSam Altman was saying we'll eventually need to build a dyson sphere. We'll see how far human society can withstand this kind of escalation. I'm guessing one more year before there are major fractures in the basic economic/social/political relationship that get so bad it actually prevents more buildout.
- davidpapermill 3mo agoI actually think we're in a strange situation with AI compute. Right now, we have models that are statistical models of language, with a world model and reasoning "falling out" of a lot of effort. It's like we've made something that's a little bit intelligent, and now we're trying to amplify that trick to create something that's quite intelligent. And - don't get me wrong - it works. But it's also super, super inefficient. We're having machines "think out loud" to compensate for the quality of their thought processes. We elongate the path to make up for the progress made on a given step. I tink there's probably a much smarter way of doing things that will require qualitative architectural (and quite possibly hardware) innovations. Right now we're on the path to a Dyson sphere: that's probably not going to be necessary once we figure out a smarter way to think.
- orbital-decay 3mo agoOr a lot better efficiency.
- dmitriy_ko 3mo agoAnd a lot more electricity to power them.
- XCSme 3mo agoNot always, in some cases, changing to a higher reasoning makes the AI doubt itself too much, and skip over the correct answer by overcomplicating the problem and polluting the context. It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.
- andai 3mo agoI think I have this problem but with my human brain.
- Schlagbohrer 3mo agoThis shows up in OpenAI's graphs on their announcement page. There is a peak performance datapoint in the graphs past which (to the right on the graph indicating more resources spent) peformance declines. And it's on every graph on that page!
- XCSme 3mo agoAnd in my tests, that point of "overthinking" depends on the problem's complexity, so it's not necessarily that using "xhigh" is always bad or good.
- hyperbovine 3mo agoOr a new model. The human brain does far more with far less.
- ld4nt3 3mo agoYes an no current models can read and output much more faster with differing quality tho.
- andai 3mo agoI said a few months ago, "man, Opus is great, but sometimes when talking with it I have the feeling like, this thing should be about 10 times bigger." When Mythos was announced after that, I was pleasantly surprised to hear about it. But when it turned out to be only two times bigger, I was a little disappointed! (I am even more disappointed with the safety filters, but that's kind of a separate discussion... "Fortunately" I find that I can usually edit my prompt by single character and get through...)
- Schlagbohrer 3mo agoWhat do you mean bigger? Bigger functional context window?
- nbardy 3mo agoWe’re definitely going to need a lot of Gpu’s