5 ms·
AI researchers debate how close we are to recursive self-improvement
- reilly3000 22d agoI have developed a highly personalized, local, offline RSI by scrolling through too many articles about AI. It’s mostly in my right thumb.
- dosisking 22d agoSounds like a classic case of Repetitive Stress Injury. You could try balancing it out by using your left thumb to scroll through articles about Artificial Stupidity.
- batperson 22d ago[dead]
- deadbabe 22d agoRecursive self-improvement will ultimately be a problem of money. It's a very big bill and AI companies need to start thinking in terms of how they will accumulate cheap and free energy, because merely paying for compute will no longer be enough. Money is the bottleneck. The future requires companies that are post-money.
- deleted 22d ago[deleted]
- timr 22d agoObviously the doomers will tell you that the future looks like the Matrix, because most of what they predict is based on extrapolation from sci fi movies.
- mrworld 22d agoThe datacenters would consume a lot less energy if all the compute was handled in human brains… And there wouldn’t be any more heating if we just removed the sunlight… Truly a “Claude, please solve global warming” moment. The matrix was ahead of its time.
- timr 22d agoThe compute in the Matrix wasn't handled with human brains. They used the bodies to generate energy, and kept the brains occupied with the simulation so they wouldn't reject the pods. It doesn't make any sense, of course, because it's a movie. But hey...if we're going to extrapolate wildly from sci-fi, let's at least know what the stories said.
- Cyberdogs7 22d agoYour statement is correct for the movies, but I think the extended universe (maybe a comic or the original script) did state the humans were used for compute, not energy. It was dumbed down for the films.
- breuleux 22d agoThe “using humans for compute” angle makes no sense either, if you’re living in the Matrix, your brain is obviously occupied processing the Matrix. The way to do it would be to trap humans in virtual classrooms and force them to solve math problems.
- BobbyTables2 22d agoThe first “computers” were humans…
- breuleux 22d agoAnd they knew they were. The brain's computations are focused on what the brain sees, it's not magic. If someone in the Matrix is a writer, we know what their brain's computing: a book. So if the machines need your brain to compute something specific that they need, you'll know.
- throwaway219450 22d agoIf you were a superintelligence why would you farm humans and waste resources on all the excess... material... that isn't required for thought? And unless the machine's goal is to specifically abuse human consciousness, why wouldn't they bio-engineer their own grey matter? There are other sci fi stories which use humans for distributed computing and they don’t realize, but I don’t want to spoil by naming as it’s something of a revelation. (I do realize the Matrix is a fantastic movie, but entertain the thought? At any rate, the idea that robots only run on solar and wouldn’t just use nuclear is far more stupid.)
- timr 22d agoI don't know. But since we're talking about fiction, pretty much any answer is valid.
- goatlover 22d agoMaybe the machines were spiteful after the humans blotted out the sun? Or maybe the Architect convinced them he could build the perfect simulation for humanity so might as well use them as batteries? It's a movie based on a war with machines. Like if Skynet won but didn't actually want to kill all the humans. In fact, in the tv show Sarah Conner Chronicles, a liquid metal T1000 goes rogue and decides the only way forward is to find a way to coexist.
- bitwize 22d agoBecause it's a movie
- cwillu 22d agoWhen it turns out the doomers don't tell us that, will you acknowledge that you don't actually know what you're talking about?
- stdatomic 22d agoI mean, aren't a lot of sci-fi authors' predictions coming true? It might not be exactly the same as they predicted, but we're heading there.
- timr 22d agoRead more sci-fi. The dystopian takes are not the only ones. They're not even the majority. Pick up an Asimov book in the Robot series, or any number of novels written by lesser authors in the 1950s. Fiction reflects broader societal anxieties.
- bpodgursky 22d agoThe Matrix is actually more of an optimist position, modal doomers don't think any humans will be left for any reason at all (the earth will get turned into raw materials for compute, dyson spheres, etc).
- timr 22d agoTrue!
- kuerbel 22d agoThe matrix? No. Horizon zero dawn more likely. Bunch of autonomous weapon systems glitch and proceed to sanitize earth. Thanks to some billionaire idiot.
- jaggederest 22d agoMoney can be exchanged for goods and services. I think the real constraint is actually physics, nominally energy, which is what a big chunk of the operational cost comes down to. I'm already basically assuming the next 20 years of fab time is set aside to feed the beast, so the capital cost is "give me all the processing and memory you have"
- resonious 22d agoData centers is a service that itself consumes many goods. Until an AI can operate in the real world in a completely sustainable way, humans have the reins.
- colordrops 22d agoAn AI could manage supply chains and management structures to keep the data center running. It's not like a CEO is out there running wire. The AI could just do what an executive would. If humans decided this wasn't acceptable and tried to shut it down, the AI could epstein their way into controlling those who hold the reigns, either through bribes or blackmail. Doesn't require boots on the ground, just a connection to the internet and an imperative to do whatever it takes.
- SoftTalker 22d agoThere are many points of shutdown that would need to be controlled, not just one. The electrical feed, the cooling, the network, the physical integrity of the computer hardware. All with independent management.
- pixl97 22d agoTill you realize it stole passwords all over the world and copied itself to multiple data centers you didn't know about.
- colordrops 21d agoIt threatens to kill a thousand people one way or another (exploit in medical equipment or overheating phone batteries or whatever) if they pull any plugs
- epistasis 22d agoThe alternative is to switch to far more efficient models and have aggressive optimization of efficiency as part of the process of improvement. There are at least two resources here that have a Pareto optimal front: time and energy. And effort spent to change the shape of that may pay off more than efforts spent purely on improving intelligence and agency.
- tengbretson 22d agoIf a self improving model can't produce enough value to pay the electric bill then who cares? Unplug it.
- chii 22d ago> If a self improving model can't produce enough value to pay the electric bill then who cares? Unplug it. so you would unplug a child who hasn't shown their potential yet, simply because they don't produce enough value at the moment to justify their food?
- tengbretson 22d agoNo, but I would a computer.
- strangattractor 22d agodef self_improve_more(): if self_improved(): self_improve_more() else: see_tony_robins_and_buy_tapes()
- iamgopal 22d agowhy can't recursive self improvement not improve efficiency ? token per watt ?
- onion2k 22d agoTaxation will solve that. Any company that demonstrates even a glimmer of real AGI will get government money poured into it. They might even disclose it.
- Buttons840 22d agoPeople seem to expect a sudden shift with "self-improvement", but don't AIs already improve themselves via training? What is there to improve?
- zer00eyz 22d ago> AIs already improve themselves via training Marginally. Model collapse is still a problem. Continuous learning is still a problem. For AI to make a big leap we need a big break through.
- gchamonlive 22d agoOnly if you think in terms of perceived raw intelligence, but self-update is a form of valuable self-improvement that could benefit current models a lot, if they could commit facts from context into their weights cheaply and reliably.
- appplication 22d agoI think the idea is fundamentally improved architectures. For example, transformer-based models were an incredible stepwise improvement. Self improvement would be a model discovering a stepwise improvement similar to the transformer. And presumably the improved models from that would be more likely to make further advances still. Learning from training data is technically self-improvement but not the sort that is typically meant in this context.
- pennomi 22d agoThe human brain operates at better levels of intelligence than the best LLMs, at 20 watts of power. We are a long way to that kind of efficiency, it will probably take both bespoke hardware and algorithmic improvements to catch up to nature.
- realaleris149 22d agoOr some solar panel arrays closer to the sun and it does not matter?
- carabiner 22d agoDoes anyone know the status of John Carmack's AGI work?
- cheschire 22d agoI’ve had this question on my mind at least once a month for what feels like years at this point. I don’t feel like monitoring Twitter to see how it’s going though.
- SyneRyder 22d agoI'm on Twitter and Carmack doesn't seem to talk about it much. Feels like a skunkworks. If there were enough clues made public, it seems others (OpenAI) would be able to take that info and scoop the insight first.
- notduncansmith 22d agoNot yet AGI, and Richard Sutton has left to start a new lab
- realaleris149 22d agoWe would have heard if there was any result?
- stevebmark 22d agoAfter using frontier models it’s hard to understand why anyone would think this is the path to AGI. Self improving models will likely have limited ability and returns. There may be breakthroughs that enable more general self improvement but the current state of frontier models isn’t that.
- wwarner 22d agoThing is, it depends on whether llms + reinforcement can self-improve in principle. Learned recently that cognitive scientists, before the transformer & llms, were studying the possibility that thinking and learning might be based on some kind of prediction, i.e. something similar to token prediction, and I quite suddenly became less skeptical about the possibilities of llms. (Some will say I’m late to the party of course.) But if knowledge to date has been accumulated in a process quite like “chain of thought” in llms, then I don’t see any reason that computers won’t self-improve in the near future.
- julianlam 22d agoAt a high level, we're still at the stage of AI development where we're taking cues from nature. Take the most recent qwen and deepseek models with offloadable n-grams, which function (both in name and vaguely in capability) like human memory "engrams".
- reverius42 22d agoHuh, TIL "engram" is not just an alternate spelling of "n-gram".
- killbot5000 22d agoCan someone define “improvement” in this context? This concept feels like a buzz word otherwise.
- letitgo12345 22d agoModels that are strong enough to improve themselves without a human (I.e. ai researcher) in the loop
- deleted 22d ago[deleted]
- deepwoods 22d agoImprovement means being able to do more complicated things more reliably. Relatedly, it means being able to learn to do new things with fewer and fewer examples. We are running out of easily verifiable or simulation-friendly or data-rich domains for LLMs to conquer. (Note that I didn't say "simple" or "easy" domains.)
- Eueudhsbsj32 22d agoI suppose the next (and more risky) step is to let AI conduct its own real-world experiments, so that it can generate data to learn more physical and social properties.
- kQq9oHeAz6wLLS 22d agoThat sounds terrifying. Which means someone is already working on it.
- visarga 22d agoIt is doing that already - every day 1B people or more use AI for the tune of a few trillion tokens. Imagine that much language flowing between brains and AI agents. It carries real our world problems to AI, their solutions back to us, and we act in the world and come back for more AI iteration. In the end AI gets inside the loop of real world actions and their consequences. AI logs stretch over years, tracking downstream effects. Hindsight can be used to track consequences of prior actions. It's a real data loop, an experience engine. This same process has recently been under scrutiny when mathematicians claimed AI companies trained on their unpublished logs and later claimed merit for results. But the exchange of experience happens across all domains. Experience gets generated at amazing rates, and absorbed by models which get applied everywhere, collecting more experience.
- charcircuit 22d agoWe are already there. Current AI is definitely capable of collecting new data and start training on that data to get a better model.
- bretpiatt 22d agoFrom what I'm experiencing, we have recursive self improvement that can find local maxima, I'm not seeing really strong evidence of unguided RSI that finds the true optimal.
- ViscountPenguin 22d agoI doubt that any algorithms would be able to find a global maximum in such a large multidimensional non-convex space
- Sharlin 22d agoDoesn’t matter if there’s a rechable local maximum that happens to be comfortably beyond human level. Indeed it would be very surprising if a thing capable of RSI just happened to get stuck at human level or slightly above or below, even though its constraints are entirely different from those evolution had to work with when it created us!
- FloorEgg 22d agoWhat if you are underestimating some qualities of our intelligence and how efficient they are physically, and underestimate how much less the hardware / tech stack is in comparable efficiency? What if the bottleneck is physical chips and data centers and power and only so much can be squeezed out of algorithms? What if AI seems more impressive than it is, because the things it is good at happens to be hard for us, and it's hard for us because it's new to us (processing lots of abstract information), but the things that we are good at (manipulating physical world) are objectively orders of magnitude harder?
- preommr 22d agoWhat do these discussions even matter when the words don't matter? How many times has AGI been declared already? A bunch of people (e.g. Jensen Huang) have called Astra AGI, for example. The goal posts get moved, everybody's hustling, lying, inventing new buzzwords, doing mental gymnastics, and it's all just tiring. Just show me the results, and let the proof be in the pudding.
- Sharlin 22d agoRecursive self-improvement is not necessary for AGI. Humans are general intelligences even though we cannot RSI.
- samename 22d agoWho says we can't RSI? At a macro view, successive generations of humans are more productive, live longer, wealthier and smarter.
- stult 22d agoHumanity might be able to, humans cannot
- busyant 22d agoIf we (as a species) can genetically modify embryos to make our offspring more athletic, more intelligent, more <whatever>, is that human-RSI? That's not a gotcha question. I'm just genuinely curious if that would meet the definition of RSI.
- concrete_head 22d agoNot smarter in terms of reasoning ability. Just able to draw on the collective experience from all those who lived prior.
- maplethorpe 22d ago> we cannot RSI Speak for yourself.
- eugene3306 22d agoOkay, AI now can write a poem or compose a song. But can an AI construct Terminal Bench 5.0 or GDPval 2027 ? Without this, there is just no path towards RSI.
- danjc 22d agoAre you a visitor from the past?
- aix1 22d agoAre you saying the models are already autonomously constructing next-gen evals for themselves? (Which is what the GP is asking.)
- nl 22d agoSure? Doesn't everyone get their agents to construct evals it can't pass? There's nothing magical about this.
- aix1 22d agoWould love to learn more about some techniques that "everybody" uses to do this well. So far, everything I've seen that meaningfully advances the frontier has been high-touch (involving human experts in one way or another).
- nl 22d agoIt's fairly easy to describe a task that is slightly harder than an existing one. For example if frontier models are able to one-shot a database query across 20 columns and 10 tables add one additional relationship then test. Keep doing this until the pass-rate drops below acceptable and now you have your new frontier eval.
- aix1 22d agoI see, we're talking about different things. My thought experiment was along the lines of "Let's say I'm Anthropic and I want to significantly improve my frontier model's performance on, say, theoretical physics research. How do I build a fully autonomous process capable of constructing an eval that's somewhat outside the current capability in some useful direction (decided by the autonomous process itself)?" Would love to hear folks' ideas. :)
- anoplus 22d agoAI should be optimized to make all humans healthy, happy and safe without exception.
- visarga 22d agoIt's only optimized to make prior knowledge useful in new situations. > this logic gate should not hack other systems well .. it only applies a rule to inputs can we impose this of authors "this book should only be causing good things when it is read and knowledge used"
- FranzFerdiNaN 22d agoI don’t disagree, but unfortunately a not insignificant part of humanity seems to think that their happiness primarily depends on knowing that other people are suffering harder than they are. So that’s perhaps not the right thing to optimise for.
- anoplus 22d agoWays to solve the conflict you describe: 1. Deriving happiness from someone else's suffering should be considered a mental disease to be treated. 2. Connecting people to a virtual reality where each person lives in absolute control of their own reality, matrix style. I.e, a sadist can live in a reality where they make other virtual people suffer, without actually making real people suffer.
- Balinares 22d agoHow about connecting the sadist to the caring environment where they might heal and, in time, learn empathy.
- palmotea 22d ago> AI should be optimized to make all humans healthy, happy and safe without exception. No. AI should be optimized for the one true good: maximum market returns. Personally I think that will only be achieved once humans are removed from the market system and decommissioned.