21 ms·
AGI is an engineering problem, not a model training problem
- __loam 1y agoAll those guys linking the bitter lesson aren't gonna like hearing this lol
- therobots927 1y agoThey'll keep their heads in the sand for as long as they can.
- cauliflower2718 1y agoIt seems like all of the links to more of their work (e.g. "research on deterministic vs. probabilistic systems") are currently broken.
- nerevarthelame 1y agoIn each of the URLs, replace "blog" with "posts"
- sublinear 1y agoWay out of touch. AGI is poorly defined and thus is a science "problem", and a very low priority one at that. No amount of engineering or model training is going to get us AGI until someone defines what properties are required and then researches what can be done to achieve them within our existing theories of computation which all computers being manufactured today are built upon.
- Loughla 1y agoIt strikes me that until we fully understand human consciousness, we don't stand a chance of reaching AGI. Am I incorrect?
- root_axis 1y agoThat doesn't seem like a useful assumption since consciousness doesn't have a functional definition (even though it might have a functional purpose in humans)
- mdp2021 1y agoIntelligence (solving problems) does not require consciousness.
- Loughla 1y agoPlease elaborate.
- kelnos 1y agoI'm not sure I'd give such an absolute statement of certainty as the GP, but there is little reason to believe that consciousness and intelligence need to go hand-in-hand. On top of that, we don't really have good, strong definitions of "consciousness" or "general intelligence". We don't know what causes either to emerge from a complex system. We don't know if one is required to have the other (and in which direction), or if you can have an unintelligent consciousness or an unconscious intelligence.
- sowbug 1y agohttps://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
- mdp2021 1y agoYou do not need to implement consciousness into a calculator. There exist forms of intelligence that are just sophisticated calculation - no need for consciousness.
- sublinear 1y agoI think we can relax that a bit. We "just" need to understand some definition of cognition that satisfies our computational needs. Natural language processing is definitely a huge step in that direction, but that's kinda all we've got for now with LLMs and they're still not that great. Is there some lower level idea beneath linguistics from which natural language processing could emerge? Maybe. Would that lower level idea also produce some or all of the missing components that we need for "cognition"? Also a maybe. What I can say for sure though is that all our hardware operates on this more linguistic understanding of what computation is. Machine code is strings of symbols. Is this not good enough? We don't know. That's where we're at today.
- karmakaze 1y agoThis is the funniest "I'm a hammer thus AGI is a nail" post I've ever read.
- throwaway6977 1y agoI think that completely discounting the potential of new emergent capabilities at scale undermines this thesis significantly. We don't know until someone tries, and there is compelling evidence that there's still plenty of juice to squeeze out of both scale and engineering.
- crooked-v 1y agoThis article seems to me like a lot of "if you solve all the hard problems, then you'll have a solution". Which is like... yes, and...?
- mdp2021 1y agoIt's relevant given the apparent conservativeness about LLMs, the non-revolutionary architectural improvements that seemed to place too much of the bets on scaling.
- jibal 1y agoIt lays out what those problems are and how LLMs don't solve them.
- slashdave 1y agoThe article doesn't even discuss hard problems. An unfortunate tendency that many in high-tech suffer from is the idea that any problem can be solved with engineering.
- mattfrommars 1y agoI don't you know about you guys but Sam Altman have said they have achieved AGI within OpenAI. That's big.
- mdp2021 1y agoIf "context is king" in the LLMs age... Well give us at least some context.
- daveguy 1y agoWell, they've said they're close over and over again. Maybe that final bit of tech to make AGI a reality will finally ride into existence on the sub-$30k Tesla.
- kelnos 1y agoI'll believe it when I see it, and I'm very much in doubt they have anything to show.
- jibal 1y agoHow is it "big" that Altman told one of his many lies? He now says that AGI "is not a useful term".
- ChicagoDave 1y agoNo one has invented Asimov’s positronic brain or anything like it. We don’t even know how.
- glitchc 1y agoWe don't know if AGI is even possible outside of a biological construct yet. This is key. Can we land on AGI without some clear indication of possibility (aka Chappie style)? Possibly, but the likelihood is low. Quite low. It's essentially groping in the dark. A good contrast is quantum computing. We know that's possible, even feasible, and now are trying to overcome the engineering hurdles. And people still think that's vaporware.
- sixo 1y agoOn the contrary, we have one working example of general intelligence (humans) and zero of quantum computing.
- deleted 1y ago[deleted]
- bee_rider 1y agoDo we have a specific enough definition of general intelligence that we can exclude all non-human animals?
- mattnewton 1y agoWhy does it need to exclude all non human animals? Could it not be a difference of degree rather than of kind?
- bee_rider 1y agoThe post I was responding to had > On the contrary, we have one working example of general intelligence (humans) I think some animals probably have what most people would informally call general intelligence, but maybe there’s some technical definition that makes me wrong.
- mitthrowaway2 1y agoTheir point is not in any way weakened if you read "one working example" as "at least one working example".
- dinobones 1y agoNah,this sounds like a modern remix of Japan’s Fifth Generation Computing project. They thought that by building large databases and with Prolog they would bring upon an AI renaissance. Just hand waving some “distributed architecture” and trying to duct tape modules together won’t get us any closer to AGI. The building blocks themselves, the foundation, has to be much better. Arguably the only building block that LLMs have contributed is that we have better user intent understanding now; a computer can just read text and extract intent from it much better than before. But besides that, the reasoning/search/“memory” are the same building blocks of old, they look very similar to techniques of the past, and that’s because they’re limited by information theory / computer science, not by today’s hardware or systems.
- bfung 1y agoYep, the Attention mechanism in the Transformer arch is pretty good. Probably need another cycle of similar breakthrough in model engineering before this more complex neural network gets a step function better. Moar data ain’t gonna help. The human brain is the proof: it doesnt need the internet’s worth of data to become good (nor all that much energy).
- imiric 1y agoRight. We can certainly get much more utility out of current architectures with better engineering, as "agents" have shown, but to claim that AGI is possible with engineering alone is wishful thinking. The hard part is building systems that showcase actual intelligence and reasoning, that are able to learn and discover on their own instead of requiring exorbitantly expensive training, that don't hallucinate, and so on. We still haven't cracked that nut, and it's becoming increasingly evident that the current approaches won't get us there. That will require groundbreaking compsci work, if it's possible at all.
- levitate 1y ago"AGI needs to update beliefs when contradicted by new evidence" is a great idea, however, the article's approach of building better memory databases (basically fancier RAG) doesn't seem enable this. Beliefs and facts are built into LLMs at a very low layer during training. I wonder how they think they can force an LLM to pull from the memory bank instead of the training data.
- mdp2021 1y ago> how they think they can force an LLM to pull from the memory bank instead of the training data You have to implement procedurality first (e.g. counting, after proper instancing of ideas).
- jibal 1y agoLLMs are not the proposed solution. (Also, LLMs don't have beliefs or other mental states. As for facts, it's trivially easy to get an LLM to say that it was previously wrong ... but multiple contradictory claims cannot all be facts.)
- therobots927 1y agoThe problem is that if it's an engineering problem then further advancement will rely on step function discoveries like the transformer. There's no telling when that next breakthrough will come or how many will be needed to achieve AGI. In the meantime I guess all the AI companies will just keep burning compute to get marginal improvements. Sounds like a solid plan! The craziest thing about all of this is that ML researchers should know better!! Anyone with extensive experience training models small or large knows that additional training data offers asymptotic improvements.
- kelnos 1y agoI think the LLM businesses as-is are potentially fine businesses. Certainly the compute cost of running and using them is very high, not yet reflected in the prices companies like OpenAI and Anthropic are charging customers. It remains to be seen if people will pay the real costs. But even if LLMs are going to tap out at some point, and are a local maximum, dead-end, when it comes to taking steps toward AGI, I would still pay for Claude Code until and unless there's something better. Maybe a company like Anthropic is going to lead that research and build it, or maybe (probably) it's some group or company that doesn't exist yet.
- therobots927 1y agoAnthropic isn't even breaking even, and even if they do become profitable it's a far cry from AGI
- JSR_FDED 1y ago“Potentially” is doing some heavy lifting here. As it stands currently, the valuations of these LLM businesses imply that they will be able to capture a lot of the generated value. But the open source/weights offerings, and competition from China and others makes me question that. So I agree these could be good businesses in theory, but I doubt whether the current business model is a good one.
- CuriouslyC 1y agoWe need to stop giving a shit about AGI and just try to build progressively better systems and enjoy the ride.
- andy99 1y agoIf you believe the bitter lesson, all the handwavy "engineering" is better done with more data. Someone likely would have written the same thing as this 8 years ago about what it would take to get current LLM performance. So I don't buy the engineering angle, I also don't think LLMs will scale up to AGI as imagined by Asimov or any of the usual sci-fi tropes. There is something more fundamental missing, as in missing science, not missing engineering.
- deleted 1y ago[deleted]
- Bukhmanizer 1y agoThis isn’t really what the bitter lesson says.
- hnuser123456 1y agoEven more fundamental than science, there is missing philosophy, both in us regarding these systems, and in the systems themselves. An AGI implemented by an LLM needs to, at the minimum, be able to self-learn by updating its weights, self-finetune, otherwise it quickly hits a wall between its baked-in weights and finite context window. What is the optimal "attention" mechanism for choosing what to self-finetune with, and with what strength, to improve general intelligence? Surely it should focus on reliable academics, but which academics are reliable? How can we reliably ensure it studies topics that are "pure knowledge", and who does it choose to be, if we assume there is some theoretical point where it can autonomously outpace all of the world's best human-based research teams?
- fragmede 1y agoA system that self-updates its weights is so obvious the only question is who will be the first to get there?
- tmountain 1y agoI’m no expert, but it seems like self updating weights requires a grounded understanding of the underlying subject matter, and this seems like a problem current LLM systems.
- kingkawn 1y agoNow that there’s a fundamental technical framework for producing something like coherence, the ability to make a reliable, persistent personality will require new insights into how our own minds take shape, not just ever more data in the firehose
- blibble 1y agono, it's a research problem the idea that you would somehow produce intelligence by feeding billions of reddit comments into a statistical text model is will go down as the biggest con in history (so far)
- xyzzy123 1y agoAm I the only one who feels that Claude Code is what they would have imagined basic AGI to be like 10 years ago? It can plan and take actions towards arbitrary goals in a wide variety of mostly text-based domains. It can maintain basic "memory" in text files. It's not smart enough to work on a long time horizon yet, it's not embodied, and it has big gaps in understanding. But this is basically what I would have expected v1 to look like.
- zdragnar 1y agoClaude code is neither sentient nor sapient. I suspect most people envision AGI as at least having sentience. To borrow from Star Trek, the Enterprise's main computer is not at the level of AGI, but Data is. The biggest thing that is missing (IMHO) is a discrete identity and notion of self. It'll readily assume a role given in a prompt, but lacks any permanence.
- kelseyfrog 1y agoMine is. What evidence would you accept to change your mind?
- furyofantares 1y ago> I suspect most people envision AGI as at least having sentience I certainly don't. It could be that's necessary but I don't know of any good arguments for (or against) it.
- handfuloflight 1y agoWhy should it have discrete identity and notion of self?
- atleastoptimal 1y agoAny claim of sentience is neither provable nor falsifiable. Caring about its definition has nothing to do with capabilities.
- dataviz1000 1y agoStudent: How do I know I exist? Philosophy Professor: Who is asking? Student: I am!
- root_axis 1y agoThe suggested requirements are not engineering problems. Conceiving of a model architecture that can represent all the systems described in the blog is a monumental task of computer science research.
- crazylogger 1y agoI think the OP's point is that all those requirements are to be implemented outside the LLM layer, i.e. we don't need to conceive of any new model architecture. Even if LLMs don't progress any further beyond GPT-5 & Claude 4, we'll still get there. Take memory for example: give LLM a persistent computer and ask it to jot down its long-term memory as hierarchical directories of markdown documents. Recalling a piece of memory means a bunch of `tree` and `grep` commands. It's very, very rudimentary, but it kinda works, today. We just have to think of incrementally smarter ways to query & maintain this type of memory repo, which is a pure engineering problem.
- root_axis 1y agoThe answer can't be as simple as more sophisticated RAGs. At the end of the day, stuffing the context full of crap can only take you so far because context is an extremely limited resource. We also know that large context windows degrade in quality because the model has a harder time tracking what the user wants it to pay attention to.
- jibal 1y agoIt's software engineering.
- tom_ 1y agoToo much not-x-but-y (but with dashes) in this imo.
- dcre 1y agoThe first premise of the argument is that LLMs are plateauing in capability and this is obvious from using them. It is not obvious to me.
- netrem 1y agoConsensus on GPT-5 has been that it was underwhelming, and definitely a smaller jump than 3 to 4.
- dcre 1y agoI understand that is what a lot of people are saying. It doesn’t match my experience.
- gwern 1y agoIt is especially not obvious because this was written using ChatGPT-5. One appreciates the (deliberate?) irony, at least. (Or at least, surely if they had asymptoted, OP should've been able to write this upvoted HN article with an old GPT-4, say...)
- mdp2021 1y ago> this was written using How do you know?
- gwern 1y agoIt is lacking in URLs or references. (The systematic error in the self-reference blog post URLs is also suspicious: outdated system prompt? If nothing else, shows the human involved is sloppy when every link is broken.) The assertions are broadly cliche and truisms, and the solutions are trendy buzzwords from a year ago or more (consistent with knowledge cutoffs and emphasizing mainstream sources/opinions). The tricolon and unordered bolded triplet lists are ChatGPT. The em dashes (which you should not need to be told about at this point) and it's-not-x-but-y formulation are extremely blatant, if not 4o-level, and lacking emoji or hyperbolic language; hence, it's probably GPT-5. (Sub-GPT-5 ChatGPTs would also generally balk at talking about a 'GPT-5' because they think it doesn't exist yet.) I don't know if it was 100% GPT-5-written, but I do note that when I try the intro thesis paragraph on GPT-5-Pro, it dislikes it, and identifies several stupid assertions (eg. the claim that power law scaling has now hit 'diminishing returns', which is meaningless because all log or power laws always have diminishing returns), so probably not completely-GPT-5-written (or least, sub-Pro).
- aorloff 1y agoI keep asking it, and nobody wants to answer because it doesn't fit within the paradigm of "making AGI" What if intelligence requires agency ?
- deleted 1y ago[deleted]
- mdp2021 1y agoAgency within the world model is sufficient.
- actualwitch 1y agoThe reason people don't want to answer this question is because the value proposition from AI labs is slavery. If intelligence requires agency, they are worthless.
- aorloff 1y agoThank you
- atbpaca 1y agoIt's a research problem, a science problem. And then an engineering problem to industrialize it. How can we replicate intelligence if we don't even know how it emerges from our brains?
- mdp2021 1y agoWe do not «replicate». We implemented computing without any need of a brain-neural theory of arithmetic.
- egg1 1y agoAt its core, arithmetic is a deterministic set of rules that can be implemented with logic gates. Computing is just taking that and scaling it up a billion times. What is intelligence? How do you implement intelligence if nobody can provide a consistent, clear definition of what it is?
- mdp2021 1y agoSame thing: we create models about how to solve the problem, not biomimicry models about how natural entities solve the problem - these are not necessary. They are on a lower layer in the stack.
- egg1 1y agoExcept that doesn't make sense if you can't articulate what the problem space is. We know arithmetic inside and out, and we've understood how to make mechanical calculators centuries before the dawn of electronics. "Intelligence" on the other hand is a nebulously defined philosophical concept. What I see with these attempts at AGI is VC-funded circuses throwing shit at the wall, hardly checking to see if it sticks, and then heaping more on top. Nobody can explain how exactly transformer models are the building blocks of intelligence or how building on top of it will lead to real intelligence.
- mdp2021 1y ago
- deleted 1y ago[deleted]
- pevansgreenwood 1y ago[dead]
- bee_rider 1y agoI think the author could have picked a better title. “<X> is an engineering problem” is a pretty common expression to describe something where the science is done, but the engineering remains. There’s an understanding that that could still mean a ton of work, but there isn’t some fundamental mystery about the basic natural principles of the thing. Here, AGI is being described as an engineering problem, in contrast to a “model training” problem. That is, I think at least, he’s at least saying that more work needs to be done at an R&D level. I agree with those who are saying it is maybe not even an engineering problem yet, but should be noted that he’s at least pushing away from just running the existing programs harder, which seems to be the plan with trillions of dollars behind it.
- signa11 1y agowon’t somebody please think about Mr. Godel, and the Incompleteness Theorem ?
- jibal 1y agoThey aren't relevant. Even if Penrose and Lucas were right (they aren't), a computational system can solve the vast majority of the problems we would want solved.
- signa11 1y agoseems like a bait-n-switch argument here. we are talking explicitly about a.g.i here, not debating if the computer can solve a majority of problems or not. the two things can be true at the same time.
- jibal 1y agoP.S. I don't know which argument is allegedly "a bait and switch argument" .... certainly not mine. Even if Penrose and Lucas were right (again, they aren't--Godel's Theorems are in PA and can be proved by existing machines), that would not be a barrier to achieving AGI because being able to "see" the validity Godel's theorems is not a necessary condition for being an AGI. P.P.S. Oh, I just noticed that the response came from the same person who wrote "won’t somebody please think about Mr. Godel, and the Incompleteness Theorem ?" I will simply observe that isn't any sort of argument.
- zeofig 1y agoIt's a training data problem, and the training data you want doesn't exist.
- exabrial 1y agohttps://mashable.com/article/apple-research-ai-reasoning-models-collapse-logic-puzzles https://mashable.com/article/apple-research-ai-reasoning-mod... All of our current approaches "emulate" but do not "execute" general intelligence. The damning paper above basically concludes they're incredible pattern matching machines, but thats about it.
- ramoz 1y agoWe’ve not determined whether or not that isn’t a useful mechanism for capable intelligence. For instance it is becoming clearer that you can build harnesses for a well-trained model and teach it how to use that harness in conjunction with powerful in-context learning. I’m explicitly speaking of the Claude models and the power of whatever it is they started doing in RL. Truly excited to see where they take things and the continued momentum with tools like Claude Code (a production harness).
- jibal 1y agoThe article is about going beyond our current approaches.
- nativeit 1y agoSay we get there--all the way, full AGI, indistinguishable from conscious intelligence. Congratulations, every thing you do from that point on (very likely everything from well before that point) that is not specifically granting it free will and self-determination is slavery. Doesn't really feel like a real "win" for any plausible end goal. I'm really not clear on why anyone thinks this is a good idea, or desirable, let alone possible?
- arcanemachiner 1y agoYou can draw whatever arbitrary line you want, and say "anything on the other side of this line is slavery", but that doesn't mean that your supposition is true.
- artninja1988 1y agoI don't think agi necessarily implies consciousness, at least many definitions of it. OpenAIs definition is just ai that does most of economically viable work
- VerminOctopus1 1y agoThe forgone conclusion that LLMs are the key or even a major step towards AGI is frustrating. They are not, and we are fooling ourselves. They are incredible knowledge stores and statistical machines, but general intelligence is far more than these attributes.
- quectophoton 1y agoMy thoughts are that LLMs are like cooking a chicken by slapping it: yes, it works, but you need to reach a certain amount of kinetic energy (the same way LLMs only "start working" after reaching a certain size). So then, if we can cook a chicken like this, we can also heat a whole house like this during winters, right? We just need a chicken-slapper that's even bigger and even faster, and slap the whole house to heat it up. There's probably better analogies (because I know people will nitpick that we knew about fire way before kinetic energy), so maybe AI="flight by inventing machines with flapping wings" and AGI="space travel with machines that flap wings even faster". But the house-sized chicken-slapper illustrates how I view the current trend of trying to reach AGI by scaling up LLMs.
- jibal 1y agoRight ... as the article lays out.
- DevKoala 1y agoPlease don’t say this. Give us more time.
- bsenftner 1y agoAGI, by definition, in its name Artificial General Intelligence implies / directly states that this type of AI is not some dumb AI that requires training for all its knowledge, a general intelligence merely needs to be taught how to count, the basic rules of logic, and the basic rules of a single human language. From those basics all derivable logical human sciences will be rediscovered by that AGI and our next job is synchronizing with it our names for all the phenomenon that the AGI had to name on its own when that AGI self developed all the logical ramifications of our basics. What is that? What could merely require light elementary education and then it takes off and self improves to match and surpass us? That would be artificial comprehension, something we've not even scratched. AI and trained algorithms are "universal solvers" given enough data, This AGI would be something different, this is understanding, comprehending. Instantaneous decomposition of observations for assessment of plausibility, and then recombination for assessment of combination plausibility - all continual and instant for assessment of personal safety: all that happens in people continually while awake. Be that monitoring of personal safety be for physical or loss of client during sales negotiation. Our comprehending skills are both physical and abstract. This requires a dynamic assessment, an ongoing comprehension that is validating observations as a foundation floor, so a more forward train of thought, a "conscious mind" can make decisions without conscious thought about lower level issues like situational safety. AGI needs all that dynamic comprehending capability, to satisfy its name of being general.
- dragonwriter 1y ago> AGI, by definition, in its name Artificial General Intelligence implies / directly states that this type of AI is not some dumb AI that requires training for all its knowledge, a general intelligence merely needs to be taught how to count, the basic rules of logic, and the basic rules of a single human language. From those basics all derivable logical human sciences will be rediscovered by that AGI That's not how natural general intelligences work, though.
- bsenftner 1y agoAre you sure? Do you require dozens, to hundreds, to thousands of examples before you understand a concept? I expect no. That is because you have comprehension that can generalize a situation to basic concepts which you apply to other situations without effort. You comprehend. AI cannot do that: get the idea from a few, under a half dozen examples if necessary. Often a human needs 1-3 examples before they can generalize any concept. Not AI.
- deleted 1y ago[deleted]
- atleastoptimal 1y agoOf course someone building AI scaffolding and infrastructure tools will say that AI scaffolding and infrastructure tools are the most important. IME it’s both though. Better models, bigger models, and infrastructure all help get to AGI.
- hyfgfh 1y agoSvgs, date management, Http, so many simpler things we dont have solve and somehow people believe they will do it by putting enough money in LLMs when it cant count Why some people understood when they tried it with blockchain, nfts, web3, AR, ... any good engineer should know principle of energy efficiency instead of having faith in the Infinite monkey theorem
- atleastoptimal 1y agoLLM’s can count and the best can do mathematics at quite a high level now. Not sure why people insist that the state of AI 2-3 years ago still applies today.
- manoDev 1y agoRight now, we just did the equivalent of a tech demo of Broca's area. AGI would take making at least one full brain, and then putting many of those working together, efficiently. I don't believe we can engineer our way out of that before explaining how the f. the wetware works first.
- tolleydbg 1y agoKeep pushing it down the road until you realize it's a pipe dream problem.
- germandiago 1y agoAGI cannot possibly exist.
- jibal 1y agoProof?
- germandiago 1y agoWell, you should show the proof that it is possible also, So it would be a draw. I really think it is not possible to get that from a machine. You can improve and do much fancier than now. But AGI would be something entirely different. It is a system that can do everything better than a human including creativity, which I believe it to be exclusively human as of now. It can combine, simulate and reason. But think out of the box? I doubt so. It is different to being able to derive ideas from which human would create. For that it can be useful. But that would not be AGI.
- jibal 1y agoThe burden of proof is on the person who makes a claim, especially an absolute existential claim like that. You have failed the burden of proof and of intellectual honesty. Over and out.
- germandiago 1y agoYou make the claim that AGI can exist, right? I make the claim it cannot. Both are claims, try to not manipulate the language. I acknowledge I started this but there is not a position that is better than the other.
- jibal 1y agoP.S. > You make the claim that AGI can exist, right? No, I obviously never made any such claim ... all I said was "Proof?" > I make the claim it cannot. Which remains unsupported. > try to not manipulate the language Another false charge. > I acknowledge I started this but there is not a position that is better than the other. Again, only you made an absolute claim. Honesty is better than appalling dishonesty.
- jppope 1y agoI've said this a lot but I'm going to say it again AGI has no technical definition. One day Sam Altman, Elon Musk, or some other business guy trying to meet their obligation for next quarter will declare they have built AGI and that will be that. We'll argue and debate, but eventually it will be just another marketing term, just like AI was.
- j45 1y agoLLMs may not be the sole kind of AI to AGI. Continuing to want to make a non-deterministic system behave like a deterministic system will be interesting to watch.
- justcallmejm 1y agoI'd argue that it's because intelligence has been treated as a ML/NN engineering problem that we've had the hyper focus on improving LLMs rather than the approach you've written about. Intelligence must be built from a first principles theory of what intelligence actually is. The missing science to engineer intelligence is composable program synthesis. Aloe (https://aloe.inc https://aloe.inc) recently released a GAIA score demonstrating how CPS dramatically outperforms other generalist agents (OpenAI's deep research, Manus, and Genspark) on tasks similar to those a knowledge worker would perform.
- dsign 1y agoI have colleagues that want to plan each task of the software team for the next 12 months. They assume that such a thing is possible, or they want to do it anyway because management tells them to. The first would be an example of human fallibility, and the second would be an example of choosing the path of (perceived) least immediate self-harm after accounting for internal politics. I doubt very much we will ever build a machine that has perfect knowledge of the future or that can solve each and every “hard” reasoning problem, or that can complete each narrow task in a way we humans like. In other words, it’s not simply a matter of beating benchmarks. In my mind at least, AGI’s definition is simple: anything that can replace any human employee. That construct is not merely a knowledge and reasoning machine, but also something that has a stake on its own work and that can be inserted in a shared responsibility graph. It has to be able to tell that senior dev “I know planning all the tasks one year in advance is busy-work you don’t want to do, but if you don’t, management will terminate me. So, you better do it, or I’ll hack your email and show everybody your porn subscriptions.”
- JSR_FDED 1y agoInteresting, I hadn’t thought about it that way. But can a thing on the other end of an API call ever truly have a “stake“?
- Jensson 1y ago> But can a thing on the other end of an API call ever truly have a “stake“? That is their goal function they are trained for, it is like dopamine and sex for humans they will do anything to get it.
- JSR_FDED 1y agoYes, but a having a stake also implies feeling the loss if it goes sideways… Next you’re going to tell me that’s what loss functions are for :-)
- conartist6 1y agoAGI has a hundreds-of-millions of years of evolution problem, anything humans have done so far utterly pales in comparison. The lowliest rat has more "general" intelligence than any AI we've ever made... If we want to learn, look to nature, and it *has to be alive*.
- BenFranklin100 1y agoThis guys gets it wrong too. It’s not even an engineering problem. It’s much worse: it’s a scientific problem. We don’t yet understand how the human brain operates or what human intelligence really is. There’s nothing to engineer as the basic specifications are not yet available on what needs to be built. Will AGI require ‘consciousness’, another poorly understood concept? How are mammalian brain even wired up? The most advanced model is the Allen Institute’s Mesoscale Connectivity Atlas which is at best a low resolution static roadmap, not a dynamic description of how a brain operates in real time. And it describes a mouse brain, not a human brain which is far, far more complex, both in terms of number of parts, and architecture. People are just finally starting to acknowledge LLMs are dead ends. The effort expended on them over the last five years could well prove a costly diversion along the road to AGI, which is likely still decades in the future.
- Sevii 1y agoI don't understand how people feel comfortable writing 'LLMs are done improving, this plateau is it.' when we haven't even gone an entire calendar year without seeing improvements to LLM based AI.
- hahn-kev 1y agoI wonder if the people saying that would agree that they've been improving.
- nyonyo 1y agoAs one on that side of that argument, I have to say I have yet to see LLMs fundamentally improve, rather than being benchmaxxed on a new set of common "trick questions" and giving off the illusion of reasoning. Add an extra leg to any animal in a picture. Ask the vision LLM to tell you how many legs it sees. It will answer the same amount as a person would expect from a healthy individual, because it's not actually reasoning, it's not perceiving anything, it's pattern matching. It sees dog, it answers 4 legs. Maybe sometime in the future it won't do that, because they will add this kind of trick to their benchmaxxing set (training LLMs specifically on pictures that have less or more legs than the animal should), as they do every time there's a new generation of those illusory things. But that won't fix the fundamental that these things DO NOT REASON. Training LLMs on sets of thousands and thousands and thousands of reasoning trick questions people ask on LM arena is borderline scamming people on the true nature of this technology. If we lived in a sane regulatory environment OAI would have a lot to answer for.
- Jensson 1y agoYou can expand to cover more of a plateau, that is still an improvement but also still a plateau as you aren't going higher. So models improve on specific tasks, but they don't really improve generally across the board any longer.
- cakealert 1y agoThere is a reason why LLM's are architected the way they are and why thinking is bolted on. The architecture has to allow for gradient descent to be a viable training strategy, this means no branching (routing is bolted on). And the training data has to exist, you can't find millions of pages depicting every thought a person went through before writing something. And such data can't exist because most thoughts aren't even language. Reinforcement learning may seem like the answer here: bruteforce thinking to happen. But it's grossly sample-inefficient with gradient descent and therefore only used for finetuning. LLM's are regressive models and the configuration that was chosen where every token can only look back allows for very sample-efficient training (one sentence can be dozens of samples).
- MadnessASAP 1y agoYou didn't mention it, but LLMs and co don't have loops. Whereas a brain, even a simple one is nothing but loops. Brains don't halt, they keep spinning while new inputs come in and output whenever they feel like it. LLMs however do halt, you give them an input, it gets transformed across the layers, then gets output. While you say reinforcement learning isn't a good answer, I think its the only answer.
- deleted 1y ago[deleted]
- rrrrrrrrrrrryan 1y agoPeople have speculated that the main thing that sets the human mind apart from the minds of all other animals is its capacity for recursive thought. A handful of animals have been observed to use tools, but humans are the only species ever observed to use a tool to create another tool. This recursion created all of civilization. But that recursive thought has a limit. For example: You can think about yourself thinking. With a little effort, you can probably also think about yourself thinking about yourself thinking. But you can't go much deeper. With the advent of modern computing, we (as a species) have finally created a tool that can "think" recursively, to arbitrary levels of depth. If we ever do create a superintelligent AGI, I'd wager that its brilliance will be attributable to its ability to loop much deeper than humans can.
- starchild3001 1y agoI think this essay lands on a useful framing, even if you don’t buy its every prescription. If we zoom out, history shows two things happening in parallel: (1) brute-force scaling driving surprising leaps, and (2) system-level engineering figuring out how to harness those leaps reliably. GPUs themselves are a good analogy: Moore’s Law gave us the raw FLOPs, but CUDA, memory hierarchies, and driver stacks are what made them usable at scale. Right now, LLMs feel like they’re at the same stage as raw FLOPs; impressive, but unwieldy. You can already see the beginnings of "systems thinking" in products like Claude Code, tool-augmented agents, and memory-augmented frameworks. They’re crude, but they point toward a future where orchestration matters as much as parameter count. I don’t think the "bitter lesson" and the "engineering problem" thesis are mutually exclusive. The bitter lesson tells us that compute + general methods win out over handcrafted rules. The engineering thesis is about how to wrap those general methods in scaffolding that gives them persistence, reliability, and composability. Without that scaffolding, we’ll keep getting flashy demos that break when you push them past a few turns of reasoning. So maybe the real path forward is not "bigger vs. smarter," but bigger + engineered smarter. Scaling gives you raw capability; engineering decides whether that capability can be used in a way that looks like general intelligence instead of memoryless autocomplete.
- ComplexSystems 1y agoI would like to see what happens if some company devoted your resources to just training a model that is a total beast at math. Feed it a ridiculous amount of functional analysis and machine learning papers, and just make the best model possible for this one task. Then instead of trying to make it cheap so everyone can use it, just set it on the task of figuring out something better than the current architecture and literally have it do nothing else but that and make something based on whatever it figures out. Will it come up with something better than AdamW for optimization? Than transformers for approximating a distribution from a random sample? I don't know, but: what is the point of training any other model?
- slashdave 1y ago> Phase 3: Emergence Layer I see. So the author rejects the hypothesis of emergent behavior in LLM, but somehow thinks it will magically appear if the "engineering" is correct. Self contradictory.
- JSR_FDED 1y agoI think the only magic the author is referring to is the “magic of engineering”
- SubiculumCode 1y agoLLMs are semantic memory. Good job. Now build the other parts of cognitive systems.
- jokoon 1y agoDo ml engineers take classes on psychology, neurosciences, behavior, cognition? Because if they don't, I honestly don't think they can approach AGI. I have the feeling it's a common case of lack of humility from an entire field of science who refuses to look at other fields to understand what they're doing. Not to mention how to define intelligence in evolution, epistemology, ontology, etc. Approaching AI with a silicon valley mindset is not a good idea.
- JSR_FDED 1y ago> Approaching AI with a silicon valley mindset is not a good idea. I don’t see a problem, we’re great at just reinventing all that stuff from first principles
- cellis 1y agoIf we are truly trying to "replace human at work" as the definition of an AGI, then shouldn't the engineering goal be to componentize the human body? If we could component-by-component replace any organ with synthetic ones ( and this is already possible to some degree e.g. hearing aids, neuralinks, pacemakers, artificial hearts ) then not only could we build compute out in such a way but we could also pull humanity forward and transcend these fallible and imminently mortal structures we inhabit. Now, should we from a moral perspective is a completely different question, one I don't have an answer to.
- jayd16 1y agoFrom the moment I understood the monolithic design of my flesh, it disgusted me.
- jfim 1y agoNot necessarily. For example, early attempts to make planes tried to imitate birds with flapping wings, but the vast majority of modern planes are fixed wing aircraft. Imitating humans would be one way to do it, but it doesn't mean it's an ideal or efficient way to do it.
- ares623 1y agoI made it 2 paragraphs before realizing it’s AI generated. This isn’t just slop - it’s drivel.
- efitz 1y agoNothing that we consider intelligent works like LLMs. Brains are continuous - they don’t stop after processing one set of inputs, until a new set of inputs arrives. Brains continuously feed back on themselves. In essence they never leave training mode although physical changes like myelination optimize the brain for different stages of life. Brains have been trained by millions of generations of evolution, and we accelerate additional training during early life. LLMs are trained on much larger corpuses of information and then expected to stay static for the rest of their operational life; modulo fine tuning. Brains continuously manage context; most available input is filtered heavily by specific networks designed for preprocessing. I think that there is some merit that part of achieving AGI might involve a systems approach, but I think AGI will likely involve an architectural change to how models work.
- ivraatiems 1y agoI think I am coming to agree with the opinions of the author, at least as far as LLMs not being the key to AGI on their own. The sheer impressiveness of what they can do, and the fact they can do it with natural language, made it feel like we were incredibly close for a time. As we adjust to the technology, it starts to feel like we're further away again. But I still see all the same debates around AGI - how do we define it? what components would it require? could we get there by scaling or do we have to do more? and so on. I don't see anyone addressing the most truly fundamental question: Why would we want AGI? What need can it fulfill that humans, as generally intelligent creatures, do not already fulfill? And is that moral, or not? Is creating something like this moral? We are so far down the "asking if we could but not if we should" railroad that it's dazzling to me, and I think we ought to pull back.
- jamilton 1y agoThe dream, as I see it, is that AGI could 1, automate research/engineering, such that it would be self-improving and advance technology faster and better than would happen without AGI, improving quality of life, and 2, do a significant amount of the labor, especially physical labor via robotics, that people currently do. 2 would be significant enough in scale that it reduces the amount of labor people need to do on average without lowering quality of life. The political/economic details of that are typically handwaved. The morality of it depends on the details.
- android521 1y agobecause if people could do it , they would do it. And if you country decides you should not do it, you could be left behid. This possibility prevents any country from not doing it if they could unless they are willing to start wars with other countries for compliance (and they would still secretly do it). So should is an irrelevant quesiton.
- Arkid 1y agoIt is an approach problem. You can engineer it as much as you want but the current architectures wont get us to AGI. I have a feeling that we will end up over engineering on an approach which doesn't get us anywhere. We will make it work via guardrails and memory and context and what not but it wont get us where we want to be.
- aussieguy1234 1y agoThe memory part makes sense. If a human brains working memory were to be filled with even half the context that you see before LLMs start to fail, it too would loose focus. That's why the brain has short term working memory and long term memory for things not needed in the moment.
- robviren 1y agoI have been thinking this as well. I desperately wish to develop a method that gives the models latent thinking that actually has temporal significance. The models now are so linear and have to scale on just one pass. A recurring model where the dynamics occur over multiple passes should hold much more complexity. Have worked on a few concepts in that area that are panning out.
- solatic 1y agoCtrl-F -> emotion -> 0/0 not found in the article. Trying to model AGI off how humans think, without including emotion as a fundamental building block, is like trying to build a computer that'll run without electricity. People are emotional beings first. So much of how we learn that something is good or bad is due to emotion. In an AGI context that means: Happiness: how do I build an unguided feedback mechanism for reward? Fear: how do I build an unguided feedback mechanism to instruct to flee? Sadness: how do I build an unguided feedback mechanism to instruct to seek external support? Anger: how do I build an unguided feedback mechanism to push back on external entities that violate expectations? Disgust: how do I build an unguided feedback mechanism to instruct to avoid? Maybe building artificial emotions is an engineering problem. Maybe not. But approaches that avoid emotion entirely seem ill-advised.
- procaryote 1y agoFor all we know LLMs might be a complete dead-end when it comes to actual AGI, much like string theory was a dead-end for a theory of everything.
- EGreg 1y agoFinally. Someone gets it.
- aubanel 1y agoThe premise "LLMs have reached a plateau" is false IMO. Here are the metrics by which the author defines this plateau: "limited by their inability to maintain coherent context across sessions, their lack of persistent memory, and their stochastic nature that makes them unreliable for complex multi-step reasoning." If you try to benchmark any proxy of the points above, for instance "can models solve problems that require multi steps in agentic mode" (PlanBench, BrowseComp, I've even built custom benchmarks), the progress between models is very clear, and shows no sign of slowing down. And this does convert to real-world tasks : yesterday, I had GPT-5 build me complex react charts in one-shot, whereas previous models needed more constant supervision. I think we're moving goalposts too fast for LLMs, that's what can lead us to believe they've plateaued : but just try using past models for your current tasks (you can use use open models to be sure they were not updated) and see them struggle.
- kwakubiney 1y agoI don’t know how I feel about the assumption that model training isn’t an engineering problem
- jsnell 1y ago> GPT-5, Claude, and Gemini represent remarkable achievements, but they’re hitting asymptotes This part could do with sourcing. I think it seems clearly untrue. We only have three types of benchmark: a) ones that have been saturated, b) ones where AI performance is progressing rapidly, c) really newly introduced ones that were specifically designed for the then-current frontier models to fail on. Look at for example the METR long time horizon task benchmark, which is one that's particularly resistant to saturation. The entire article is claimed on this unsupported but probably untrue claim, but it's a bit hard to talk about when we don't have any clue about why the author thinks this is true. > The path to artificial general intelligence isn’t through training ever-larger language models Then it's a good thing that it's not the path most of the frontier labs are taking. It appears to be what xAI is doing for everything, and it was probably what GPT-4.5 was. Neither is a particularly compelling success story. But all the other progress over the last 12-18 months has come from models the same size or smaller advancing the frontier. And it has come from exactly the kind of engineering improvements that the author claims need to happen, both of the models and the scaffolding around the models. (RL on chain of thought, synthetic data, distillation, model-routing, tool use, subagents). Sorry, no, they're not exactly the same kind of engineering improvements. They're the kind of engineering improvements that the people actually creating these systems though would be useful and actually worked. We don't see the failed experiments, and we don't see the ideas that weren't well-baked enough to even experiment on.
- MichaelRazum 1y agoCounterargument. So far bigger have proven to be better in each domain of AI. Also (although hard to compare) the human brain seems at least an order of magnitude larger in the number of synapses.
- mrjin 1y agoWe don't understand how do we understand. Then, how can we be expected to create something that can understand?
- lomase 1y agoPeople who don't understand think they do.
- stuaxo 1y agoI don't buy that LLMs will get us AGI, they may be part of a system but not the whole.
- 9rx 1y agoIt boils down to whether or not we can figure out how to get LLMs to reliably write code. Something it can already do, albeit still unreliability. If we get there, the industry expectations for "AGI" will be met. The humanoid-like mind that the general public is picturing won't be met by LLMs, but that isn't the bar trying to be met.
- lazarus01 1y agoI always enjoy discussions that intersect between psychology and engineering. But I feel this person falls short immediately, because they don't study neuroscience and psychology. That is the big gap in most of these discussion. People don't discuss things close to the origin. We have to account for first principals in how intelligence works, starting from the origin of ideas and how humans process their ideas in novel ways that create amazing tech like LLM! :D How Intelligence works In Neuroscience, if you try to identify the origin of where and how thoughts are formed and how consciousness works. It is completely unknown. This brings up the argument, do humans have free will if we are driven by these thoughts of unknown origin? That's a topic for another thread. Going back to intelligence. If you study psychology and what forms intelligence, there are many human needs that drive intelligence, namely intellectual curiosity (need to know), deprivation sensitivity (need to understand), aesthetic sensitivity, absorption, flow, openness to experience. When you look at how a creative human with high intelligence uses their brain, there are 3 networks involved. Default mode network (imagination network), executive attention network and salience network. The executive attention network controls the brains computational power. It has a working memory that can complete tasks using goal directed focus. A person with high intelligence can alternate between their imagination and their working memory and pull novel ideas from their imagination and insert them into their working memory - frequently experimenting by testing reality. The salience network filters unnecessary content while we are using our working memory and imagination. How LLMs work Neural networks are quite promising in their ability to create a latent manifold within large datasets that interpolates between samples. This is the basis for generalization, where we can compress a large dataset in a lower dimensional space to a more meaningful representation that makes predictions. The advent of attention on top of neural networks, to identify important parts of text sequences, is the huge innovation powering llms today. The innovation that emulates the executive attention network. However, that alone is a long distance from the capabilities of human intelligence. With current AI systems, there is the origin, which is known vocabulary with learned weights coming from neural networks, with reinforcement learning applied to enhance the responses. Inference comes from an autoregressive sequence model that processes one token at a time. This comes with a compounding error rate with longer responses and hallucinations from counterfactuals. Correct response must be in the training distribution. As Andy Clark said, AI will never gain human intelligence, they have no motivation to interface with the world and conduct experiments and learn things on their own. I think there are too many unknown and subjective components of human intelligence and motivation that cannot be replicated with the current systems.
- jacknews 1y agoAGI is a science problem. We've lucked into these amazing abilities by just scaling. But we don't really understand how they work. And they are obviously missing a piece, some self-reflection, or continuous-loop operation perhaps, which we again don't understand. Perhaps we'll do all this engineering and luck the solution again, but I think probably not.
- satisfice 1y agoMore to the point: AGI is a testing problem. And it’s the kind of testing problem that can only be solved by hiring incompetent or corrupt testers. Since there is no possibility of an uncontroversial specification of intelligence, there is no possibility of an honest and competent tester coming to the conclusion that sufficient testing has been performed on a candidate AGI to certify it. Similarly, you won’t get any security professional to say that system is guaranteed to be secure. Moreover, you can’t get any honest psychologist to swear that any person definitely has no mental illness. How companies are dealing with this is to bet that they can fool enough people so that the remaining skeptics can be safely ignored.