5 ms·

AGI isn't going to happen within the next 30 years so this is moot. The actual researchers have said so many times. It's only the business people and laypeople

AGI isn't going to happen within the next 30 years so this is moot. The actual researchers have said so many times. It's only the business people and laypeople whooping about AGI always being imminent. You cannot get real, actual AGI (the same ability to perform tasks as a human) without a continuous cycle of learning and deep memory, which LLMs cannot do. The best LLM "memory" is a search engine and document summarizer stuffed into a context window (which is like having someone take an entire physics course, writing down everything they learn on post-it notes, then you ask a different person a physics question, and that different person has to skim all the post-it notes, and then write a new post-it note to answer you). To learn it would need RL (which requires specific novel inputs) and retraining (so that it can retain and compute answers with the learned input). This would all take too much time and careful input/engineering along with novel techniques. So AGI is too expensive, time consuming, and difficult for us to achieve without radically different designs and a whole lot more effort. Not only are LLMs not AGI, they're still not even that great at being LLMs. Sure, they can do a lot of cool things, like write working code and tests. But tell one "don't delete files in X/", and after a while, it will delete all the files in "X/", whereas a human would likely remember it's not supposed to delete some files, and go check first. It also does fun stuff like follow arbitrary instructions from an attacker found in random documents, which most humans also wouldn't do. If they had a real memory and RL in real-time, they wouldn't have these problems. But we're a long way away from that. LLMs are fine. They aren't AGI.

nerdsniper 8mo ago
> which is like having someone take an entire physics course, writing down everything they learn on post-it notes, then you ask a different person a physics question, and that different person has to skim all the post-it notes, and then write a new post-it note to answer you This is the best summary of an LLM that I’ve ever seen (for laypeople to “get it”) and is the first that accurately describes my experience. I will say, usually the notes passed to the second person are very impressive quality for the topic. But the “2nd person” still rarely has a deep understanding of it.
- trollbridge 8mo ago
  A layperson analogy I use is that an LLM is like Dora with a really high IQ - it effectively needs everything reexplained to it, and you can’t give it more than a few seconds of context before it just forgets.
  - slavik81 8mo ago
    Do you mean Dory, the fish from Finding Nemo?
    - neurocline 8mo ago
      I have to imagine the poster was referring to Dora the Explorer, a popular and charming cartoon from the start of this century.
      - tim333 8mo ago
        I think it must be Dory who has short term memory loss https://youtu.be/B6178Ac90S4?t=22 https://youtu.be/B6178Ac90S4?t=22
        trollbridge 8mo ago
        Obviously my long-term memory is not working so well either.
herodoturtle 8mo ago
Agree with your overall point. Curious what you’re basing the “not in the next 30 years” claim on, if you’d care to expand.
- tim333 8mo ago
  I was going to say similar. Fair enough that you need ongoing learning and current LLMs don't cut it but not in the next 30 years seems dubious. The hardware seems adequate so what we need is some new software ideas and who knows how long that will take?
mattlondon 8mo ago
I disagree. There is some argument to be had that they're already generally intelligent. They're already certainly better than me in basically anything I can ask them to do. So that leads to the question of what qualifies as intelligent? And do we need sentience for intelligence? What about self-agency/-actuation? Is that needed for "generally intelligent"? I don't know. But I feel like we're not there yet, even for non-sentient intelligence. I personally think we need an "unlimited" context (as good as human memory context windows anyway, which some argue we've already surpassed) and genuine self-learning before we get close. I don't think we need it to be an infallible genius (i.e ASI) to qualify as generally intelligent ... or to put it another way "about as smart and reliable as the average human adult" which frankly is quite a low bar! One thing for sure though, I think this will creep up on us and one day it will suddenly become apparent that it's already there and we just didn't appreciate/notice/comprehend. There won't be a big fireworks display the moment it happens, more of a creeping realisation I think. I give it 5 years +/-2.
- kovek 8mo ago
  Models need pre-training and fine tuning. Humans can do online learning.
  - mattlondon 8mo ago
    How much of our brain's innate wiring is "pre-training"? We're born pre-wired to breath, to swallow, to blink, sleep, cry etc. Fine tuned over many many many epochs and baked into our model weights/DNA. Is a newborn baby without learnt-knowledge not an intelligent being to you? Or is an empty vessel such as a newborn baby intelligent merely because it has the ability to learn? It gets pretty philosophical pretty quick. This is why I don't think there'll be a "moment" when AGI happens - there is so many ways to interpret what constitutes intelligence. But yeah I agree that until models can learn in real time then I think we're probably not there yet. As I said - 5 years give or take I reckon.
    - irishcoffee 8mo ago
      > Is a newborn baby without learnt-knowledge not an intelligent being to you? A newborn baby without learnt knowledge is a phenomenal comparison to an LLM: reactionary, incapable of communicating a novel thought, extremely inconsistent reactions to similar stimuli, costs a lot of money, and the best part is they almost never turn out to be an income-bearing investment. Despite all this, people vehemently defend their ugly, obnoxious, screaming, drooling babies with the fierceness of a lion, because they’re so blinded by emotion they’re incapable of logical thought. I have three kids, I’ve earned the right to say this. What a great comparison!
fwipsy 8mo ago
Given how many "fundamental" limitations of AI have been resolved within the past few years, I'm skeptical. Even if you're right, I am not sure that the limitations you identified matter all that much in practice. I think very few human engineers are working on problems which are so novel and unique that AIs cannot grasp them without additional reinforcement learning. > it will delete all the files in "X/" How many "I deleted the prod database" stories have you seen? Humans do this too. > follow arbitrary instructions from an attacker found in random documents This is just the AI equivalent of phishing - inability to distinguish authorized from unauthorized requests. Whenever people start criticizing AI, they always seem to conveniently leave out all the stupid crap humans do and compare AI against an idealized human instead.
- vor_ 8mo ago
  > How many "I deleted the prod database" stories have you seen? Humans do this too. Humans do it accidentally.
  - esafak 8mo ago
    It's going to be a sad day when AI starts messing with us deliberately.
- surgical_fire 8mo ago
  > Given how many "fundamental" limitations of AI have been resolved within the past few years Eh? Which limitations were solved?
  - monsieurbanana 8mo ago
    I can answer that one: none. The only thing I can think of is massively increased context windows (around 4k for gpt3), but a million context token with degraded performance when full is not what I'd qualify as resolved.
  - deleted 8mo ago
    [deleted]
- deleted 8mo ago
mattlondon 8mo ago
I don't think humans learn any differently than post it notes TBH We call them text books though!
- deleted 8mo ago
  [deleted]
deleted 8mo ago
[deleted]
rishabhaiover 8mo ago
I don't agree. The recent emergent behavior displayed by LLMs and test-time scaling (10x YoY revenue for Anthropic) is worth some hype. Of course, you are correct that most people who rally behind AGI do not understand the fundamental limitations of next-token prediction.
- ACCount37 8mo ago
  The "fundamental limitations" being what exactly?
  - rishabhaiover 8mo ago
    I used to think it was the quadratic complexity of attention but I guess that's not a concern anymore as they've made more hardware aware kernels of attention? The other I remember is continual learning but that may be solved in near-term future. I am not completely confident about it.
    - ACCount37 8mo ago
      Humans do have an upper limit on how much working memory they have. Which I see as the closest thing to the "O(N^2) attention curse" of LLMs. That doesn't stop an LLM from manipulating its context window to take full advantage of however much context capacity it has. Today's tools like file search and context compression are crude versions of that.
      - rishabhaiover 8mo ago
        Human brain's prediction loop is bayesian in nature.
        rishabhaiover 8mo ago
        Damn, the research moves fast. I was wrong again: https://arxiv.org/abs/2507.11768 https://arxiv.org/abs/2507.11768
stratos123 8mo ago
> AGI isn't going to happen within the next 30 years so this is moot. The actual researchers have said so many times. It's only the business people and laypeople whooping about AGI always being imminent. The statements of what "actual researchers" are you relying upon for your "next 30 years" estimate? How do you reconcile them with the sub-10- or even sub-5-years timelines of other AI researchers, like Daniel Kokotajlo[1] or Andrej Karpathy[2]? For that matter, what about polls of AI researchers, which usually obtain a median much shorter than 30 years [3]? [1] https://x.com/DKokotajlo/status/1991564542103662729 https://x.com/DKokotajlo/status/1991564542103662729 [2] https://x.com/karpathy/status/1980669343479509025 https://x.com/karpathy/status/1980669343479509025 [3] https://80000hours.org/2025/03/when-do-experts-expect-agi-to-arrive/ https://80000hours.org/2025/03/when-do-experts-expect-agi-to...
- MadxX79 8mo ago
  I'm guessing they have a lot of shares in the AI companies they work(ed) for, and they would like to pump their value so they can buy an even nicer carribean island than they can already afford?
  - dwohnitmok 8mo ago
    Kokotajlo gave up all his shares in OpenAI as part of his refusal to sign a nondisparagement agreement with OpenAI.
  - stratos123 8mo ago
    Kokotajlo in particular is notable for being the guy who quit OpenAI in 2024 in protest of their policy of requiring researchers to abide by a non-disparagement agreement to retain their equity. In the end OpenAI caved and changed their policy, but if he was lying all along to inflate the value of his shares, it would have been quite a 4d chess move of him to gamble the shares themselves on doing so.
    - MadxX79 8mo ago
      Isn't it just that he left way before gpt-5, then? At that point a sufficiently naive person could have believed that scaling was going to lead to AGI, but that sort of optimism died after he was already an outsider.
charcircuit 8mo ago
LLMs are AGI because they offer intelligence on any subject. >the same ability to perform tasks as a human The first chess AIs lost to chess grandmasters. AI does not need to be better than humans to be considered AI. >without a continuous cycle of learning and deep memory, which LLMs cannot do. But harnesses like Claude Code can with how they can store and read files along with building tools to work with them. >which is like having someone take an entire physics course, writing down everything they learn on post-it notes, then you ask a different person a physics question, and that different person has to skim all the post-it notes, and then write a new post-it note to answer you This don't matter. You could say a chess AI is a bunch of different people who work together to explore distant paths of the search space. The idea you can split things into steps does not disqualify it from being AI. >But tell one "don't delete files in X/", and after a while, it will delete all the files in "X/" Humans make mistakes and mess up things too. LLMs are better at needle in a haystack tests than humans. >It also does fun stuff like follow arbitrary instructions from an attacker A ton of people get phished or social engineered by attackers. This is the number 1 way people get hacked. Do not underestimate people's willingness to follow instructions from strangers.
deleted 8mo ago
[deleted]
mirekrusin 8mo ago
I think you're somehow right and wrong at the same. All those "it's like ..." are faulty – "post-it notes" are not 3k pages of text that can be recalled instantly in one go, copied in fraction of a second to branch off, quickly rewritten, put into hierarchy describing virtually infinite amount of information (outside of 3k pages of text limit), generated on the fly in minutes on any topic pulling all information available from computer etc. Poor man's RL on test time context (skills and friends) is something that shouldn't be discarded, we're at 1M tokens and growing and pogressive disclosure (without anything fancy, just bunch of markdowns in directories) means you can already stuff-in more information than human can remember during whole lifetime into always-on agents/swarms. Currently latest models use more compute on RL than pre-training and this upward trend continues (from orders of magnitude smaller than pre-training to larger that pre-training). In that sense some form of continous RL is already happening, it's just quantified on new model releases, not realtime. With LoRA and friends it's also already possible to do continuous training that directly affects weights, it's just that economy of it is not that great – you get much better value/cost ratio with above instead. For some definitions of AGI it already happened ie. "someboy's computer use based work" even though "it can't actually flip burgers, can it?" is true, just not relevant. ps. I should also mention that I don't believe in "programmers loosing jobs", on the contrary, we will have to ramp up on computational thinking large numbers of people and those who are already verse with it will keep reaping benefits – regardless if somebody agrees or not that AGI is already here, it arrives through computational doors speaking computational language first and imho this property will be here to stay as it's an expression of rationality etc
- 0xbadcafebee 8mo ago
  > you can already stuff-in more information than human can remember during whole lifetime The human eye processes between 100GB and 800GB of data per day. We then continuously learn and adapt from this firehose of information, using short-term and long-term memory, which is continuously retrained and weighted. This isn't "book knowledge", but the same capability is needed to continuously learn and reason on a human-equivalent level. You'd need a supercomputer to attempt it, for a single human's learning and reasoning. RL is used for SOTA models, but it's a constant game of catch-up with limited data and processing. It's like self-driving cars. How many millions of miles have they already captured? Yet they still fail at some basic driving tasks. It's because the cars can't learn or form long-term memories, much less process and act on the vast amount of data a human can in real time. Same for LLM. Training and tweaking gets you pretty far, but not matching humans. > With LoRA and friends it's also already possible to do continuous training that directly affects weights, it's just that economy of it is not that great And that means we're stuck with non-AGI. Which is fine! We could've had flying cars decades ago, but that was hard, expensive and unnecessary, so we didn't do that. There's not enough money in the global economy to "spend" our way to AGI in a short timeframe, even if we wanted to spend it all, even if we could build all the datacenters quickly enough, which we can't (despite being a huge nation, there are many limitations). > For some definitions of AGI Changing the goalposts is dangerous. A lot of scary real-world stuff is hung on the idea of AGI being here or not. People will keep getting more and more freaked out and acting out if we're not clear on what is really happening. We don't have AGI. We have useful LLMs and VLMs.
takwatanabe 8mo ago
The post-it note analogy is good, but as a psychiatrist, I'd frame it differently: LLMs are essentially patients with anterograde amnesia. They can reason brilliantly within a single conversation — just like an amnesic patient can hold an intelligent discussion — but the moment the session ends, everything is gone. No learning happened. No memory formed. What's worse, even within a session, they degrade. Research shows that effective context utilization drops to <1% of the nominal window on some tasks (Paulsen 2025). Claude 3.5 Sonnet's 200K context has an effective window of ~4K on certain benchmarks. Du et al. (EMNLP 2025) found that context length alone causes 13-85% performance degradation — even when all irrelevant tokens are removed. Length itself is the poison. This pattern is structurally identical to what I see in clinical practice every day. Anxiety fills working memory with background worry, hallucinations inject noise tokens, depressive rumination creates circular context that blocks updating. In every case, the treatment is the same: clear the context. Medication, sleep, or — for an LLM — a fresh session. The industry keeps betting on bigger context windows, but that's expanding warehouse floor space while the desk stays the same size. The human brain solved this hundreds of millions of years ago: store everything in long-term memory, recall selectively when needed, consolidate during sleep, and actively forget what's no longer useful. We can build the smartest single model in the world — the greatest genius humanity has ever seen — but a genius with no memory and no sleep is still just an amnesic savant. The ceiling isn't intelligence. It's architecture.
- torginus 8mo ago
  I want to believe I'm reading an insightful comment from an actual human deeply familiar with both human congnition and how LLMs work, but this post is chock full of LLMisms
  - takwatanabe 8mo ago
    Yeah, fair enough. I leaned on Claude to clean up my English. I normally write in Japanese. The clinical stuff is mine though, I run a psych clinic in Japan (link in profile). Should've just written it messier.
    - 8mo ago
stavros 8mo ago
> tell one "don't delete files in X/", and after a while, it will delete all the files in "X/", whereas a human would likely remember it's not supposed to delete some files, and go check first. Have you seriously never had someone to go do something you told them not to do? > It also does fun stuff like follow arbitrary instructions from an attacker found in random documents, which most humans also wouldn't do. I guess my coworker didn't actually fall for that "hey this is your CEO, please change my password" WhatsApp message then, phew. I've seen people move the goalposts on what it means for AI to be intelligent, but this is the first time I've seen someone move the goalposts on what it means for humans to be intelligent.
orbital-decay 8mo ago
>You cannot get real, actual AGI (the same ability to perform tasks as a human) without a continuous cycle of learning and deep memory, which LLMs cannot do I disagree that this prerequisite is more necessary than e.g. having legs to move over the ground. But besides that, current LLMs are literally a result of the continuous cycle of learning and deep memory. It's pretty crude compared to what evolution and human process had to do, but that's precisely how the iterative model development cycle with the hierarchical bootstrap looks like. It's not fully autonomous though (engineer-driven/humans in the loop). Moreover, the distillation process you describe is precisely what "learning" is.

nerdsniper 8mo ago

> which is like having someone take an entire physics course, writing down everything they learn on post-it notes, then you ask a different person a physics question, and that different person has to skim all the post-it notes, and then write a new post-it note to answer you This is the best summary of an LLM that I’ve ever seen (for laypeople to “get it”) and is the first that accurately describes my experience. I will say, usually the notes passed to the second person are very impressive quality for the topic. But the “2nd person” still rarely has a deep understanding of it.

trollbridge 8mo ago
A layperson analogy I use is that an LLM is like Dora with a really high IQ - it effectively needs everything reexplained to it, and you can’t give it more than a few seconds of context before it just forgets.
- slavik81 8mo ago
  Do you mean Dory, the fish from Finding Nemo?
  - neurocline 8mo ago
    I have to imagine the poster was referring to Dora the Explorer, a popular and charming cartoon from the start of this century.
    - tim333 8mo ago
      I think it must be Dory who has short term memory loss https://youtu.be/B6178Ac90S4?t=22 https://youtu.be/B6178Ac90S4?t=22
      - trollbridge 8mo ago
        Obviously my long-term memory is not working so well either.

herodoturtle 8mo ago

Agree with your overall point. Curious what you’re basing the “not in the next 30 years” claim on, if you’d care to expand.

tim333 8mo ago
I was going to say similar. Fair enough that you need ongoing learning and current LLMs don't cut it but not in the next 30 years seems dubious. The hardware seems adequate so what we need is some new software ideas and who knows how long that will take?

mattlondon 8mo ago

I disagree. There is some argument to be had that they're already generally intelligent. They're already certainly better than me in basically anything I can ask them to do. So that leads to the question of what qualifies as intelligent? And do we need sentience for intelligence? What about self-agency/-actuation? Is that needed for "generally intelligent"? I don't know. But I feel like we're not there yet, even for non-sentient intelligence. I personally think we need an "unlimited" context (as good as human memory context windows anyway, which some argue we've already surpassed) and genuine self-learning before we get close. I don't think we need it to be an infallible genius (i.e ASI) to qualify as generally intelligent ... or to put it another way "about as smart and reliable as the average human adult" which frankly is quite a low bar! One thing for sure though, I think this will creep up on us and one day it will suddenly become apparent that it's already there and we just didn't appreciate/notice/comprehend. There won't be a big fireworks display the moment it happens, more of a creeping realisation I think. I give it 5 years +/-2.

kovek 8mo ago
Models need pre-training and fine tuning. Humans can do online learning.
- mattlondon 8mo ago
  How much of our brain's innate wiring is "pre-training"? We're born pre-wired to breath, to swallow, to blink, sleep, cry etc. Fine tuned over many many many epochs and baked into our model weights/DNA. Is a newborn baby without learnt-knowledge not an intelligent being to you? Or is an empty vessel such as a newborn baby intelligent merely because it has the ability to learn? It gets pretty philosophical pretty quick. This is why I don't think there'll be a "moment" when AGI happens - there is so many ways to interpret what constitutes intelligence. But yeah I agree that until models can learn in real time then I think we're probably not there yet. As I said - 5 years give or take I reckon.
  - irishcoffee 8mo ago
    > Is a newborn baby without learnt-knowledge not an intelligent being to you? A newborn baby without learnt knowledge is a phenomenal comparison to an LLM: reactionary, incapable of communicating a novel thought, extremely inconsistent reactions to similar stimuli, costs a lot of money, and the best part is they almost never turn out to be an income-bearing investment. Despite all this, people vehemently defend their ugly, obnoxious, screaming, drooling babies with the fierceness of a lion, because they’re so blinded by emotion they’re incapable of logical thought. I have three kids, I’ve earned the right to say this. What a great comparison!

fwipsy 8mo ago

Given how many "fundamental" limitations of AI have been resolved within the past few years, I'm skeptical. Even if you're right, I am not sure that the limitations you identified matter all that much in practice. I think very few human engineers are working on problems which are so novel and unique that AIs cannot grasp them without additional reinforcement learning. > it will delete all the files in "X/" How many "I deleted the prod database" stories have you seen? Humans do this too. > follow arbitrary instructions from an attacker found in random documents This is just the AI equivalent of phishing - inability to distinguish authorized from unauthorized requests. Whenever people start criticizing AI, they always seem to conveniently leave out all the stupid crap humans do and compare AI against an idealized human instead.

vor_ 8mo ago
> How many "I deleted the prod database" stories have you seen? Humans do this too. Humans do it accidentally.
- esafak 8mo ago
  It's going to be a sad day when AI starts messing with us deliberately.
surgical_fire 8mo ago
> Given how many "fundamental" limitations of AI have been resolved within the past few years Eh? Which limitations were solved?
- monsieurbanana 8mo ago
  I can answer that one: none. The only thing I can think of is massively increased context windows (around 4k for gpt3), but a million context token with degraded performance when full is not what I'd qualify as resolved.
- deleted 8mo ago
  [deleted]
deleted 8mo ago

mattlondon 8mo ago

I don't think humans learn any differently than post it notes TBH We call them text books though!

deleted 8mo ago
[deleted]

deleted 8mo ago

[deleted]

rishabhaiover 8mo ago

I don't agree. The recent emergent behavior displayed by LLMs and test-time scaling (10x YoY revenue for Anthropic) is worth some hype. Of course, you are correct that most people who rally behind AGI do not understand the fundamental limitations of next-token prediction.

ACCount37 8mo ago
The "fundamental limitations" being what exactly?
- rishabhaiover 8mo ago
  I used to think it was the quadratic complexity of attention but I guess that's not a concern anymore as they've made more hardware aware kernels of attention? The other I remember is continual learning but that may be solved in near-term future. I am not completely confident about it.
  - ACCount37 8mo ago
    Humans do have an upper limit on how much working memory they have. Which I see as the closest thing to the "O(N^2) attention curse" of LLMs. That doesn't stop an LLM from manipulating its context window to take full advantage of however much context capacity it has. Today's tools like file search and context compression are crude versions of that.
    - rishabhaiover 8mo ago
      Human brain's prediction loop is bayesian in nature.
      - rishabhaiover 8mo ago
        Damn, the research moves fast. I was wrong again: https://arxiv.org/abs/2507.11768 https://arxiv.org/abs/2507.11768

stratos123 8mo ago

> AGI isn't going to happen within the next 30 years so this is moot. The actual researchers have said so many times. It's only the business people and laypeople whooping about AGI always being imminent. The statements of what "actual researchers" are you relying upon for your "next 30 years" estimate? How do you reconcile them with the sub-10- or even sub-5-years timelines of other AI researchers, like Daniel Kokotajlo[1] or Andrej Karpathy[2]? For that matter, what about polls of AI researchers, which usually obtain a median much shorter than 30 years [3]? [1] https://x.com/DKokotajlo/status/1991564542103662729 https://x.com/DKokotajlo/status/1991564542103662729 [2] https://x.com/karpathy/status/1980669343479509025 https://x.com/karpathy/status/1980669343479509025 [3] https://80000hours.org/2025/03/when-do-experts-expect-agi-to-arrive/ https://80000hours.org/2025/03/when-do-experts-expect-agi-to...

MadxX79 8mo ago
I'm guessing they have a lot of shares in the AI companies they work(ed) for, and they would like to pump their value so they can buy an even nicer carribean island than they can already afford?
- dwohnitmok 8mo ago
  Kokotajlo gave up all his shares in OpenAI as part of his refusal to sign a nondisparagement agreement with OpenAI.
- stratos123 8mo ago
  Kokotajlo in particular is notable for being the guy who quit OpenAI in 2024 in protest of their policy of requiring researchers to abide by a non-disparagement agreement to retain their equity. In the end OpenAI caved and changed their policy, but if he was lying all along to inflate the value of his shares, it would have been quite a 4d chess move of him to gamble the shares themselves on doing so.
  - MadxX79 8mo ago
    Isn't it just that he left way before gpt-5, then? At that point a sufficiently naive person could have believed that scaling was going to lead to AGI, but that sort of optimism died after he was already an outsider.

charcircuit 8mo ago

LLMs are AGI because they offer intelligence on any subject. >the same ability to perform tasks as a human The first chess AIs lost to chess grandmasters. AI does not need to be better than humans to be considered AI. >without a continuous cycle of learning and deep memory, which LLMs cannot do. But harnesses like Claude Code can with how they can store and read files along with building tools to work with them. >which is like having someone take an entire physics course, writing down everything they learn on post-it notes, then you ask a different person a physics question, and that different person has to skim all the post-it notes, and then write a new post-it note to answer you This don't matter. You could say a chess AI is a bunch of different people who work together to explore distant paths of the search space. The idea you can split things into steps does not disqualify it from being AI. >But tell one "don't delete files in X/", and after a while, it will delete all the files in "X/" Humans make mistakes and mess up things too. LLMs are better at needle in a haystack tests than humans. >It also does fun stuff like follow arbitrary instructions from an attacker A ton of people get phished or social engineered by attackers. This is the number 1 way people get hacked. Do not underestimate people's willingness to follow instructions from strangers.

deleted 8mo ago

[deleted]

mirekrusin 8mo ago

I think you're somehow right and wrong at the same. All those "it's like ..." are faulty – "post-it notes" are not 3k pages of text that can be recalled instantly in one go, copied in fraction of a second to branch off, quickly rewritten, put into hierarchy describing virtually infinite amount of information (outside of 3k pages of text limit), generated on the fly in minutes on any topic pulling all information available from computer etc. Poor man's RL on test time context (skills and friends) is something that shouldn't be discarded, we're at 1M tokens and growing and pogressive disclosure (without anything fancy, just bunch of markdowns in directories) means you can already stuff-in more information than human can remember during whole lifetime into always-on agents/swarms. Currently latest models use more compute on RL than pre-training and this upward trend continues (from orders of magnitude smaller than pre-training to larger that pre-training). In that sense some form of continous RL is already happening, it's just quantified on new model releases, not realtime. With LoRA and friends it's also already possible to do continuous training that directly affects weights, it's just that economy of it is not that great – you get much better value/cost ratio with above instead. For some definitions of AGI it already happened ie. "someboy's computer use based work" even though "it can't actually flip burgers, can it?" is true, just not relevant. ps. I should also mention that I don't believe in "programmers loosing jobs", on the contrary, we will have to ramp up on computational thinking large numbers of people and those who are already verse with it will keep reaping benefits – regardless if somebody agrees or not that AGI is already here, it arrives through computational doors speaking computational language first and imho this property will be here to stay as it's an expression of rationality etc

0xbadcafebee 8mo ago
> you can already stuff-in more information than human can remember during whole lifetime The human eye processes between 100GB and 800GB of data per day. We then continuously learn and adapt from this firehose of information, using short-term and long-term memory, which is continuously retrained and weighted. This isn't "book knowledge", but the same capability is needed to continuously learn and reason on a human-equivalent level. You'd need a supercomputer to attempt it, for a single human's learning and reasoning. RL is used for SOTA models, but it's a constant game of catch-up with limited data and processing. It's like self-driving cars. How many millions of miles have they already captured? Yet they still fail at some basic driving tasks. It's because the cars can't learn or form long-term memories, much less process and act on the vast amount of data a human can in real time. Same for LLM. Training and tweaking gets you pretty far, but not matching humans. > With LoRA and friends it's also already possible to do continuous training that directly affects weights, it's just that economy of it is not that great And that means we're stuck with non-AGI. Which is fine! We could've had flying cars decades ago, but that was hard, expensive and unnecessary, so we didn't do that. There's not enough money in the global economy to "spend" our way to AGI in a short timeframe, even if we wanted to spend it all, even if we could build all the datacenters quickly enough, which we can't (despite being a huge nation, there are many limitations). > For some definitions of AGI Changing the goalposts is dangerous. A lot of scary real-world stuff is hung on the idea of AGI being here or not. People will keep getting more and more freaked out and acting out if we're not clear on what is really happening. We don't have AGI. We have useful LLMs and VLMs.

takwatanabe 8mo ago

The post-it note analogy is good, but as a psychiatrist, I'd frame it differently: LLMs are essentially patients with anterograde amnesia. They can reason brilliantly within a single conversation — just like an amnesic patient can hold an intelligent discussion — but the moment the session ends, everything is gone. No learning happened. No memory formed. What's worse, even within a session, they degrade. Research shows that effective context utilization drops to <1% of the nominal window on some tasks (Paulsen 2025). Claude 3.5 Sonnet's 200K context has an effective window of ~4K on certain benchmarks. Du et al. (EMNLP 2025) found that context length alone causes 13-85% performance degradation — even when all irrelevant tokens are removed. Length itself is the poison. This pattern is structurally identical to what I see in clinical practice every day. Anxiety fills working memory with background worry, hallucinations inject noise tokens, depressive rumination creates circular context that blocks updating. In every case, the treatment is the same: clear the context. Medication, sleep, or — for an LLM — a fresh session. The industry keeps betting on bigger context windows, but that's expanding warehouse floor space while the desk stays the same size. The human brain solved this hundreds of millions of years ago: store everything in long-term memory, recall selectively when needed, consolidate during sleep, and actively forget what's no longer useful. We can build the smartest single model in the world — the greatest genius humanity has ever seen — but a genius with no memory and no sleep is still just an amnesic savant. The ceiling isn't intelligence. It's architecture.

torginus 8mo ago
I want to believe I'm reading an insightful comment from an actual human deeply familiar with both human congnition and how LLMs work, but this post is chock full of LLMisms
- takwatanabe 8mo ago
  Yeah, fair enough. I leaned on Claude to clean up my English. I normally write in Japanese. The clinical stuff is mine though, I run a psych clinic in Japan (link in profile). Should've just written it messier.
  - 8mo ago

stavros 8mo ago

> tell one "don't delete files in X/", and after a while, it will delete all the files in "X/", whereas a human would likely remember it's not supposed to delete some files, and go check first. Have you seriously never had someone to go do something you told them not to do? > It also does fun stuff like follow arbitrary instructions from an attacker found in random documents, which most humans also wouldn't do. I guess my coworker didn't actually fall for that "hey this is your CEO, please change my password" WhatsApp message then, phew. I've seen people move the goalposts on what it means for AI to be intelligent, but this is the first time I've seen someone move the goalposts on what it means for humans to be intelligent.

orbital-decay 8mo ago

>You cannot get real, actual AGI (the same ability to perform tasks as a human) without a continuous cycle of learning and deep memory, which LLMs cannot do I disagree that this prerequisite is more necessary than e.g. having legs to move over the ground. But besides that, current LLMs are literally a result of the continuous cycle of learning and deep memory. It's pretty crude compared to what evolution and human process had to do, but that's precisely how the iterative model development cycle with the hierarchical bootstrap looks like. It's not fully autonomous though (engineer-driven/humans in the loop). Moreover, the distillation process you describe is precisely what "learning" is.