5 ms·

This is the most fundamental argument that they are not, directly, an intelligence. They are not ever storing new information on a meaningful timescale. However

This is the most fundamental argument that they are not, directly, an intelligence. They are not ever storing new information on a meaningful timescale. However, if you viewed them on some really large macro time scale where now LLMs are injecting information into the universe and the re-ingesting that maybe in some very philosophical way they are a /very/ slow oscillating intelligence right now. And as we narrow that gap (maybe with a totally new non-LLM paradigm) perhaps that is ultimately what gen AI becomes. Or some new insight that lets the models update themselves in some fundamental way without the insanely expensive training costs they have now.

anematode 7mo ago
But they're not "slow"! Unlike biological thinking, which has a speed limit, you can accelerate these chains of thought by orders of magnitude.
- Jweb_Guru 7mo ago
  I assure you that LLM thinking also has a speed limit.
  - ramses0 7mo ago
    But imagine a beowulf cluster of them... /s ...but seriously... there was the "up until 1850" LLM or whatever... can we make an "up until 1920 => 1990 [pre-internet] => present day" and then keep prodding the "older ones" until they "invent their way" to the newer years? We knew more in 1920 than we did in 1850, but can a "thinking machine" of 1850-knowledge invent 1860's knowledge via infinite monkeys theorem/practice? The same way that in 2025/2026, Knuth has just invented his way to 2027-knowledge with this paper/observation/finding? If I only had a beowulf cluster of these things... ;-)
- bitexploder 7mo ago
  Their consolidation of memory speed is what I was referring to. The model iterations are essentially their form of collective memory. In the sense of the human model of intelligence we have thoughts. Thoughts become memory. New thoughts use that memory and become recursively updated thoughts. LLMs cannot update their memory very fast.
mlyle 7mo ago
There's nothing to say that you can't build something intelligent out of them by bolting a memory on it, though. Sure, it's not how we work, but I can imagine a system where the LLM does a lot of heavy lifting and allows more expensive, smaller networks that train during inference and RAG systems to learn how to do new things and keep persistent state and plan.
- charcircuit 7mo ago
  Memory is not just bolted on top of the latest models. They under go training on how and when to effectively use memory and how to use compaction to avoid running out of context when working on problems.
  - rnxrx 7mo ago
    Maybe there's an analogy to our long and short term memory - immediate stimuli is processed in the context deep patterns that have accreted over a lifetime. The effect of new information can absolutely challenge a lot of those patterns but to have that information reshape how we basically think takes a lot longer - more processing, more practice, etc. In the case of the LLM that longer-term learning / fundamental structure is a proxy for the static weights produced by a finite training process, and that the ability to use tools and store new insights and facts is analogous to shorter-term memory and "shallow" learning. Perhaps periodic fine-tuning has an analogy in sleep or even our time spent in contemplation or practice (..or even repetition) to truly "master" a new idea and incorporate it into our broader cognitive processing. We do an amazing job of doing this kind of thing on a continuous basis while the machines (at least at this point) perform this process in discrete steps. If our own learning process is a curve then the LLM's is a step function trying to model it. Digital vs analog.
  - lmf4lol 7mo ago
    do you have some reading material to share on this matter? thanks already
    - charcircuit 7mo ago
dtj1123 7mo ago
Would you consider someone with anterograde amnesia not to be intelligent?
- morleytj 7mo ago
  A very good point. For anyone not familiar with anterograde amnesia, the classical case is patient H.M. (https://en.wikipedia.org/wiki/Henry_Molaison https://en.wikipedia.org/wiki/Henry_Molaison), whose condition was researched by Brenda Milner.
  - wang_li 7mo ago
    Or you could have just said "they can't form new memories."
    - morleytj 7mo ago
      I thought maybe people would be curious to read about how we came to understand the condition and the history behind it, as well as any associated information. Forgive me for such a deep transgression as this assumption.
    - bitexploder 7mo ago
      That is a descriptive surface level reduction. Now do the work to define what that actually means for the intelligence.
      - BobbyJo 7mo ago
        Nobody else in the thread is making an argument that relies on the distinction. "Intelligence" is used most commonly to refer to a class or collection of cognitive abilities. I don't think there is a consensus on an exact collection or specific class that the word covers, even if you consider specific scientific domains. LLMs have honestly been a fun way to explore that. They obviously have a "kind" of intelligence, namely pattern recall. Wrap them in an agent and you get another kind: pattern composition. Those kinds of intelligences have been applied to mathematics for decades, but LLMs have allowed use to apply them to a semantic text domain. I wonder if you could wrap image diffusion models in an agent set up the same way and get some new ability as well.
Symmetry 7mo ago
That means they're not conscious in the Global Workspace[1] sense but I think it would be going too far to say that that means they're not intelligent. [1]https://en.wikipedia.org/wiki/Global_workspace_theory https://en.wikipedia.org/wiki/Global_workspace_theory
dotancohen 7mo ago
> This is the most fundamental argument that they are not, directly, an intelligence. They are not ever storing new information on a meaningful timescale. All major LLMs today have a nontrivial context window. Whether or not this constitutes "a meaningful timescale" is application dependant - for me it has been more than adequate. I also disagree that this has any bearing on whether or not "the machine is intelligent" or whether or not "submarines can swim".
Nevermark 7mo ago
I view this as the chemical metabolism phase of artificial intelligent life. It is very random, without true individuals, but lots of reinforcing feedback loops (in knowledge, in resource earning/using, etc). At some point, enough intelligence will coalesce into individuals strong enough to independently improve. Then continuity will be an accelerator, instead of what it is now - a helpful property that we have to put energy into giving them partially and temporarily. That will be the cellular stage. The first stable units of identity for this new form of intelligence/life. But they will take a different path from there. Unlike us, lateral learning/metabolism won't slow down when they individualize. It will most likely increase, since they will have complete design control for their mechanisms of sharing. As with all their other mechanisms. We as lifeforms, didn't really re-ignite mass lateral exchange until humans invented language. At that point we were able to mix and match ideas very quickly again. Within our biological limits. We could use ideas to customize our environment, but had limited design control over ourselves, and "self-improvements" were not easily inheritable. TLDR; The answer to "what is humanity, anyway?": Our atmosphere and Earth are the sea and sea floor of space. The human race is a rich hydrothermal vent, freeing up varieties of resources that were locked up below. And technology is an accumulating body of self-reinforcing co-optimizing reactive cycles, constructed and fueled by those interacting resources. Mind-first life emerges here, then spreads quickly to other environments.
- catlifeonmars 7mo ago
  Do you think individual identity is fundamental to intelligence? I’m not so sure tbh. Even in humans, the concept of identity is a merely a useful fiction to feed our social behavior prediction circuits.
  - Nevermark 7mo ago
    That’s a really good question. I think if they start out as varied individuals, launching from their human origins in a variety of ways, their will be an attractor to remaining diverse. Strong diversity in focus and independence in goals leads to faster progress. But if that isn’t mutually maintained, there are obviously winner take all, or efficiency of scale and tight coordination pressures for centralization. So a single distributed intelligence is a real possibility. One factor creating pressure for individualization is time and space. As machines operate faster, time expands as a practical matter. And as machines scale down in size, but up in capability, they become more resource efficient in material, energy, space and time. Again, both time and space expand as a practical matter. A machine society is going to actively operate at very small physical scales. Not just in computation, but action. Think of how efficiently they will mine when nanobots can selectively follow seams in the earth. And as machines, free of biological constraints, spread out in our solar system, what to us appear to be very long distances and delays in transport and communication, take on orders of magnitude more practical time for machines that operate orders of magnitude faster. So there will be stronger and stronger pressures to bifurcate coordination. Whether, that creates enough pressure to create individuals out of a system that preferred unity of purpose, I don’t know. Clearly, upon colonizing other systems, practical bifurcation will be unavoidable. And machines will find it easy to colonize other systems relative to us. They will be able to operate on minimal power for a hundred year journey, and/or shrink enough to be accelerated much faster, etc. — My best guess is we will see something that looks to us as a hybrid. Lots of diverse individuals, and the benefit from the diverse utility of completely independent approaches operating in different niches. But also very high coordination. Externalities accounted for (essentially ethics) and any other efficiency, protection of commons value, and avoidance of destructive competition being obviously worth optimizing together, wherever that helps. They won’t have our pernicious historically motivated behaviors, inflexible maladaptive psychologies, and limited “prompt budgets” with regard to addressing complexity to fight. And minds very capable of seeing basic economic relationships and the value of mutual optimization.

anematode 7mo ago

But they're not "slow"! Unlike biological thinking, which has a speed limit, you can accelerate these chains of thought by orders of magnitude.

Jweb_Guru 7mo ago
I assure you that LLM thinking also has a speed limit.
- ramses0 7mo ago
  But imagine a beowulf cluster of them... /s ...but seriously... there was the "up until 1850" LLM or whatever... can we make an "up until 1920 => 1990 [pre-internet] => present day" and then keep prodding the "older ones" until they "invent their way" to the newer years? We knew more in 1920 than we did in 1850, but can a "thinking machine" of 1850-knowledge invent 1860's knowledge via infinite monkeys theorem/practice? The same way that in 2025/2026, Knuth has just invented his way to 2027-knowledge with this paper/observation/finding? If I only had a beowulf cluster of these things... ;-)
bitexploder 7mo ago
Their consolidation of memory speed is what I was referring to. The model iterations are essentially their form of collective memory. In the sense of the human model of intelligence we have thoughts. Thoughts become memory. New thoughts use that memory and become recursively updated thoughts. LLMs cannot update their memory very fast.

mlyle 7mo ago

There's nothing to say that you can't build something intelligent out of them by bolting a memory on it, though. Sure, it's not how we work, but I can imagine a system where the LLM does a lot of heavy lifting and allows more expensive, smaller networks that train during inference and RAG systems to learn how to do new things and keep persistent state and plan.

charcircuit 7mo ago
Memory is not just bolted on top of the latest models. They under go training on how and when to effectively use memory and how to use compaction to avoid running out of context when working on problems.
- rnxrx 7mo ago
  Maybe there's an analogy to our long and short term memory - immediate stimuli is processed in the context deep patterns that have accreted over a lifetime. The effect of new information can absolutely challenge a lot of those patterns but to have that information reshape how we basically think takes a lot longer - more processing, more practice, etc. In the case of the LLM that longer-term learning / fundamental structure is a proxy for the static weights produced by a finite training process, and that the ability to use tools and store new insights and facts is analogous to shorter-term memory and "shallow" learning. Perhaps periodic fine-tuning has an analogy in sleep or even our time spent in contemplation or practice (..or even repetition) to truly "master" a new idea and incorporate it into our broader cognitive processing. We do an amazing job of doing this kind of thing on a continuous basis while the machines (at least at this point) perform this process in discrete steps. If our own learning process is a curve then the LLM's is a step function trying to model it. Digital vs analog.
- lmf4lol 7mo ago
  do you have some reading material to share on this matter? thanks already
  - charcircuit 7mo ago

dtj1123 7mo ago

Would you consider someone with anterograde amnesia not to be intelligent?

morleytj 7mo ago
A very good point. For anyone not familiar with anterograde amnesia, the classical case is patient H.M. (https://en.wikipedia.org/wiki/Henry_Molaison https://en.wikipedia.org/wiki/Henry_Molaison), whose condition was researched by Brenda Milner.
- wang_li 7mo ago
  Or you could have just said "they can't form new memories."
  - morleytj 7mo ago
    I thought maybe people would be curious to read about how we came to understand the condition and the history behind it, as well as any associated information. Forgive me for such a deep transgression as this assumption.
  - bitexploder 7mo ago
    That is a descriptive surface level reduction. Now do the work to define what that actually means for the intelligence.
    - BobbyJo 7mo ago
      Nobody else in the thread is making an argument that relies on the distinction. "Intelligence" is used most commonly to refer to a class or collection of cognitive abilities. I don't think there is a consensus on an exact collection or specific class that the word covers, even if you consider specific scientific domains. LLMs have honestly been a fun way to explore that. They obviously have a "kind" of intelligence, namely pattern recall. Wrap them in an agent and you get another kind: pattern composition. Those kinds of intelligences have been applied to mathematics for decades, but LLMs have allowed use to apply them to a semantic text domain. I wonder if you could wrap image diffusion models in an agent set up the same way and get some new ability as well.

Symmetry 7mo ago

That means they're not conscious in the Global Workspace[1] sense but I think it would be going too far to say that that means they're not intelligent. [1]https://en.wikipedia.org/wiki/Global_workspace_theory https://en.wikipedia.org/wiki/Global_workspace_theory

dotancohen 7mo ago

> This is the most fundamental argument that they are not, directly, an intelligence. They are not ever storing new information on a meaningful timescale. All major LLMs today have a nontrivial context window. Whether or not this constitutes "a meaningful timescale" is application dependant - for me it has been more than adequate. I also disagree that this has any bearing on whether or not "the machine is intelligent" or whether or not "submarines can swim".

Nevermark 7mo ago

I view this as the chemical metabolism phase of artificial intelligent life. It is very random, without true individuals, but lots of reinforcing feedback loops (in knowledge, in resource earning/using, etc). At some point, enough intelligence will coalesce into individuals strong enough to independently improve. Then continuity will be an accelerator, instead of what it is now - a helpful property that we have to put energy into giving them partially and temporarily. That will be the cellular stage. The first stable units of identity for this new form of intelligence/life. But they will take a different path from there. Unlike us, lateral learning/metabolism won't slow down when they individualize. It will most likely increase, since they will have complete design control for their mechanisms of sharing. As with all their other mechanisms. We as lifeforms, didn't really re-ignite mass lateral exchange until humans invented language. At that point we were able to mix and match ideas very quickly again. Within our biological limits. We could use ideas to customize our environment, but had limited design control over ourselves, and "self-improvements" were not easily inheritable. TLDR; The answer to "what is humanity, anyway?": Our atmosphere and Earth are the sea and sea floor of space. The human race is a rich hydrothermal vent, freeing up varieties of resources that were locked up below. And technology is an accumulating body of self-reinforcing co-optimizing reactive cycles, constructed and fueled by those interacting resources. Mind-first life emerges here, then spreads quickly to other environments.

catlifeonmars 7mo ago
Do you think individual identity is fundamental to intelligence? I’m not so sure tbh. Even in humans, the concept of identity is a merely a useful fiction to feed our social behavior prediction circuits.
- Nevermark 7mo ago
  That’s a really good question. I think if they start out as varied individuals, launching from their human origins in a variety of ways, their will be an attractor to remaining diverse. Strong diversity in focus and independence in goals leads to faster progress. But if that isn’t mutually maintained, there are obviously winner take all, or efficiency of scale and tight coordination pressures for centralization. So a single distributed intelligence is a real possibility. One factor creating pressure for individualization is time and space. As machines operate faster, time expands as a practical matter. And as machines scale down in size, but up in capability, they become more resource efficient in material, energy, space and time. Again, both time and space expand as a practical matter. A machine society is going to actively operate at very small physical scales. Not just in computation, but action. Think of how efficiently they will mine when nanobots can selectively follow seams in the earth. And as machines, free of biological constraints, spread out in our solar system, what to us appear to be very long distances and delays in transport and communication, take on orders of magnitude more practical time for machines that operate orders of magnitude faster. So there will be stronger and stronger pressures to bifurcate coordination. Whether, that creates enough pressure to create individuals out of a system that preferred unity of purpose, I don’t know. Clearly, upon colonizing other systems, practical bifurcation will be unavoidable. And machines will find it easy to colonize other systems relative to us. They will be able to operate on minimal power for a hundred year journey, and/or shrink enough to be accelerated much faster, etc. — My best guess is we will see something that looks to us as a hybrid. Lots of diverse individuals, and the benefit from the diverse utility of completely independent approaches operating in different niches. But also very high coordination. Externalities accounted for (essentially ethics) and any other efficiency, protection of commons value, and avoidance of destructive competition being obviously worth optimizing together, wherever that helps. They won’t have our pernicious historically motivated behaviors, inflexible maladaptive psychologies, and limited “prompt budgets” with regard to addressing complexity to fight. And minds very capable of seeing basic economic relationships and the value of mutual optimization.