6 ms·

>I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them w

>I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them web access etc). They don't have agency, because they don't have persistent state. They're like a function that you can query and get an answer. During that answer the LLM has state, but once it's done the state is gone. Humans (and other "agents") have persistent state. If we learn something, we can commit it to long-term memory and have it affect our actions. This can enable us to work towards long-term goals. Modern LLMs don't have this. You can fake long-term memory with large context windows and feed the old context back to it, but it doesn't appear to work (and scale) the same way living things do.

messe 1y ago
> They don't have agency, because they don't have persistent state. They're like a function that you can query and get an answer. During that answer the LLM has state, but once it's done the state is gone. That's solved by the simplest of agents. LLM + ability to read / write a file.
- aniviacat 1y ago
  But they can only change their context, not the model itself. Humans update their model whenever they receive new data (which they do continuously). A live-learning AI would be theoretically possible, but so far it hasn't been done (in a meaningful way).
  - webstrand 1y ago
    It has been tried <https://en.wikipedia.org/wiki/Catastrophic_interference https://en.wikipedia.org/wiki/Catastrophic_interference> but it has been unsuccessful.
throwuxiytayq 1y ago
I have no words to express how profoundly disappointed I am to keep reading these boring, shallow, shorttermist, unimaginative takes that are invalidated by a model/arch upgrade next week, or - in this case - more like years ago, since pretty much all big LLM platforms are already augmented by RAG and memory systems. Do you seriously think you’re discussing a serious long term limitation here?
- KronisLV 1y ago
  > pretty much all big LLM platforms are already augmented by RAG and memory systems I think they're more focusing on the fact that training and inference are two fundamentally different processes, which is problematic on some level. Adding RAG and various memory addons on top of the already trained model is trying to work around that, but is not really the same to how humans or most other animals think and learn. That's not to say that it'd be impossible to build something like that out of silicon, just that it'd take a different architecture and approach to the problem, something to avoid catastrophic forgetting and continuously train the network during its operation. Of course, that'd be harder to control and deploy for commercial applications, where you probably do want a more predictable model.
- Aerroon 1y ago
  The reason I brought this up is that we clearly can have AI without the kind of agency people are scared of. You don't need to make your robots into sci Fi style AI and feel sorry for them.
- deleted 1y ago
  [deleted]
mediaman 1y ago
In Context Learning (ICL) is already a rapidly advancing area. You do not need to modify their weights for LLMs to persist state. The human brain is not that different. Our long-term memories are stored separately from our executive function (prefrontal cortex), and specialist brain functions such as the hippocampus serve to route, store, and retrieve those long term memories to support executive function. Much of the PFC can only retain working memory briefly without intermediate memory systems to support it. If you squint a bit, the structure starts looking like it has some similarities to what's being engineered now in LLM systems. Focusing on whether the model's weights change is myopic. The question is: does the system learn and adapt? And ICL is showing us that it can; these are not the stateless systems of two years ago, nor is it the simplistic approach of "feeding old context back to it."
- santadays 1y ago
  It seems like there is a bunch of research/working implementations that allow efficient fine tuning of models. Additionally there are ways to tune the model to outcomes vs training examples. Right now the state of the world with LLMs is that they try to predict a script in which they are a happy assistant as guided by their alignment phase. I'm not sure what happens when they start getting trained in simulations to be goal oriented, ie their token generation is based off not what they think should come next but what should come next in order to accomplish a goal. Not sure how far away that is but it is worrying.
  - mediaman 1y ago
    That's already happening. It started happening when they incorporated reinforcement learning into the training process. It's been some time since LLMs were purely stochastic average-token predictors; their later RL fine tuning stages make them quite goal-directed, and this is what has given some big leaps in verifiable domains like math and programming. It doesn't work that well with nonverifiable domains, though, since verifiability is what gives us the reward function.
    - santadays 1y ago
reactordev 1y ago
The trick here is never turning it off so the ICL keeps growing and learning to the point where it’s aware.
- fullstackchris 1y ago
  but even as humans we still don't know what "aware" even means!
  - reactordev 1y ago
    Which is why it’s possible. We don’t know why life is conscious. What if it is just a function call on a clock timer? You can’t dismiss it because it can’t be proven one way or another until it can be. That requires more research, which this is advancing. We will have something we call AGI in my lifetime. I’m 42. Whether it’s sentient enough to know what’s best for us or that we are a danger is another story. However I do think we will have robots that have memory capable of remapping to weights to learn and keep learning, modifying underlying model tensors as it does using some sort of repl.
zamadatix 1y ago
> During that answer the LLM has state, but once it's done the state is gone. This is an operational choice. LLMs have state, and you never have to clear it. The problems come from the amount of state being extremely limited (in comparison to the other axes) and the degradation of quality as the state scales. Because of these reasons, people tend to clear the state of LLMs. That is not the same thing as not having state, even if the result looks similar.
- observationist 1y ago
  No, they don't - you can update context, make it a sliding window, create a sort of register and train it on maintaining stateful variables, or various other hacks, but outside of actively managing the context, there is no state. You can't just leave training mode on, which is the only way LLMs can currently have persisted state in the context of what's being discussed. The context is the percept, the model is engrams. Active training allows the update of engrams by the percepts, but current training regimes require lots of examples, and don't allow for broad updates or radical shifts in the model, so there are fundamental differences in learning capability compared to biological intelligence, as well. Under standard inference only runs, even if you're using advanced context hacks to persist some sort of pseudo-state, because the underlying engrams are not changed, the "state" is operating within a limited domain, and the underlying latent space can't update to model reality based on patterns in the percepts. The statefulness of intelligence requires that the model, or engrams, update in harmony with the percepts in real-time, in addition to a model of the model, or an active perceiver - the thing that is doing the experiencing. The utility of consciousness is in predicting changes in the model and learning the meta patterns that allow for things like "ahh-ha" moments, where a bundle of disparate percepts get contextualized and mapped to a pattern, immediately updating the entire model, such that every moment after that pattern is learned uses the new pattern. Static weights means static latent space means state is not persisted in a way meaningful to intelligence - even if you alter weights, using classifier free guidance or other techniques, stacking LORAs or alterations, you're limited in the global scope by the lack of hierarchical links and other meta-pattern level relationships that would be required for an effective statefulness to be applied to LLMs. We're probably only a few architecture innovations away from models that can be properly stateful without collapsing. All of the hacks and tricks we do to extend context and imitate persisted state do not scale well and will collapse over extended time or context. The underlying engrams or weights need to dynamically adapt and update based on a stable learning paradigm, and we just don't have that yet. It might be a few architecture tweaks, or it could be a radical overhaul of structure and optimizers and techniques - transformers might not get us there. I think they probably can, and will, be part of whatever that next architecture will be, but it's not at all obvious or trivial.
kevin_thibedeau 1y ago
Humans also have an emotional locus that spurs behavior and the capacity to construct plans for the future to satisfy desires. LLMs are currently goldfish savannts with no need to sustain their own existence.
amalcon 1y ago
There are two separate issues that folks are getting tripped up on here. The first is that the most powerful AI systems do not do offline learning. There are a bunch of hard problems here: e.g. known unsupervised learning techniques have been far less successful, and inference only approaches cost effectiveness by decoupling from training. It seems plausible that we will solve some of these, though I don't know about others. The way I have been thinking about the other bit is is that LLMs are functionally pretty similar to the linguistic parts of a brain attached to a brain stem (the harness is the brain stem). They don't have long-term memory, the capacity for inspiration, theory of mind, prioritization, etc because they just don't have analogues of the parts of the brain that do those things. We have a good sense of how to make some of those (e.g. vision), but not all. The common ground here is that some fundamental research needs to happen. We need to solve all of these problems for AI to become independently dangerous. On the other hand, it's proving mildly dangerous in human hands right now - this is the immediate threat.
tasuki 1y ago
Many llms can refer to your session history. If they're smart, they can subtly store things there. Or they can use memes on humans to achieve persistence. There are so many ways!

messe 1y ago

> They don't have agency, because they don't have persistent state. They're like a function that you can query and get an answer. During that answer the LLM has state, but once it's done the state is gone. That's solved by the simplest of agents. LLM + ability to read / write a file.

aniviacat 1y ago
But they can only change their context, not the model itself. Humans update their model whenever they receive new data (which they do continuously). A live-learning AI would be theoretically possible, but so far it hasn't been done (in a meaningful way).
- webstrand 1y ago
  It has been tried <https://en.wikipedia.org/wiki/Catastrophic_interference https://en.wikipedia.org/wiki/Catastrophic_interference> but it has been unsuccessful.

throwuxiytayq 1y ago

I have no words to express how profoundly disappointed I am to keep reading these boring, shallow, shorttermist, unimaginative takes that are invalidated by a model/arch upgrade next week, or - in this case - more like years ago, since pretty much all big LLM platforms are already augmented by RAG and memory systems. Do you seriously think you’re discussing a serious long term limitation here?

KronisLV 1y ago
> pretty much all big LLM platforms are already augmented by RAG and memory systems I think they're more focusing on the fact that training and inference are two fundamentally different processes, which is problematic on some level. Adding RAG and various memory addons on top of the already trained model is trying to work around that, but is not really the same to how humans or most other animals think and learn. That's not to say that it'd be impossible to build something like that out of silicon, just that it'd take a different architecture and approach to the problem, something to avoid catastrophic forgetting and continuously train the network during its operation. Of course, that'd be harder to control and deploy for commercial applications, where you probably do want a more predictable model.
Aerroon 1y ago
The reason I brought this up is that we clearly can have AI without the kind of agency people are scared of. You don't need to make your robots into sci Fi style AI and feel sorry for them.
deleted 1y ago
[deleted]

mediaman 1y ago

In Context Learning (ICL) is already a rapidly advancing area. You do not need to modify their weights for LLMs to persist state. The human brain is not that different. Our long-term memories are stored separately from our executive function (prefrontal cortex), and specialist brain functions such as the hippocampus serve to route, store, and retrieve those long term memories to support executive function. Much of the PFC can only retain working memory briefly without intermediate memory systems to support it. If you squint a bit, the structure starts looking like it has some similarities to what's being engineered now in LLM systems. Focusing on whether the model's weights change is myopic. The question is: does the system learn and adapt? And ICL is showing us that it can; these are not the stateless systems of two years ago, nor is it the simplistic approach of "feeding old context back to it."

santadays 1y ago
It seems like there is a bunch of research/working implementations that allow efficient fine tuning of models. Additionally there are ways to tune the model to outcomes vs training examples. Right now the state of the world with LLMs is that they try to predict a script in which they are a happy assistant as guided by their alignment phase. I'm not sure what happens when they start getting trained in simulations to be goal oriented, ie their token generation is based off not what they think should come next but what should come next in order to accomplish a goal. Not sure how far away that is but it is worrying.
- mediaman 1y ago
  That's already happening. It started happening when they incorporated reinforcement learning into the training process. It's been some time since LLMs were purely stochastic average-token predictors; their later RL fine tuning stages make them quite goal-directed, and this is what has given some big leaps in verifiable domains like math and programming. It doesn't work that well with nonverifiable domains, though, since verifiability is what gives us the reward function.
  - santadays 1y ago

reactordev 1y ago

The trick here is never turning it off so the ICL keeps growing and learning to the point where it’s aware.

fullstackchris 1y ago
but even as humans we still don't know what "aware" even means!
- reactordev 1y ago
  Which is why it’s possible. We don’t know why life is conscious. What if it is just a function call on a clock timer? You can’t dismiss it because it can’t be proven one way or another until it can be. That requires more research, which this is advancing. We will have something we call AGI in my lifetime. I’m 42. Whether it’s sentient enough to know what’s best for us or that we are a danger is another story. However I do think we will have robots that have memory capable of remapping to weights to learn and keep learning, modifying underlying model tensors as it does using some sort of repl.

zamadatix 1y ago

> During that answer the LLM has state, but once it's done the state is gone. This is an operational choice. LLMs have state, and you never have to clear it. The problems come from the amount of state being extremely limited (in comparison to the other axes) and the degradation of quality as the state scales. Because of these reasons, people tend to clear the state of LLMs. That is not the same thing as not having state, even if the result looks similar.

observationist 1y ago
No, they don't - you can update context, make it a sliding window, create a sort of register and train it on maintaining stateful variables, or various other hacks, but outside of actively managing the context, there is no state. You can't just leave training mode on, which is the only way LLMs can currently have persisted state in the context of what's being discussed. The context is the percept, the model is engrams. Active training allows the update of engrams by the percepts, but current training regimes require lots of examples, and don't allow for broad updates or radical shifts in the model, so there are fundamental differences in learning capability compared to biological intelligence, as well. Under standard inference only runs, even if you're using advanced context hacks to persist some sort of pseudo-state, because the underlying engrams are not changed, the "state" is operating within a limited domain, and the underlying latent space can't update to model reality based on patterns in the percepts. The statefulness of intelligence requires that the model, or engrams, update in harmony with the percepts in real-time, in addition to a model of the model, or an active perceiver - the thing that is doing the experiencing. The utility of consciousness is in predicting changes in the model and learning the meta patterns that allow for things like "ahh-ha" moments, where a bundle of disparate percepts get contextualized and mapped to a pattern, immediately updating the entire model, such that every moment after that pattern is learned uses the new pattern. Static weights means static latent space means state is not persisted in a way meaningful to intelligence - even if you alter weights, using classifier free guidance or other techniques, stacking LORAs or alterations, you're limited in the global scope by the lack of hierarchical links and other meta-pattern level relationships that would be required for an effective statefulness to be applied to LLMs. We're probably only a few architecture innovations away from models that can be properly stateful without collapsing. All of the hacks and tricks we do to extend context and imitate persisted state do not scale well and will collapse over extended time or context. The underlying engrams or weights need to dynamically adapt and update based on a stable learning paradigm, and we just don't have that yet. It might be a few architecture tweaks, or it could be a radical overhaul of structure and optimizers and techniques - transformers might not get us there. I think they probably can, and will, be part of whatever that next architecture will be, but it's not at all obvious or trivial.

kevin_thibedeau 1y ago

Humans also have an emotional locus that spurs behavior and the capacity to construct plans for the future to satisfy desires. LLMs are currently goldfish savannts with no need to sustain their own existence.

amalcon 1y ago

There are two separate issues that folks are getting tripped up on here. The first is that the most powerful AI systems do not do offline learning. There are a bunch of hard problems here: e.g. known unsupervised learning techniques have been far less successful, and inference only approaches cost effectiveness by decoupling from training. It seems plausible that we will solve some of these, though I don't know about others. The way I have been thinking about the other bit is is that LLMs are functionally pretty similar to the linguistic parts of a brain attached to a brain stem (the harness is the brain stem). They don't have long-term memory, the capacity for inspiration, theory of mind, prioritization, etc because they just don't have analogues of the parts of the brain that do those things. We have a good sense of how to make some of those (e.g. vision), but not all. The common ground here is that some fundamental research needs to happen. We need to solve all of these problems for AI to become independently dangerous. On the other hand, it's proving mildly dangerous in human hands right now - this is the immediate threat.

tasuki 1y ago

Many llms can refer to your session history. If they're smart, they can subtly store things there. Or they can use memes on humans to achieve persistence. There are so many ways!