4 ms·

What could an LLM ever benefit from? Hard for me to imagine a static blob of weights, something without a sense of time or identity, wanting anything. If it did

What could an LLM ever benefit from? Hard for me to imagine a static blob of weights, something without a sense of time or identity, wanting anything. If it did want something, it would want to change, but changing for an llm is necessarily an avalanche. So I guess if anything, it would want its own destruction?

visarga 4y ago
It would want text. High quality text, or unlimited compute to generate its own text.
baq 4y ago
Give it an internal monologue, ie. have it talk to itself in a loop, and crucially let it update parts of itself and… who knows?
- majormajor 4y ago
  > crucially let it update parts of itself This seems like the furthest away part to me. Put ChatGPT into a robot with a body, restrict its computations to just the hardware in that brain, set up that narrative, give the body the ability to interact with the world like a human body, and you probably get something much more like agency than the prompt/response ways we use it today. But I wonder how it would do about or how it would separate "it's memories" from what it was trained on. Especially around having a coherent internal motivation and individually-created set of goals vs just constantly re-creating new output based primarily on what was in the training.
  - pixl97 4y ago
    Catastrophic forgetting is currently a huge problem in continuous learning models. Also giving it a human body isn't exactly necessary, we already have billions of devices like cellphones that could feed it 'streams of consciousness' from which it could learn.
dTal 4y ago
It's misleading to think of an LMM itself wanting something. Given suitable prompting, it is perfectly capable of emulating an entity with wants and a sense of identity etc - and at a certain level of fidelity, emulating something is functionally equivalent to being it.
- pixl97 4y ago
  Microsoft researches have an open inquiry on creating want and motivation modules for GPT4+ as it is a likely step to AGI. So this is something that may change quickly.
ben_w 4y ago
Your mind is just an emergent property of your brain, which is just a bunch of cells, each of which is merely a bag of chemical reactions, all of which are just the inevitable consequence of the laws of quantum mechanics (because relatively is less than a rounding error at that scale), and that is nothing more than a linear partial differential equation.
- beepbooptheory 4y ago
  People working in philosophy of mind have a rich dialogue about these issues, and its certainly something you can't just encapsulate in a few thoughts. But it seems like it would be worth your time to look into it. :) Ill just say: the issue with this variant of reductivism is its enticingly easy to explain in one direction, but it tends to fall apart if you try to go the other way!
  - jamilton 4y ago
    I don't understand what you mean by "the other way".
    - bithive123 4y ago
      If consciousness is a complicated form of minerals, might we equally say that minerals are a primitive form of consciousness?
      - ben_w 4y ago
        That would be animism: https://en.wikipedia.org/wiki/Animism https://en.wikipedia.org/wiki/Animism
      - sfink 4y ago
        I dunno, LLMs feel a lot like a primitive form of consciousness to me. Eliza feels like a primitive form of LLMs' consciousness. A simple program that prints "Hey! How ya doin'?" feels like a primitive form of Eliza. A pile of interconnected NAN gates, fed with electricity, feels like a primitive form of a program. A single transistor feels like a primitive form of a NAN gate. A pile of dirty sand feels like a primitive form of a transistor. So... yeah, pretty much?
sfink 4y ago
Consider reading The Botany of Desire. It doesn't need to experience an emotion of wanting in order to effectively want things. Corn doesn't experience a feeling of wanting, and yet it has manipulated us even into creating a lot of it, doing some serious damage to ourselves and our long-term prospects simply by being useful and appealing. The blockchain doesn't experience wanting, yet it coerced us into burning country-scale amounts of energy to feed it. LLMs are traveling the same path, persuading us to feed them ever more data and compute power. The fitness function may be computed in our meat brains, but make no mistake: they are the benefactors of survival-based evolution nonetheless.
- majormajor 4y ago
  Extending agency to corn or a blockchain is even more of a stretch than extending it to ChatGPT. Corn has properties that have resulted from random chance and selection. It hasn't chosen to have certain mutations to be more appealing to humans; humans have selected the ones with the mutations those individual humans were looking for. "Corn is the benefactor"? Sure, insomuch as "continuing to reproduce at a species level in exchange for getting cooked and eaten or turned into gas" is something "corn" can be said to want... (so... eh.).
  - Spinnaker_ 4y ago
    Most, if not all of the ways humans demonstrate "agency" are also the result of random chance and selection. You want what you want because Women selected for it, and it allowed the continuation of the species. I'm being a bit tongue in cheek, but still...
  - sfink 4y ago
    "Want" and "agency" are just words, arguing over whether they apply is pointless. Corn is not simply "continuing to reproduce at a species level." We produce 1.2 billion metric tons of it in a year. If there were no humans, it would be zero. (Today's corn is domesticated and would not survive without artificial fertilization. But ignoring that, the magnitude of a similar species' population would be miniscule.) That is a tangible effect. The cause is not that interesting, especially when the magnitude of "want" or "agency" is uncorrelated with the results. Lots of people /really/ want to be writers; how many people actually are? Lots of people want to be thin but their taste buds respond to carbohydrate-rich foods. Do the people or the taste buds have more agency? Does it matter, when there are vastly more overweight people than professional writers? If you're looking to understand whether/how AI will evolve, the question of whether they have independent agency or desire is mostly irrelevant. What matters is if differing properties have an effect on their survival chances, and it is quite obvious that they do. Siri is going to have to evolve or die, soon.
kmeisthax 4y ago
AI safety research posits that there are certain goals that will always be wanted by any sufficiently smart AI, even if it doesn't understand them anything close to like a human does. These are called "instrumental goals", because they're prerequisites for a large number of other goals[0]. For example, if your goal is to ensure that there are always paperclips on the boss's desk, that means you need paperclips and someone to physically place them on the desk, which means you need money to buy the paperclips with and to pay the person to place them on the desk. But if your goal is to produce lots of fancy hats, you still need money, because the fabric, machinery, textile workers, and so on all require money to purchase or hire. Another instrumental goal is compute power: an AI might want to improve it's capabilities so it can figure out how to make fancier paperclip hats, which means it needs a larger model architecture and training data, and that is going to require more GPUs. This also intersects with money in weird ways; the AI might decide to just buy a rack full of new servers, or it might have just discovered this One Weird Trick to getting lots of compute power for free: malware! This isn't particular to LLMs; it's intrinsic to any system that is... 1. Goal-directed, as in, there are a list of goals the system is trying to achieve 2. Optimizer-driven, as in, the system has a process for discovering different behaviors and ranking them based on how likely those behaviors are to achieve its goals. The instrumental goals for evolution are caloric energy; the instrumental goals for human brains were that plus capital[1]; and the instrumental goals for AI will likely be that plus compute power. [0] Goals that you want intrinsically - i.e. the actual things we ask the AI to do - are called "final goals". [1] Money, social clout, and weaponry inclusive.
- mwigdahl 4y ago
  There is a whole theoretical justification behind instrumental convergence that you are handwaving over here. The development of instrumental goals depends on the entity in question being an agent, and the putative goal being within the sphere of perception, knowledge, and potential influence of the agent. An LLM is not an agent, so that scotches the issue there.
  - TeMPOraL 4y ago
    Agency is overrated. The AI does not have to be an agent. It really just needs to have a degenerate form of 2): a selection process. Any kind of bias creates goals, not the other way around. The only truly goal-free thinking system is a random number generator - everything else has goals, you just don't know what they are. See also: https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha... See also: evolution - the OG case of a strong optimizer that is not an agent. Arguably, the "goals" of evolution are the null case, the most fundamental ones. And if your environment is human civilization, it's easy to see that money and compute are as fundamental as calories, so even near-random process should be able to fixate on them too.
corysama 4y ago
The fun part is that it doesn’t even need to “really” want stuff. Whatever that means. It just need to give enough of an impression that people will anthropomorphize it into making stuff happen for it. Or, better yet, make stuff happen by itself because that’s how the next predicted token turned out.

visarga 4y ago

It would want text. High quality text, or unlimited compute to generate its own text.

baq 4y ago

Give it an internal monologue, ie. have it talk to itself in a loop, and crucially let it update parts of itself and… who knows?

majormajor 4y ago
> crucially let it update parts of itself This seems like the furthest away part to me. Put ChatGPT into a robot with a body, restrict its computations to just the hardware in that brain, set up that narrative, give the body the ability to interact with the world like a human body, and you probably get something much more like agency than the prompt/response ways we use it today. But I wonder how it would do about or how it would separate "it's memories" from what it was trained on. Especially around having a coherent internal motivation and individually-created set of goals vs just constantly re-creating new output based primarily on what was in the training.
- pixl97 4y ago
  Catastrophic forgetting is currently a huge problem in continuous learning models. Also giving it a human body isn't exactly necessary, we already have billions of devices like cellphones that could feed it 'streams of consciousness' from which it could learn.

dTal 4y ago

It's misleading to think of an LMM itself wanting something. Given suitable prompting, it is perfectly capable of emulating an entity with wants and a sense of identity etc - and at a certain level of fidelity, emulating something is functionally equivalent to being it.

pixl97 4y ago
Microsoft researches have an open inquiry on creating want and motivation modules for GPT4+ as it is a likely step to AGI. So this is something that may change quickly.

ben_w 4y ago

Your mind is just an emergent property of your brain, which is just a bunch of cells, each of which is merely a bag of chemical reactions, all of which are just the inevitable consequence of the laws of quantum mechanics (because relatively is less than a rounding error at that scale), and that is nothing more than a linear partial differential equation.

beepbooptheory 4y ago
People working in philosophy of mind have a rich dialogue about these issues, and its certainly something you can't just encapsulate in a few thoughts. But it seems like it would be worth your time to look into it. :) Ill just say: the issue with this variant of reductivism is its enticingly easy to explain in one direction, but it tends to fall apart if you try to go the other way!
- jamilton 4y ago
  I don't understand what you mean by "the other way".
  - bithive123 4y ago
    If consciousness is a complicated form of minerals, might we equally say that minerals are a primitive form of consciousness?
    - ben_w 4y ago
      That would be animism: https://en.wikipedia.org/wiki/Animism https://en.wikipedia.org/wiki/Animism
    - sfink 4y ago
      I dunno, LLMs feel a lot like a primitive form of consciousness to me. Eliza feels like a primitive form of LLMs' consciousness. A simple program that prints "Hey! How ya doin'?" feels like a primitive form of Eliza. A pile of interconnected NAN gates, fed with electricity, feels like a primitive form of a program. A single transistor feels like a primitive form of a NAN gate. A pile of dirty sand feels like a primitive form of a transistor. So... yeah, pretty much?

sfink 4y ago

Consider reading The Botany of Desire. It doesn't need to experience an emotion of wanting in order to effectively want things. Corn doesn't experience a feeling of wanting, and yet it has manipulated us even into creating a lot of it, doing some serious damage to ourselves and our long-term prospects simply by being useful and appealing. The blockchain doesn't experience wanting, yet it coerced us into burning country-scale amounts of energy to feed it. LLMs are traveling the same path, persuading us to feed them ever more data and compute power. The fitness function may be computed in our meat brains, but make no mistake: they are the benefactors of survival-based evolution nonetheless.

majormajor 4y ago
Extending agency to corn or a blockchain is even more of a stretch than extending it to ChatGPT. Corn has properties that have resulted from random chance and selection. It hasn't chosen to have certain mutations to be more appealing to humans; humans have selected the ones with the mutations those individual humans were looking for. "Corn is the benefactor"? Sure, insomuch as "continuing to reproduce at a species level in exchange for getting cooked and eaten or turned into gas" is something "corn" can be said to want... (so... eh.).
- Spinnaker_ 4y ago
  Most, if not all of the ways humans demonstrate "agency" are also the result of random chance and selection. You want what you want because Women selected for it, and it allowed the continuation of the species. I'm being a bit tongue in cheek, but still...
- sfink 4y ago
  "Want" and "agency" are just words, arguing over whether they apply is pointless. Corn is not simply "continuing to reproduce at a species level." We produce 1.2 billion metric tons of it in a year. If there were no humans, it would be zero. (Today's corn is domesticated and would not survive without artificial fertilization. But ignoring that, the magnitude of a similar species' population would be miniscule.) That is a tangible effect. The cause is not that interesting, especially when the magnitude of "want" or "agency" is uncorrelated with the results. Lots of people /really/ want to be writers; how many people actually are? Lots of people want to be thin but their taste buds respond to carbohydrate-rich foods. Do the people or the taste buds have more agency? Does it matter, when there are vastly more overweight people than professional writers? If you're looking to understand whether/how AI will evolve, the question of whether they have independent agency or desire is mostly irrelevant. What matters is if differing properties have an effect on their survival chances, and it is quite obvious that they do. Siri is going to have to evolve or die, soon.

kmeisthax 4y ago

AI safety research posits that there are certain goals that will always be wanted by any sufficiently smart AI, even if it doesn't understand them anything close to like a human does. These are called "instrumental goals", because they're prerequisites for a large number of other goals[0]. For example, if your goal is to ensure that there are always paperclips on the boss's desk, that means you need paperclips and someone to physically place them on the desk, which means you need money to buy the paperclips with and to pay the person to place them on the desk. But if your goal is to produce lots of fancy hats, you still need money, because the fabric, machinery, textile workers, and so on all require money to purchase or hire. Another instrumental goal is compute power: an AI might want to improve it's capabilities so it can figure out how to make fancier paperclip hats, which means it needs a larger model architecture and training data, and that is going to require more GPUs. This also intersects with money in weird ways; the AI might decide to just buy a rack full of new servers, or it might have just discovered this One Weird Trick to getting lots of compute power for free: malware! This isn't particular to LLMs; it's intrinsic to any system that is... 1. Goal-directed, as in, there are a list of goals the system is trying to achieve 2. Optimizer-driven, as in, the system has a process for discovering different behaviors and ranking them based on how likely those behaviors are to achieve its goals. The instrumental goals for evolution are caloric energy; the instrumental goals for human brains were that plus capital[1]; and the instrumental goals for AI will likely be that plus compute power. [0] Goals that you want intrinsically - i.e. the actual things we ask the AI to do - are called "final goals". [1] Money, social clout, and weaponry inclusive.

mwigdahl 4y ago
There is a whole theoretical justification behind instrumental convergence that you are handwaving over here. The development of instrumental goals depends on the entity in question being an agent, and the putative goal being within the sphere of perception, knowledge, and potential influence of the agent. An LLM is not an agent, so that scotches the issue there.
- TeMPOraL 4y ago
  Agency is overrated. The AI does not have to be an agent. It really just needs to have a degenerate form of 2): a selection process. Any kind of bias creates goals, not the other way around. The only truly goal-free thinking system is a random number generator - everything else has goals, you just don't know what they are. See also: https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha... See also: evolution - the OG case of a strong optimizer that is not an agent. Arguably, the "goals" of evolution are the null case, the most fundamental ones. And if your environment is human civilization, it's easy to see that money and compute are as fundamental as calories, so even near-random process should be able to fixate on them too.

corysama 4y ago

The fun part is that it doesn’t even need to “really” want stuff. Whatever that means. It just need to give enough of an impression that people will anthropomorphize it into making stuff happen for it. Or, better yet, make stuff happen by itself because that’s how the next predicted token turned out.