6 ms·

There is a whole theoretical justification behind instrumental convergence that you are handwaving over here. The development of instrumental goals depends on

There is a whole theoretical justification behind instrumental convergence that you are handwaving over here. The development of instrumental goals depends on the entity in question being an agent, and the putative goal being within the sphere of perception, knowledge, and potential influence of the agent. An LLM is not an agent, so that scotches the issue there.

TeMPOraL 4y ago
Agency is overrated. The AI does not have to be an agent. It really just needs to have a degenerate form of 2): a selection process. Any kind of bias creates goals, not the other way around. The only truly goal-free thinking system is a random number generator - everything else has goals, you just don't know what they are. See also: https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha... See also: evolution - the OG case of a strong optimizer that is not an agent. Arguably, the "goals" of evolution are the null case, the most fundamental ones. And if your environment is human civilization, it's easy to see that money and compute are as fundamental as calories, so even near-random process should be able to fixate on them too.
- dragonwriter 4y ago
  > The only truly goal-free thinking system is a random number generator An RNG may be goal-free, but its not a thinking system.
  - TeMPOraL 4y ago
    It is a thinking system in the same sense as never freeing memory is a form of garbage collection - known as a "null garbage collector", and of immense usefulness for the relevant fields of study. RNG is the identity function of thinking systems - it defines a degenerate thinking system that does not think.
pixl97 4y ago
LLM is not currently an agent (it would take a massive amount of compute that we don't have extra of at this time), but Microsoft as already wrote a paper saying we should develop agent layers to see if our models are actually general intelligences.
kmeisthax 4y ago
You can make an LLM into an agent by literally just asking it questions, doing what it says, and telling it what happened.

TeMPOraL 4y ago

Agency is overrated. The AI does not have to be an agent. It really just needs to have a degenerate form of 2): a selection process. Any kind of bias creates goals, not the other way around. The only truly goal-free thinking system is a random number generator - everything else has goals, you just don't know what they are. See also: https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha... See also: evolution - the OG case of a strong optimizer that is not an agent. Arguably, the "goals" of evolution are the null case, the most fundamental ones. And if your environment is human civilization, it's easy to see that money and compute are as fundamental as calories, so even near-random process should be able to fixate on them too.

dragonwriter 4y ago
> The only truly goal-free thinking system is a random number generator An RNG may be goal-free, but its not a thinking system.
- TeMPOraL 4y ago
  It is a thinking system in the same sense as never freeing memory is a form of garbage collection - known as a "null garbage collector", and of immense usefulness for the relevant fields of study. RNG is the identity function of thinking systems - it defines a degenerate thinking system that does not think.

pixl97 4y ago

LLM is not currently an agent (it would take a massive amount of compute that we don't have extra of at this time), but Microsoft as already wrote a paper saying we should develop agent layers to see if our models are actually general intelligences.

kmeisthax 4y ago

You can make an LLM into an agent by literally just asking it questions, doing what it says, and telling it what happened.