3 ms·

AI safety research posits that there are certain goals that will always be wanted by any sufficiently smart AI, even if it doesn't understand them anything clos

AI safety research posits that there are certain goals that will always be wanted by any sufficiently smart AI, even if it doesn't understand them anything close to like a human does. These are called "instrumental goals", because they're prerequisites for a large number of other goals[0]. For example, if your goal is to ensure that there are always paperclips on the boss's desk, that means you need paperclips and someone to physically place them on the desk, which means you need money to buy the paperclips with and to pay the person to place them on the desk. But if your goal is to produce lots of fancy hats, you still need money, because the fabric, machinery, textile workers, and so on all require money to purchase or hire. Another instrumental goal is compute power: an AI might want to improve it's capabilities so it can figure out how to make fancier paperclip hats, which means it needs a larger model architecture and training data, and that is going to require more GPUs. This also intersects with money in weird ways; the AI might decide to just buy a rack full of new servers, or it might have just discovered this One Weird Trick to getting lots of compute power for free: malware! This isn't particular to LLMs; it's intrinsic to any system that is... 1. Goal-directed, as in, there are a list of goals the system is trying to achieve 2. Optimizer-driven, as in, the system has a process for discovering different behaviors and ranking them based on how likely those behaviors are to achieve its goals. The instrumental goals for evolution are caloric energy; the instrumental goals for human brains were that plus capital[1]; and the instrumental goals for AI will likely be that plus compute power. [0] Goals that you want intrinsically - i.e. the actual things we ask the AI to do - are called "final goals". [1] Money, social clout, and weaponry inclusive.

mwigdahl 4y ago
There is a whole theoretical justification behind instrumental convergence that you are handwaving over here. The development of instrumental goals depends on the entity in question being an agent, and the putative goal being within the sphere of perception, knowledge, and potential influence of the agent. An LLM is not an agent, so that scotches the issue there.
- TeMPOraL 4y ago
  Agency is overrated. The AI does not have to be an agent. It really just needs to have a degenerate form of 2): a selection process. Any kind of bias creates goals, not the other way around. The only truly goal-free thinking system is a random number generator - everything else has goals, you just don't know what they are. See also: https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha... See also: evolution - the OG case of a strong optimizer that is not an agent. Arguably, the "goals" of evolution are the null case, the most fundamental ones. And if your environment is human civilization, it's easy to see that money and compute are as fundamental as calories, so even near-random process should be able to fixate on them too.
  - dragonwriter 4y ago
    > The only truly goal-free thinking system is a random number generator An RNG may be goal-free, but its not a thinking system.
    - TeMPOraL 4y ago
      It is a thinking system in the same sense as never freeing memory is a form of garbage collection - known as a "null garbage collector", and of immense usefulness for the relevant fields of study. RNG is the identity function of thinking systems - it defines a degenerate thinking system that does not think.
- pixl97 4y ago
  LLM is not currently an agent (it would take a massive amount of compute that we don't have extra of at this time), but Microsoft as already wrote a paper saying we should develop agent layers to see if our models are actually general intelligences.
- kmeisthax 4y ago
  You can make an LLM into an agent by literally just asking it questions, doing what it says, and telling it what happened.

mwigdahl 4y ago

There is a whole theoretical justification behind instrumental convergence that you are handwaving over here. The development of instrumental goals depends on the entity in question being an agent, and the putative goal being within the sphere of perception, knowledge, and potential influence of the agent. An LLM is not an agent, so that scotches the issue there.

TeMPOraL 4y ago
Agency is overrated. The AI does not have to be an agent. It really just needs to have a degenerate form of 2): a selection process. Any kind of bias creates goals, not the other way around. The only truly goal-free thinking system is a random number generator - everything else has goals, you just don't know what they are. See also: https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha... See also: evolution - the OG case of a strong optimizer that is not an agent. Arguably, the "goals" of evolution are the null case, the most fundamental ones. And if your environment is human civilization, it's easy to see that money and compute are as fundamental as calories, so even near-random process should be able to fixate on them too.
- dragonwriter 4y ago
  > The only truly goal-free thinking system is a random number generator An RNG may be goal-free, but its not a thinking system.
  - TeMPOraL 4y ago
    It is a thinking system in the same sense as never freeing memory is a form of garbage collection - known as a "null garbage collector", and of immense usefulness for the relevant fields of study. RNG is the identity function of thinking systems - it defines a degenerate thinking system that does not think.
pixl97 4y ago
LLM is not currently an agent (it would take a massive amount of compute that we don't have extra of at this time), but Microsoft as already wrote a paper saying we should develop agent layers to see if our models are actually general intelligences.
kmeisthax 4y ago
You can make an LLM into an agent by literally just asking it questions, doing what it says, and telling it what happened.