5 ms·

Let’s say you got AGI, and it approximated a not so bright impulsive 12 year old boy. That would be an insane technological leap, but hardly one you’d want runn

Let’s say you got AGI, and it approximated a not so bright impulsive 12 year old boy. That would be an insane technological leap, but hardly one you’d want running the show. AGI doesn’t mean smarter than the best humans.

__MatrixMan__ 2y ago
For an intelligence to be "General" there would have to be no type of intelligence that it did not have access to (even if its capabilities in that domain were limited). The idea that that's what humans have strikes me as the same kind of thinking that led us to believe that earth was in the center of the universe. Surely there are ways of thinking that we have no concept of. General intelligence would be like an impulsive 12 year old boy who could see 6 spatial dimensions and regarded us as cartoons for only sticking to 3.
- throwaway314155 2y ago
  Humans can survive in space and on the moon because our intelligence is robust to environments we never encountered (or evolved to encounter). That's "all" general intelligence is meant to refer to. General just means robust to the unknown. I've seen some use "super" (as in superhuman) intelligence lately to describe what you're getting at.
  - __MatrixMan__ 2y ago
    Super feels to me like it's a difference in degree rather than in kind. Something with intelligences {A, b, c} might be super intelligent compared with {a, b, c}. i.e. more intelligent in domain A. But if one has {a, b, c} and the other has {b, c, d} neither is more or less intelligent than the other, they just have different capabilities. "Super" is a bit too one-dimensional for the job.
    - throwaway314155 2y ago
      I don't think using set notation is really helping your case here but (I think?) I agree.
- tialaramex 2y ago
  No. AGI is talking about the "general" human intelligence rather than specialism. An AGI would be as good as a human at both composing poetry and playing chess, both identifying bird calls and proving mathematical theorems. We don't know what a categorically superior intelligence would be so we can't rate some machine on that basis. The Lem story "Golem XIV" concerns a machine which claims it possesses categorically superior intelligence, and further that another machine humans have built (which runs but seems unwilling to communicate with them at all) is even more intelligent still. Golem tries to explain using analogies, but it's apparent that it finds communicating with humans frustrating, the way it might be frustrating to try to explain things to a golden retriever. Lem wrote elsewhere that the single commonality between Golem's intelligence and our own is curiosity, unlike Annie, Golem is curious about the humans which is why it's bothering to communicate with them. Humans (of course) plot to destroy both machines. Annie eliminates the conspirators, betraying a hitherto un-imagined capability to act at great distance, and the story remarks that it seems she does so the way a human would swat a buzzing insect. She doesn't fear the humans, but they're annoying so she destroyed them without a thought.
Triphibian 2y ago
All it would want to do is talk about Minecraft and the funny memes they saw.
nobodyandproud 2y ago
This is great insight. Science fiction does a great job inspiring technology research, but we forget that real-world accomplishments can be much humbler and still astonishing all the same. I'm a bit tired of the hype surrounding LLMs, but all the same for very mundane and humbler tasks that require some intelligence modern LLMs manage to surprise me on a daily basis. But it rarely accomplishes more than what a small collection of humans with some level of expertise can achieve, when asked.
- Nevermark 2y ago
  Or accomplishments can genuinely be astounding. Surely, the LLM models we have today are astounding by any measure, relative to just a few years ago. But pronouncements of how this will lead to utopia, without introducing a major revision of economic arrangements, are completely, and surely intentionally/conveniently (Sam isn't an idiot) misleading. Is OpenAI creating a class of stock so everyone can share in their gains? If not, then AGI owned by OpenAI will make OpenAI shareholders rich, very much to the degree its AGI eliminates human jobs for itself and other corporations. How does that, as an economic situation, result in the general population being able to do anything beyond be a customer, assuming they can still make money in some way not taken over by AGI? Utopia needs an actual plan. Not a concept of a plan. The latter just keeps people snowed and calm before an historic level rug pull.
prng2021 2y ago
If it's stuck at 12 year old level intelligence then it's not generally intelligent. 12 years can learn to think like 13 year olds and so on.
- azinman2 2y ago
  A set of weights siting on your hard disk don’t evolve to get smarter on their own. AGI does however require the ability to learn new skills in new domains; if you kept exposing it I don’t see why it wouldn’t be similar. But fundamental capacities in how good it is a reasoning / planning don’t have to exceed any given human to be considered AGI.
richardw 2y ago
A 12 year old that doesn’t sleep, eat, or forget, learns anything within seconds, can replicate and work together with fast as light communication and no data loss when doing so. Arrives out of the womb knowing everything the internet knows. Can work on one problem until it gets it right. Doesn’t get stale or old. Only had emotion if it’s useful. Can do math and simulate. Doesn’t get bored. Can theoretically improve its own design and run experiments with aforementioned focus, simulation ability, etc. What could you do at 12, with half of these advantages? Choose any of them, then give yourself infinite time to use them.
- pstuart 2y ago
  And then with advancement, reaches teenage maturity and tells everyone to fuck off.
  - thordenmark 2y ago
    As someone who has teenagers right now, I can confirm that this is accurate.
- SoftTalker 2y ago
  Why do we assume an AGI would not be forgetful or would never get bored?
  - richardw 2y ago
    Because we can change the hardware or software to be what we want. It can write to storage and read it, like we can but faster.
- rktan 2y ago
  A 12 year old could determine that "AI" is boring and counterproductive for humanity and switch off a computer or data center. Greta Thunberg did similar for the climate, perhaps we need a new child saint who fights "AI".
- bad_user 2y ago
  A 12 year old that doesn't interact with the world. Many believe that AGI will happen in robots, and not in online services, simply because interacting with the environment might be a prerequisite for developing consciousness. You mentioned boredom, which is interesting, as boredom may also be a trait of intelligence. An interesting question is if it will want to live at all. Humans have all these pleasure sensors and programming for staying alive and reproducing. The unburdened AGI in your description might not have good reasons to live. Marvin, the depressed robot, might become real.
pounderstanding 2y ago
> AGI doesn’t mean smarter than the best humans. Technically no, but practically... 12 year old limitations are: A. gets tired, needs sleep B. I/O limited by muscles Probably there are more, but if 12 year old could talk directly to electric circuits and would not need sleep or even a break, then that 12 year old would be leaps and bounds above the best human in his field of interest. (Well motivation to finish the task is needed though)
esskay 2y ago
What stops it getting more intelegent? That's litterally its primary aim. Its only limit as to how quickly it does that is hardware capacity, and I'd be shocked if it didn't somehow theorise to itself that it can and should expand its knowledge in any way possible.
- echoangle 2y ago
  Why would the only limit be hardware capacity? Can’t there be an innate limit due to the model size/architecture? Maybe LLMs can’t be superintelligent because they are missing a critical ability which can’t be overcome with any amount of training? It’s not obvious to me that it’s going to get smarter infinitely.
  - casta 2y ago
    The limit could also be in the training data. If we could train LLMs with a corpus generated by some super intelligent entity, maybe we'd consider them super intelligent even with the same model size/architecture we have today.

__MatrixMan__ 2y ago

For an intelligence to be "General" there would have to be no type of intelligence that it did not have access to (even if its capabilities in that domain were limited). The idea that that's what humans have strikes me as the same kind of thinking that led us to believe that earth was in the center of the universe. Surely there are ways of thinking that we have no concept of. General intelligence would be like an impulsive 12 year old boy who could see 6 spatial dimensions and regarded us as cartoons for only sticking to 3.

throwaway314155 2y ago
Humans can survive in space and on the moon because our intelligence is robust to environments we never encountered (or evolved to encounter). That's "all" general intelligence is meant to refer to. General just means robust to the unknown. I've seen some use "super" (as in superhuman) intelligence lately to describe what you're getting at.
- __MatrixMan__ 2y ago
  Super feels to me like it's a difference in degree rather than in kind. Something with intelligences {A, b, c} might be super intelligent compared with {a, b, c}. i.e. more intelligent in domain A. But if one has {a, b, c} and the other has {b, c, d} neither is more or less intelligent than the other, they just have different capabilities. "Super" is a bit too one-dimensional for the job.
  - throwaway314155 2y ago
    I don't think using set notation is really helping your case here but (I think?) I agree.
tialaramex 2y ago
No. AGI is talking about the "general" human intelligence rather than specialism. An AGI would be as good as a human at both composing poetry and playing chess, both identifying bird calls and proving mathematical theorems. We don't know what a categorically superior intelligence would be so we can't rate some machine on that basis. The Lem story "Golem XIV" concerns a machine which claims it possesses categorically superior intelligence, and further that another machine humans have built (which runs but seems unwilling to communicate with them at all) is even more intelligent still. Golem tries to explain using analogies, but it's apparent that it finds communicating with humans frustrating, the way it might be frustrating to try to explain things to a golden retriever. Lem wrote elsewhere that the single commonality between Golem's intelligence and our own is curiosity, unlike Annie, Golem is curious about the humans which is why it's bothering to communicate with them. Humans (of course) plot to destroy both machines. Annie eliminates the conspirators, betraying a hitherto un-imagined capability to act at great distance, and the story remarks that it seems she does so the way a human would swat a buzzing insect. She doesn't fear the humans, but they're annoying so she destroyed them without a thought.

Triphibian 2y ago

All it would want to do is talk about Minecraft and the funny memes they saw.

nobodyandproud 2y ago

This is great insight. Science fiction does a great job inspiring technology research, but we forget that real-world accomplishments can be much humbler and still astonishing all the same. I'm a bit tired of the hype surrounding LLMs, but all the same for very mundane and humbler tasks that require some intelligence modern LLMs manage to surprise me on a daily basis. But it rarely accomplishes more than what a small collection of humans with some level of expertise can achieve, when asked.

Nevermark 2y ago
Or accomplishments can genuinely be astounding. Surely, the LLM models we have today are astounding by any measure, relative to just a few years ago. But pronouncements of how this will lead to utopia, without introducing a major revision of economic arrangements, are completely, and surely intentionally/conveniently (Sam isn't an idiot) misleading. Is OpenAI creating a class of stock so everyone can share in their gains? If not, then AGI owned by OpenAI will make OpenAI shareholders rich, very much to the degree its AGI eliminates human jobs for itself and other corporations. How does that, as an economic situation, result in the general population being able to do anything beyond be a customer, assuming they can still make money in some way not taken over by AGI? Utopia needs an actual plan. Not a concept of a plan. The latter just keeps people snowed and calm before an historic level rug pull.

prng2021 2y ago

If it's stuck at 12 year old level intelligence then it's not generally intelligent. 12 years can learn to think like 13 year olds and so on.

azinman2 2y ago
A set of weights siting on your hard disk don’t evolve to get smarter on their own. AGI does however require the ability to learn new skills in new domains; if you kept exposing it I don’t see why it wouldn’t be similar. But fundamental capacities in how good it is a reasoning / planning don’t have to exceed any given human to be considered AGI.

richardw 2y ago

A 12 year old that doesn’t sleep, eat, or forget, learns anything within seconds, can replicate and work together with fast as light communication and no data loss when doing so. Arrives out of the womb knowing everything the internet knows. Can work on one problem until it gets it right. Doesn’t get stale or old. Only had emotion if it’s useful. Can do math and simulate. Doesn’t get bored. Can theoretically improve its own design and run experiments with aforementioned focus, simulation ability, etc. What could you do at 12, with half of these advantages? Choose any of them, then give yourself infinite time to use them.

pstuart 2y ago
And then with advancement, reaches teenage maturity and tells everyone to fuck off.
- thordenmark 2y ago
  As someone who has teenagers right now, I can confirm that this is accurate.
SoftTalker 2y ago
Why do we assume an AGI would not be forgetful or would never get bored?
- richardw 2y ago
  Because we can change the hardware or software to be what we want. It can write to storage and read it, like we can but faster.
rktan 2y ago
A 12 year old could determine that "AI" is boring and counterproductive for humanity and switch off a computer or data center. Greta Thunberg did similar for the climate, perhaps we need a new child saint who fights "AI".
bad_user 2y ago
A 12 year old that doesn't interact with the world. Many believe that AGI will happen in robots, and not in online services, simply because interacting with the environment might be a prerequisite for developing consciousness. You mentioned boredom, which is interesting, as boredom may also be a trait of intelligence. An interesting question is if it will want to live at all. Humans have all these pleasure sensors and programming for staying alive and reproducing. The unburdened AGI in your description might not have good reasons to live. Marvin, the depressed robot, might become real.

pounderstanding 2y ago

> AGI doesn’t mean smarter than the best humans. Technically no, but practically... 12 year old limitations are: A. gets tired, needs sleep B. I/O limited by muscles Probably there are more, but if 12 year old could talk directly to electric circuits and would not need sleep or even a break, then that 12 year old would be leaps and bounds above the best human in his field of interest. (Well motivation to finish the task is needed though)

esskay 2y ago

What stops it getting more intelegent? That's litterally its primary aim. Its only limit as to how quickly it does that is hardware capacity, and I'd be shocked if it didn't somehow theorise to itself that it can and should expand its knowledge in any way possible.

echoangle 2y ago
Why would the only limit be hardware capacity? Can’t there be an innate limit due to the model size/architecture? Maybe LLMs can’t be superintelligent because they are missing a critical ability which can’t be overcome with any amount of training? It’s not obvious to me that it’s going to get smarter infinitely.
- casta 2y ago
  The limit could also be in the training data. If we could train LLMs with a corpus generated by some super intelligent entity, maybe we'd consider them super intelligent even with the same model size/architecture we have today.