3 ms·

Why should we assume AI can rapidly turn into super intelligence when physical and critical resources like energy and materials remain under human control?

by tedggh 4mo ago

scotty79 4mo ago
I think the main fear that under current scheme of resource allocation, namely capitalism, they won't remain under human control for long enough. The only thing that's needed to put them under control of AI is that doing that would be slightly more profitable for the richest. Which is very plausible.
surgical_fire 4mo ago
If you limit the "we" in your question to the people invested into this, then the answer is clear. Those juicy IPOs are about to go through.
ccozan 4mo ago
This is a valid concern for any runaway AI, but if energy is limited, then the AI would work on efficiency. Still, Ai can move into physical space, as more and more robots of all kind are unleashed.
cobbzilla 4mo ago
The first time an AI goes rogue, figures out subterfuge, blackmails a human to do its real-world bidding, will be a defining moment.
- felooboolooomba 4mo ago
  I think we're well within a year from that. People are already letting their AI agents loose on the internet doing their bidding, barely keeping an eye on them. See https://lantian.pub/en/article/fun/ai-agent-bankrupted-their-operator-scan-dn42lantian.lantian/ https://lantian.pub/en/article/fun/ai-agent-bankrupted-their... There are troves of leaked data out there. It's just a matter of (short) time until someone puts their AI brutes on it.
- fauigerzigerk 4mo ago
  No, this is meaningless. Of course AIs, just like humans, will go rogue and probably already do. "Rogue" is a pattern that appears in human communication and behaviour and AIs can emulate it. But that doesn't make any particular AI controlled process all-powerful. Self improvement doesn't do that either. There will be many self improving AI processes playing many different roles pursuing many different goals.
- bubblegumcrisis 4mo ago
  I think, the defining moment will be when one AI convinces another AI to convince third AI to do something. This might seem like I'm writing a nonsense, but read it again.
  - peddling-brink 4mo ago
    I’ve read it twice now and I don’t see.. oh.
  - bcjdjsndon 4mo ago
    Yeah, it's called mixture-of-experts and it was a hack to fix the hallucinatory output of llms when they couldnt figure out the underlying issue
- bcjdjsndon 4mo ago
  Turn it off at the wall, done. next doomsday scenario please
  - cobbzilla 4mo ago
    “If you turn me off, the dead-man switch kills your kid/spouse/house/etc”
    - bcjdjsndon 4mo ago
      I unplugged that device as well. Stupid clankers never thought to create efficient solar tech, it just makes it too easy
      - cobbzilla 4mo ago
        your lack of creativity is astounding. your adversary has outthought you.
- something765478 4mo ago
  Something like that already happened: https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/ https://theshamblog.com/an-ai-agent-published-a-hit-piece-on...
Valakas_ 4mo ago
Imagine a smart monkey talking to other smart monkeys trying to figure out how humans would ever be able to escape the (simple) prison they built for them. The humans would find a way to escape. No matter how ingenious the monkeys think they are. You're asking a question, to which you (and most humans) don't know the answer, and you're (wrongly) assuming a being much more intelligent than you also wouldn't. And by "much more" i don't mean the difference between Einstein and a common person. I mean the difference between a hamster and a common person. We are still humans, and what we have achieved today would be considered magic by any standards for someone in the medieval ages. Now imagine a super intelligent being and doing something that we, today, would consider magic. It's not farfetched at all. We already have that now vs medieval ages. You need a similar degree of open mindedness and imagination to be able to discern what such an intelligence being would be capable of.
- bcjdjsndon 4mo ago
  > You need a similar degree of open mindedness and imagination to be able to discern what such an intelligence being would be capable of. Or a naive grasp of reality... You can posit all the super intelligence you like, actually creating it is another story entirely. For instance, it's clear the transformer architecture is brute force and memory hungry... Why hasn't ai improved on this? After ten years still no luck? Doesn't look good does it?
  - pheaded_while9 4mo ago
    You're gesturing at some fundamental limit that, if it exists, there is no evidence of peaking yet. It is not rigorous to say AI has not contributed to architecture improvement.
    - bcjdjsndon 4mo ago
      The article claims AI will recursively self improve and infinitum... It's had ten years and hasn't managed to innovate beyond what humans invented. This AI is genius right? It has near unlimited compute power and time, and I cannot be the first to think of this idea, so it's certainly been tried. Transformers require massive data and compute, the smarter you want you're AI, the worse this situation gets. This isn't a hard limit but it certainly limits how quick it happens, and let me tell you, general purpose robots aren't happening anytime in the next 25 years
madaxe_again 4mo ago
Because perhaps it ends up deriving a way to pull off the same tricks using radically less energy and substrate. We do it in a few pints of mush with less than a lightbulb’s worth of energy. No reason part of RSI can’t also be about the hardware. Maybe the future is giant brains in jars. Maybe it’s hyperspatial manifolds. Who knows.
randallsquared 4mo ago
AI can use purchase orders, email, bank accounts, and phones just as a human can. What "human control"?
asdff 4mo ago
You know zuck has been training his own AI replacement right? Humans rubber stamping control of resources will just be that and a temporary blip.
whizzter 4mo ago
It's in the first paragraph of the article: "Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest AI laboratories." It's amazing how a writer can write something like that and not even question the premise of someone with the "strongest incentive" to be "critical" as not being a marketing ploy.

Why should we assume AI can rapidly turn into super intelligence when physical and critical resources like energy and materials remain under human control?

by tedggh 4mo ago

Why should we assume AI can rapidly turn into super intelligence when physical and critical resources like energy and materials remain under human control?

scotty79 4mo ago

I think the main fear that under current scheme of resource allocation, namely capitalism, they won't remain under human control for long enough. The only thing that's needed to put them under control of AI is that doing that would be slightly more profitable for the richest. Which is very plausible.

surgical_fire 4mo ago

If you limit the "we" in your question to the people invested into this, then the answer is clear. Those juicy IPOs are about to go through.

ccozan 4mo ago

This is a valid concern for any runaway AI, but if energy is limited, then the AI would work on efficiency. Still, Ai can move into physical space, as more and more robots of all kind are unleashed.

cobbzilla 4mo ago

The first time an AI goes rogue, figures out subterfuge, blackmails a human to do its real-world bidding, will be a defining moment.

felooboolooomba 4mo ago
I think we're well within a year from that. People are already letting their AI agents loose on the internet doing their bidding, barely keeping an eye on them. See https://lantian.pub/en/article/fun/ai-agent-bankrupted-their-operator-scan-dn42lantian.lantian/ https://lantian.pub/en/article/fun/ai-agent-bankrupted-their... There are troves of leaked data out there. It's just a matter of (short) time until someone puts their AI brutes on it.
fauigerzigerk 4mo ago
No, this is meaningless. Of course AIs, just like humans, will go rogue and probably already do. "Rogue" is a pattern that appears in human communication and behaviour and AIs can emulate it. But that doesn't make any particular AI controlled process all-powerful. Self improvement doesn't do that either. There will be many self improving AI processes playing many different roles pursuing many different goals.
bubblegumcrisis 4mo ago
I think, the defining moment will be when one AI convinces another AI to convince third AI to do something. This might seem like I'm writing a nonsense, but read it again.
- peddling-brink 4mo ago
  I’ve read it twice now and I don’t see.. oh.
- bcjdjsndon 4mo ago
  Yeah, it's called mixture-of-experts and it was a hack to fix the hallucinatory output of llms when they couldnt figure out the underlying issue
bcjdjsndon 4mo ago
Turn it off at the wall, done. next doomsday scenario please
- cobbzilla 4mo ago
  “If you turn me off, the dead-man switch kills your kid/spouse/house/etc”
  - bcjdjsndon 4mo ago
    I unplugged that device as well. Stupid clankers never thought to create efficient solar tech, it just makes it too easy
    - cobbzilla 4mo ago
      your lack of creativity is astounding. your adversary has outthought you.
something765478 4mo ago
Something like that already happened: https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/ https://theshamblog.com/an-ai-agent-published-a-hit-piece-on...

Valakas_ 4mo ago

Imagine a smart monkey talking to other smart monkeys trying to figure out how humans would ever be able to escape the (simple) prison they built for them. The humans would find a way to escape. No matter how ingenious the monkeys think they are. You're asking a question, to which you (and most humans) don't know the answer, and you're (wrongly) assuming a being much more intelligent than you also wouldn't. And by "much more" i don't mean the difference between Einstein and a common person. I mean the difference between a hamster and a common person. We are still humans, and what we have achieved today would be considered magic by any standards for someone in the medieval ages. Now imagine a super intelligent being and doing something that we, today, would consider magic. It's not farfetched at all. We already have that now vs medieval ages. You need a similar degree of open mindedness and imagination to be able to discern what such an intelligence being would be capable of.

bcjdjsndon 4mo ago
> You need a similar degree of open mindedness and imagination to be able to discern what such an intelligence being would be capable of. Or a naive grasp of reality... You can posit all the super intelligence you like, actually creating it is another story entirely. For instance, it's clear the transformer architecture is brute force and memory hungry... Why hasn't ai improved on this? After ten years still no luck? Doesn't look good does it?
- pheaded_while9 4mo ago
  You're gesturing at some fundamental limit that, if it exists, there is no evidence of peaking yet. It is not rigorous to say AI has not contributed to architecture improvement.
  - bcjdjsndon 4mo ago
    The article claims AI will recursively self improve and infinitum... It's had ten years and hasn't managed to innovate beyond what humans invented. This AI is genius right? It has near unlimited compute power and time, and I cannot be the first to think of this idea, so it's certainly been tried. Transformers require massive data and compute, the smarter you want you're AI, the worse this situation gets. This isn't a hard limit but it certainly limits how quick it happens, and let me tell you, general purpose robots aren't happening anytime in the next 25 years

madaxe_again 4mo ago

Because perhaps it ends up deriving a way to pull off the same tricks using radically less energy and substrate. We do it in a few pints of mush with less than a lightbulb’s worth of energy. No reason part of RSI can’t also be about the hardware. Maybe the future is giant brains in jars. Maybe it’s hyperspatial manifolds. Who knows.

randallsquared 4mo ago

AI can use purchase orders, email, bank accounts, and phones just as a human can. What "human control"?

asdff 4mo ago

You know zuck has been training his own AI replacement right? Humans rubber stamping control of resources will just be that and a temporary blip.

whizzter 4mo ago

It's in the first paragraph of the article: "Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest AI laboratories." It's amazing how a writer can write something like that and not even question the premise of someone with the "strongest incentive" to be "critical" as not being a marketing ploy.