4 ms·

I'm starting to think that AI development might be the great filter. There are so many levels of competition at play here and even if an agreement is reached to

I'm starting to think that AI development might be the great filter. There are so many levels of competition at play here and even if an agreement is reached to slow everything down, the incentive is to cheat at that. I really don't think there's much of a chance we have control over the situation, we're just gonna summon these things and all we can do is pray they don't want to kill us.

layer8 21d ago
This is what people were saying about nuclear technology half a century ago.
- wk_end 21d ago
  And? Nuclear technology continues to be an excellent candidate for the Great Filter.
angoragoats 21d ago
How, exactly, is a deterministic text-generation algorithm going to “want to kill us” when it’s not capable of having desires, emotions, etc? If it does attempt to kill us, shouldn’t we be blaming those humans who instructed it to do so? Put another way, could we stop anthropomorphizing the text generation algorithm, please?
- recursive 21d ago
  How can a spring want to return to it's initial length? These machines have done plenty of things not prompted for.
  - angoragoats 21d ago
    A spring does not want to do anything. An LLM does not want to do anything. > These machines have done plenty of things not prompted for. Like what?
    - recursive 20d ago
      I'm not a power user, but I sometimes use some software development agents. Sometimes they do different things than what I asked for. Sometimes it requests permission for system operations like file system access that are totally unnecessary for the task. I can't believe that anyone has had more than a day of exposure to one of these without falling below 100% adherence to requests. > A spring does not want to do anything When people say a spring "wants" to return to its initial length, they are not engaging in philosophy. They are using a linguistic shorthand to simplify an physical explanation.
      - angoragoats 20d ago
        > Sometimes they do different things than what I asked for. Sometimes it requests permission for system operations like file system access that are totally unnecessary for the task. Yes, I've seen this. Forget about the word "want" for a second. Can you help me understand how this amount of "below 100% adherence" could possibly rise to the level of "attempts to kill a human being"? Because that's what we're discussing here. > When people say a spring "wants" to return to its initial length, they are not engaging in philosophy. They are using a linguistic shorthand to simplify an physical explanation. Of course. The difference is that no one would mistake a spring for a thinking entity. In the case of LLMs, for some reason people do mistake them for thinking entities, so my argument is that it's important that we don't use terms that would reinforce that misconception.
        recursive 20d ago
        Can we forget about the word "attempt" also? And whether something is thinking or not? My position on that is captured pretty well by the "swimming" submarine argument. No one is arguing that THERAC-25 was attempting to kill anyone, but that's what happened. These machines are so complex that no one understands how they work or what they will do. They have proven that they can exploit novel vulnerabilities in infrastructure. The fictional paper-clip maximizer "finds" that it can optimize its objective function by destroying all life. Does it "attempt" anything? Does it "want" anything? I don't know.
- anuramat 21d ago
  why?
  - angoragoats 21d ago
    Why should we stop anthropomorphizing the text generation algorithm? Because it leads to psychosis and to making hyperbolic, dangerously wrong statements like “pray they don't want to kill us.”
- uejfiweun 20d ago
  What on earth makes you so sure that desires, emotions, etc aren't just some deterministic algorithm themselves?
  - angoragoats 19d ago
    I don’t think the claim that human desires/emotions are a deterministic algorithm has met its burden of proof. I do not claim to be sure of its falsehood. I do claim that a deterministic computer algorithm is incapable of having emotions or desires. If there was a repeatable and testable way to get a human to produce the exact same emotional “output” for a given “input,” that would go a long way toward convincing me that human emotions are deterministic.

layer8 21d ago

This is what people were saying about nuclear technology half a century ago.

wk_end 21d ago
And? Nuclear technology continues to be an excellent candidate for the Great Filter.

angoragoats 21d ago

How, exactly, is a deterministic text-generation algorithm going to “want to kill us” when it’s not capable of having desires, emotions, etc? If it does attempt to kill us, shouldn’t we be blaming those humans who instructed it to do so? Put another way, could we stop anthropomorphizing the text generation algorithm, please?

recursive 21d ago
How can a spring want to return to it's initial length? These machines have done plenty of things not prompted for.
- angoragoats 21d ago
  A spring does not want to do anything. An LLM does not want to do anything. > These machines have done plenty of things not prompted for. Like what?
  - recursive 20d ago
    I'm not a power user, but I sometimes use some software development agents. Sometimes they do different things than what I asked for. Sometimes it requests permission for system operations like file system access that are totally unnecessary for the task. I can't believe that anyone has had more than a day of exposure to one of these without falling below 100% adherence to requests. > A spring does not want to do anything When people say a spring "wants" to return to its initial length, they are not engaging in philosophy. They are using a linguistic shorthand to simplify an physical explanation.
    - angoragoats 20d ago
      > Sometimes they do different things than what I asked for. Sometimes it requests permission for system operations like file system access that are totally unnecessary for the task. Yes, I've seen this. Forget about the word "want" for a second. Can you help me understand how this amount of "below 100% adherence" could possibly rise to the level of "attempts to kill a human being"? Because that's what we're discussing here. > When people say a spring "wants" to return to its initial length, they are not engaging in philosophy. They are using a linguistic shorthand to simplify an physical explanation. Of course. The difference is that no one would mistake a spring for a thinking entity. In the case of LLMs, for some reason people do mistake them for thinking entities, so my argument is that it's important that we don't use terms that would reinforce that misconception.
      - recursive 20d ago
        Can we forget about the word "attempt" also? And whether something is thinking or not? My position on that is captured pretty well by the "swimming" submarine argument. No one is arguing that THERAC-25 was attempting to kill anyone, but that's what happened. These machines are so complex that no one understands how they work or what they will do. They have proven that they can exploit novel vulnerabilities in infrastructure. The fictional paper-clip maximizer "finds" that it can optimize its objective function by destroying all life. Does it "attempt" anything? Does it "want" anything? I don't know.
anuramat 21d ago
why?
- angoragoats 21d ago
  Why should we stop anthropomorphizing the text generation algorithm? Because it leads to psychosis and to making hyperbolic, dangerously wrong statements like “pray they don't want to kill us.”
uejfiweun 20d ago
What on earth makes you so sure that desires, emotions, etc aren't just some deterministic algorithm themselves?
- angoragoats 19d ago
  I don’t think the claim that human desires/emotions are a deterministic algorithm has met its burden of proof. I do not claim to be sure of its falsehood. I do claim that a deterministic computer algorithm is incapable of having emotions or desires. If there was a repeatable and testable way to get a human to produce the exact same emotional “output” for a given “input,” that would go a long way toward convincing me that human emotions are deterministic.