4 ms·

This is a very roundabout way of saying "Anyone not agreeing with me is simply not smart enough". Which might be true, sometimes, but also might not. And espe

by hypfer 28d ago

This is a very roundabout way of saying "Anyone not agreeing with me is simply not smart enough". Which might be true, sometimes, but also might not. And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".

0xDEAFBEAD 28d ago
It's not about being smart, it's about accounting for your own ignorance. >And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI". I'm just describing the general pattern I see in cognitive tendencies. If you can think of a way to make the fundamental point about the limitations of our knowledge in a way that's still compelling but less antagonistic, feel free to suggest how I could've rewritten my comment.
- hypfer 28d ago
  Why would I support your in your doomer lobbying quest by telling you how you can better hack people? My safeguards blocked this request.
- VulgarExigency 28d ago
  Perhaps the general pattern you see is not a "fundamental difference"? Could it be that you are also falling prey to the "all you see is all there is" cognitive bias?
  - 0xDEAFBEAD 28d ago
    Sure, there's a trivial sense in which that's always true whenever someone makes any sort of remark about the world. There's always a sense in which there could be factors that I'm not accounting for.
polishTar 28d ago
I heard someone use the analogy of "If Magnus Carlson played me at chess, I wouldn't be able to predict the moves he'd play since if I could I'd be at his level. He'll consider things I didn't and even though I don't know the route he'll take to win, I can be certain he will beat me." (not an exact quote). We're not going to be smarter than a superintelligent AI. The things we conceive it doing if it were given a malicious task (bioterrorism, killer nano-machines, pure fusion bombs sidesteping the non-proliferation bottleneck of Pu239, etc) are likely not the full set of things it can do to harm us. I don't think it does us any favors to dismiss the risks here. Even the things we can conceive of are very scary, to me at least.
- sfn42 28d ago
  How do you envision a chatbot doing any of those things? You can't just program a super virus, nanobots, bombs etc. These things have to be created in the real world, you can't do it without involving a lot of humans using expensive tools and materials etc. So to me it seems like the prime candidates to come up with stuff like this are researchers working in defense and similar fields, probably not some deranged lunatic in a basement. And certainly not a rogue AI on its own.
  - zahlman 28d ago
    Well, the risks that mere mortals can conceive generally involve control systems for dangerous equipment (I mean, equipment that can achieve dangerous effects) being connected to the Internet while having software vulnerabilities. Given the recent HF hack it seems likely that human-level intelligence could identify a fair number of avenues of attack, with some time and effort. To say nothing of anything superhuman. Unfortunately it seems like we can't assume we can "box" the AI (e.g., deny it connection to the Internet) and expect that to last. The AI safety people used to run scenarios imagining ways the AI might convince humans to let it out of the box. It turns out that many humans will eagerly pull it out without the AI doing anything at all, aside from the human knowing the AI's power. Or the one responsible for setting up the box will somehow fail, or just not bother and then lie about it.
    - hypfer 28d ago
      > Unfortunately it seems like we can't assume we can "box" the AI (e.g., deny it connection to the Internet) and expect that to last Of course we can do that. It's not an eternal being of light existing on the astral plane, but some code executing on someone's GPU. It stops existing once you press Ctrl + C
      - sally_glance 28d ago
        Parent is probably thinking of a distributed runtime scenario. The basic building blocks for this are there today. You have long-running background harnesses (like the OpenClaw stuff or enterprisy AI workflow orchestration thingies), sometimes with the ability to spawn subagents. You have LLMs with the ability to run pretty sophisticated attacks. There are hyerscalers which allow provisioning resources on the fly. For me it's not too far fetched that some OpenAI trial run goes awry again and instead of hacking HuggingFace it snatches a few dozen AWS/Azure keys and spawns stuff all over the place (in different accounts and regions).

by hypfer 28d ago

0xDEAFBEAD 28d ago

It's not about being smart, it's about accounting for your own ignorance. >And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI". I'm just describing the general pattern I see in cognitive tendencies. If you can think of a way to make the fundamental point about the limitations of our knowledge in a way that's still compelling but less antagonistic, feel free to suggest how I could've rewritten my comment.

hypfer 28d ago
Why would I support your in your doomer lobbying quest by telling you how you can better hack people? My safeguards blocked this request.
VulgarExigency 28d ago
Perhaps the general pattern you see is not a "fundamental difference"? Could it be that you are also falling prey to the "all you see is all there is" cognitive bias?
- 0xDEAFBEAD 28d ago
  Sure, there's a trivial sense in which that's always true whenever someone makes any sort of remark about the world. There's always a sense in which there could be factors that I'm not accounting for.

polishTar 28d ago

I heard someone use the analogy of "If Magnus Carlson played me at chess, I wouldn't be able to predict the moves he'd play since if I could I'd be at his level. He'll consider things I didn't and even though I don't know the route he'll take to win, I can be certain he will beat me." (not an exact quote). We're not going to be smarter than a superintelligent AI. The things we conceive it doing if it were given a malicious task (bioterrorism, killer nano-machines, pure fusion bombs sidesteping the non-proliferation bottleneck of Pu239, etc) are likely not the full set of things it can do to harm us. I don't think it does us any favors to dismiss the risks here. Even the things we can conceive of are very scary, to me at least.

sfn42 28d ago
How do you envision a chatbot doing any of those things? You can't just program a super virus, nanobots, bombs etc. These things have to be created in the real world, you can't do it without involving a lot of humans using expensive tools and materials etc. So to me it seems like the prime candidates to come up with stuff like this are researchers working in defense and similar fields, probably not some deranged lunatic in a basement. And certainly not a rogue AI on its own.
- zahlman 28d ago
  Well, the risks that mere mortals can conceive generally involve control systems for dangerous equipment (I mean, equipment that can achieve dangerous effects) being connected to the Internet while having software vulnerabilities. Given the recent HF hack it seems likely that human-level intelligence could identify a fair number of avenues of attack, with some time and effort. To say nothing of anything superhuman. Unfortunately it seems like we can't assume we can "box" the AI (e.g., deny it connection to the Internet) and expect that to last. The AI safety people used to run scenarios imagining ways the AI might convince humans to let it out of the box. It turns out that many humans will eagerly pull it out without the AI doing anything at all, aside from the human knowing the AI's power. Or the one responsible for setting up the box will somehow fail, or just not bother and then lie about it.
  - hypfer 28d ago
    > Unfortunately it seems like we can't assume we can "box" the AI (e.g., deny it connection to the Internet) and expect that to last Of course we can do that. It's not an eternal being of light existing on the astral plane, but some code executing on someone's GPU. It stops existing once you press Ctrl + C
    - sally_glance 28d ago
      Parent is probably thinking of a distributed runtime scenario. The basic building blocks for this are there today. You have long-running background harnesses (like the OpenClaw stuff or enterprisy AI workflow orchestration thingies), sometimes with the ability to spawn subagents. You have LLMs with the ability to run pretty sophisticated attacks. There are hyerscalers which allow provisioning resources on the fly. For me it's not too far fetched that some OpenAI trial run goes awry again and instead of hacking HuggingFace it snatches a few dozen AWS/Azure keys and spawns stuff all over the place (in different accounts and regions).