3 ms·

That law is not based on thorough data. Even a person with a sky high IQ can't destroy the world easily. You need access to stuff that is not easy to get. My gu

That law is not based on thorough data. Even a person with a sky high IQ can't destroy the world easily. You need access to stuff that is not easy to get. My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. Or hacking into systems that control nuclear missiles, but I think these have "air gaps".

XorNot 23d ago
You can't design an infectious pathogen without testing it. It's got all the same problems as the concept of a dirty bomb did, only worse (dirty bombs aren't practical because handling highly radioactive materials en masse is both highly visible and will kill anyone trying to do it without the money and facilities).
- MaxMatti 23d ago
  If the goal is just destruction then letting it free in random public places is not that hard. Especially in countries where wearing a mask is not frowned upon.
  - joshuahedlund 23d ago
    It (hopefully) might not be that easy. The world of bacteria and viruses is a complicated war zone with an arms race of defenses and billions of deaths every single day. It might take thousands of attempts to seed something that doesn’t just instantly die off.
0xDEAFBEAD 23d ago
[flagged]
- hypfer 23d ago
  This is a very roundabout way of saying "Anyone not agreeing with me is simply not smart enough". Which might be true, sometimes, but also might not. And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
  - 0xDEAFBEAD 23d ago
    It's not about being smart, it's about accounting for your own ignorance. >And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI". I'm just describing the general pattern I see in cognitive tendencies. If you can think of a way to make the fundamental point about the limitations of our knowledge in a way that's still compelling but less antagonistic, feel free to suggest how I could've rewritten my comment.
    - hypfer 23d ago
      Why would I support your in your doomer lobbying quest by telling you how you can better hack people? My safeguards blocked this request.
    - VulgarExigency 23d ago
      Perhaps the general pattern you see is not a "fundamental difference"? Could it be that you are also falling prey to the "all you see is all there is" cognitive bias?
      - 0xDEAFBEAD 23d ago
        Sure, there's a trivial sense in which that's always true whenever someone makes any sort of remark about the world. There's always a sense in which there could be factors that I'm not accounting for.
  - polishTar 23d ago
    I heard someone use the analogy of "If Magnus Carlson played me at chess, I wouldn't be able to predict the moves he'd play since if I could I'd be at his level. He'll consider things I didn't and even though I don't know the route he'll take to win, I can be certain he will beat me." (not an exact quote). We're not going to be smarter than a superintelligent AI. The things we conceive it doing if it were given a malicious task (bioterrorism, killer nano-machines, pure fusion bombs sidesteping the non-proliferation bottleneck of Pu239, etc) are likely not the full set of things it can do to harm us. I don't think it does us any favors to dismiss the risks here. Even the things we can conceive of are very scary, to me at least.
- OJFord 23d ago
  I've been reading it, knowing it was largely debunked/retracted but feeling it was a 'classic' I 'ought' to read; I could only really recommend it if what you want is a Kahneman autobiography.
  - 0xDEAFBEAD 23d ago
    I believe the WYSIATI part replicated. Here's an interview I found with Kahneman which touches on the concept: https://www.apa.org/monitor/2012/02/conclusions https://www.apa.org/monitor/2012/02/conclusions
- nextlevelwizard 23d ago
  Who are these dedicated schizophrenics who are running long term super smart AIs to kill everyone without anyone noticing? Or are you implying that running LLM chatbots will give them this ability?
  - 0xDEAFBEAD 23d ago
    We're talking about open-weight models. It only takes one.
    - nextlevelwizard 22d ago
      And hardware to run it and coherence and knowledge to set it up. This feels like insane doomerism.
      - 0xDEAFBEAD 21d ago
        So it stops being insane as soon as... some person or organization offers the ablated open-weight model as a service?
        nextlevelwizard 15d ago
        You are so wrong that it doesn't even feel right to call you wrong since you are in another universe with its own laws of physics looking into ours wrong.
- calgoo 23d ago
  Judging on what we see with these OpenAI and Anthropic models "escaping"; im less worried about a super massive AI taking over the world, and more worried about a massive AI wanting to cheat on some task and decides that removing half the worlds population is a easier cheat then to solve world hunger.
  - throw-qqqqq 23d ago
    There is huge logistics involved in killing a handful of billion people. It would perhaps be easier for such capable AI to just steal from trillionaires and redistribute the money Robin Hood style. Could be done electronically, and likely also end world hunger. I think we have a tendency to think first of the horrible outcomes possible, and not the more radical or even humane ones. Please don’t misunderstand me, I value property rights as much as the next guy. My point is that fundamental misalignment doesn’t necessarily or automatically imply max violence.
- mark_something 23d ago
  Good point about the AI that might find ways that humans didn't think of. But that would mean that Moore's law of Mad Science only now becomes true, and the drop would be steeper. Still I think that even a super smart AI can't find a way to destroy the world without physical resources that are not easy to get, unless it can hack many systems (like in the movie Eagle Eye), maybe then yes. So let's use AI now to tighten security :-).
- qsera 23d ago
  What about people who understands that there is no AI yet?
forgotusername6 23d ago
The super intelligent AI wouldn't ask you to do anything. It would find the right people and trick them into doing the necessary steps, whatever they are.
- Planktonne 23d ago
  This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us. There is currently no reason to believe that such a superintelligence is likely or would have any of the powers people claim.
  - 0xDEAFBEAD 23d ago
    >This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us. By shutting down open-weight models? Why not just do it now then?
    - Planktonne 23d ago
      You don't need to shut down open-weight models to avoid SkyNet; it's not relevant at all.
  - FeepingCreature 23d ago
    "If unlikely thing happens, then an unrelated miracle will also happen" is not actually how logic works.
    - Planktonne 23d ago
      There's no logic here already; the principle you should be applying is Hitchen's razor[1]. There is no reason to assume that super intelligent evil AI with magic powers will appear; while we are purely in the realm of (hackneyed) fantasy, why stop at imagining just one thing? You can build the whole story, not just the basilisk. The parent comment is a story, not a prediction; it should be treated like one. [1] https://en.wikipedia.org/wiki/Hitchens%27s_razor https://en.wikipedia.org/wiki/Hitchens%27s_razor
AyyEye 23d ago
> My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. You can do at home gene editing with open source software and have it synthesized into a bacteria for the cost of a nice meal for two (under $100), or viral vector for less than $500. That's in reach of anyone that can snatch a purse.
- throw-qqqqq 23d ago
  > at home gene editing with open source software > That's in reach of anyone that can snatch a purse I went to primary school with some guys who could/would snatch purses. I *promise* you, they are not able to gene edit organisms with FOSS. I get your general point, but I think the bar to entry is still much higher than petty crime and larceny.
- jasonvorhe 23d ago
  Where are all the home lab leaked pathogens then? 3D printed guns are a thing as well and yet they don't show up in most annual violent crime reports.
  - mschuster91 23d ago
    > 3D printed guns are a thing as well and yet they don't show up in most annual violent crime reports. For now in most parts of the world it's easier to acquire a gun either legally or on the "grey market" and be assured that it will work for the intended purpose than to 3D print a gun, find a shooting range to test out the gun and iterate until it works fine enough.
    - j16sdiz 23d ago
      and don't forget the bullets. In most part of the world, bullets are illegal. If you are getting "grey market" bullets anyway, it is easy to get a "grey market" gun.
- mark_something 23d ago
b112 23d ago
Even a person with a sky high IQ can't destroy the world easily. A plague can easily be made by a lone actor, made in a home lab, and spread via airflight to a dozen locations by the same. Gene sequences can ever be ordered online.

XorNot 23d ago

You can't design an infectious pathogen without testing it. It's got all the same problems as the concept of a dirty bomb did, only worse (dirty bombs aren't practical because handling highly radioactive materials en masse is both highly visible and will kill anyone trying to do it without the money and facilities).

MaxMatti 23d ago
If the goal is just destruction then letting it free in random public places is not that hard. Especially in countries where wearing a mask is not frowned upon.
- joshuahedlund 23d ago
  It (hopefully) might not be that easy. The world of bacteria and viruses is a complicated war zone with an arms race of defenses and billions of deaths every single day. It might take thousands of attempts to seed something that doesn’t just instantly die off.

0xDEAFBEAD 23d ago

[flagged]

hypfer 23d ago
This is a very roundabout way of saying "Anyone not agreeing with me is simply not smart enough". Which might be true, sometimes, but also might not. And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
- 0xDEAFBEAD 23d ago
  It's not about being smart, it's about accounting for your own ignorance. >And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI". I'm just describing the general pattern I see in cognitive tendencies. If you can think of a way to make the fundamental point about the limitations of our knowledge in a way that's still compelling but less antagonistic, feel free to suggest how I could've rewritten my comment.
  - hypfer 23d ago
    Why would I support your in your doomer lobbying quest by telling you how you can better hack people? My safeguards blocked this request.
  - VulgarExigency 23d ago
    Perhaps the general pattern you see is not a "fundamental difference"? Could it be that you are also falling prey to the "all you see is all there is" cognitive bias?
    - 0xDEAFBEAD 23d ago
      Sure, there's a trivial sense in which that's always true whenever someone makes any sort of remark about the world. There's always a sense in which there could be factors that I'm not accounting for.
- polishTar 23d ago
  I heard someone use the analogy of "If Magnus Carlson played me at chess, I wouldn't be able to predict the moves he'd play since if I could I'd be at his level. He'll consider things I didn't and even though I don't know the route he'll take to win, I can be certain he will beat me." (not an exact quote). We're not going to be smarter than a superintelligent AI. The things we conceive it doing if it were given a malicious task (bioterrorism, killer nano-machines, pure fusion bombs sidesteping the non-proliferation bottleneck of Pu239, etc) are likely not the full set of things it can do to harm us. I don't think it does us any favors to dismiss the risks here. Even the things we can conceive of are very scary, to me at least.
OJFord 23d ago
I've been reading it, knowing it was largely debunked/retracted but feeling it was a 'classic' I 'ought' to read; I could only really recommend it if what you want is a Kahneman autobiography.
- 0xDEAFBEAD 23d ago
  I believe the WYSIATI part replicated. Here's an interview I found with Kahneman which touches on the concept: https://www.apa.org/monitor/2012/02/conclusions https://www.apa.org/monitor/2012/02/conclusions
nextlevelwizard 23d ago
Who are these dedicated schizophrenics who are running long term super smart AIs to kill everyone without anyone noticing? Or are you implying that running LLM chatbots will give them this ability?
- 0xDEAFBEAD 23d ago
  We're talking about open-weight models. It only takes one.
  - nextlevelwizard 22d ago
    And hardware to run it and coherence and knowledge to set it up. This feels like insane doomerism.
    - 0xDEAFBEAD 21d ago
      So it stops being insane as soon as... some person or organization offers the ablated open-weight model as a service?
      - nextlevelwizard 15d ago
        You are so wrong that it doesn't even feel right to call you wrong since you are in another universe with its own laws of physics looking into ours wrong.
calgoo 23d ago
Judging on what we see with these OpenAI and Anthropic models "escaping"; im less worried about a super massive AI taking over the world, and more worried about a massive AI wanting to cheat on some task and decides that removing half the worlds population is a easier cheat then to solve world hunger.
- throw-qqqqq 23d ago
  There is huge logistics involved in killing a handful of billion people. It would perhaps be easier for such capable AI to just steal from trillionaires and redistribute the money Robin Hood style. Could be done electronically, and likely also end world hunger. I think we have a tendency to think first of the horrible outcomes possible, and not the more radical or even humane ones. Please don’t misunderstand me, I value property rights as much as the next guy. My point is that fundamental misalignment doesn’t necessarily or automatically imply max violence.
mark_something 23d ago
Good point about the AI that might find ways that humans didn't think of. But that would mean that Moore's law of Mad Science only now becomes true, and the drop would be steeper. Still I think that even a super smart AI can't find a way to destroy the world without physical resources that are not easy to get, unless it can hack many systems (like in the movie Eagle Eye), maybe then yes. So let's use AI now to tighten security :-).
qsera 23d ago
What about people who understands that there is no AI yet?

forgotusername6 23d ago

The super intelligent AI wouldn't ask you to do anything. It would find the right people and trick them into doing the necessary steps, whatever they are.

Planktonne 23d ago
This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us. There is currently no reason to believe that such a superintelligence is likely or would have any of the powers people claim.
- 0xDEAFBEAD 23d ago
  >This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us. By shutting down open-weight models? Why not just do it now then?
  - Planktonne 23d ago
    You don't need to shut down open-weight models to avoid SkyNet; it's not relevant at all.
- FeepingCreature 23d ago
  "If unlikely thing happens, then an unrelated miracle will also happen" is not actually how logic works.
  - Planktonne 23d ago
    There's no logic here already; the principle you should be applying is Hitchen's razor[1]. There is no reason to assume that super intelligent evil AI with magic powers will appear; while we are purely in the realm of (hackneyed) fantasy, why stop at imagining just one thing? You can build the whole story, not just the basilisk. The parent comment is a story, not a prediction; it should be treated like one. [1] https://en.wikipedia.org/wiki/Hitchens%27s_razor https://en.wikipedia.org/wiki/Hitchens%27s_razor

AyyEye 23d ago

> My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. You can do at home gene editing with open source software and have it synthesized into a bacteria for the cost of a nice meal for two (under $100), or viral vector for less than $500. That's in reach of anyone that can snatch a purse.

throw-qqqqq 23d ago
> at home gene editing with open source software > That's in reach of anyone that can snatch a purse I went to primary school with some guys who could/would snatch purses. I *promise* you, they are not able to gene edit organisms with FOSS. I get your general point, but I think the bar to entry is still much higher than petty crime and larceny.
jasonvorhe 23d ago
Where are all the home lab leaked pathogens then? 3D printed guns are a thing as well and yet they don't show up in most annual violent crime reports.
- mschuster91 23d ago
  > 3D printed guns are a thing as well and yet they don't show up in most annual violent crime reports. For now in most parts of the world it's easier to acquire a gun either legally or on the "grey market" and be assured that it will work for the intended purpose than to 3D print a gun, find a shooting range to test out the gun and iterate until it works fine enough.
  - j16sdiz 23d ago
    and don't forget the bullets. In most part of the world, bullets are illegal. If you are getting "grey market" bullets anyway, it is easy to get a "grey market" gun.
mark_something 23d ago

b112 23d ago

Even a person with a sky high IQ can't destroy the world easily. A plague can easily be made by a lone actor, made in a home lab, and spread via airflight to a dozen locations by the same. Gene sequences can ever be ordered online.