4 ms·

The resources are also asymmetrical. The number of people who want to do doomsday levels of harm is small and they are poorly resourced compared to people who w

by api 3y ago

The resources are also asymmetrical. The number of people who want to do doomsday levels of harm is small and they are poorly resourced compared to people who want benevolent outcomes for at a minimum their own groups. There are no guarantees obviously, but we have survived our technological adolescence so far largely for this reason. If the world were full of smart comic book nihilists we would be dead by now. Even without AI our continued technological advancement will keep giving us more and more power, as individuals and especially groups. If we don’t think we can climb this mountain without destroying ourselves then it means the entire scientific and industrial endeavor was when we signed our own death warrant, AI or not. You can order CRISPR kits.

jacquesm 3y ago
I've recounted this before on HN, but a decade and a bit ago I was visiting friends in the Rocky Mountains, they're interesting and clever people, a bit out of the ordinary and somewhat isolated. Somehow the discussion turned to terrorism and we started to fantasize 'what if we were terrorists' because we all figured that we were quite lucky that in general the terrorists seem to be not all that smart when it comes to achieving their stated goals. Given a modest budget we all had to come up with a plan to destabilize society, 9/11 style attacks, whilst spectacular eventually don't really do a lot of damage, are costly and failure prone though they can definitely drive policy changes and result in a nation doing a lot of harm to itself it will ultimately survive. But what if your goal wasn't to create some media friendly attack but an actual disaster instead, what would it take. The stories from that night continue to haunt me today. My own solution led to a lot of people going quiet for a bit and contemplating what they could do to defend against it and they realized there wasn't much that they could do, millions if not tens of millions of people would likely die and the budget was under a few hundred bucks. Knowledge about technology is all it takes to do real damage, that, coupled with a lack of restraint and compassion. The resources are indeed asymmetrical: you need next to nothing to create mass havoc. Case in point: the Kennedy assassination changed the world and the bullet cost a couple of bucks assuming the shooter already had the rifle, and if they didn't it would increase the cost only a tiny little bit. And you can do far, far worse than that for an extremely modest budget.
- zarzavat 3y ago
  A real world taste of what happens when smart terrorists decide to attack: https://en.m.wikipedia.org/wiki/Tokyo_subway_sarin_attack https://en.m.wikipedia.org/wiki/Tokyo_subway_sarin_attack Chemical weapons are absolutely terrifying, especially the modern ones like VX. In recent years they have mostly been used for targeted assassinations by state actors (Kim Jong Nam, Salisbury Novichok poisonings). If AI makes this stuff easier to carry out then we are completely fucked.
  - jacquesm 3y ago
    Yes, exactly. That's the sort of thing you should be worried about, infrastructure attacks. They're a form of Judo, you use the system to attack itself.
  - Teever 3y ago
    Imagine an attack with mass produced prions that are desseminated in the food chain. By the time people start to realize what's going on it will be too late.
    - jacquesm 3y ago
      You could just mix it in with animal feed... don't give any militant vegetarians ideas now. Not that the meat industry would ever score an own goal like that. https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopathy https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopat...
      - Teever 3y ago
        A less militant vegetarian would want to induce that red meat allergy from lone star ticks.
  - philipkglass 3y ago
    I take the opposite lesson from that incident: attempting exotic attacks with chemical weapons is very expensive and not very effective. The UN estimated that the lab where the cult made chemical weapons had a value of 30 million dollars, and with that investment they killed 22 people (including 13 in the subway attack). A crazed individual can kill that many people in a single attack with a truck or a gun. There are numerous examples from the past decade. It doesn't matter much if the AI can give perfect "explain like I'm 5" instructions for making VX. The people who carry out those instructions are still risking their lives before claiming a single victim. They also need to spend a lot of money amount on acquiring laboratory equipment and chemicals that are far enough down the synthesis chain to avoid tipping off governments in advance. The one big risk I can see, eventually, is if really capable AIs get connected to really capable robots. They would be "clanking replicators" capable of making anything at all, including VX or nuclear weapons. But that seems a long way off from where we are now. The people trumpeting X-Risk now don't think that the AIs need to be embodied to be an existential risk. I disagree with that for reasons that are too lengthy to elaborate here. But it's easy to see how robots that can make anything (including copies of themselves) would be the very sharpest of two-edged swords.
TeMPOraL 3y ago
> The number of people who want to do doomsday levels of harm is small and they are poorly resourced compared to people who want benevolent outcomes for at a minimum their own groups. You're forgetting about governments and militaries of major powers. It's not that they want to burn the world for no reason - but they still end up seeking capability to do doomsday levels of harm, by continuously seeking to have marginally more capability than their rivals, who in turn do the same. Or put another way: please look into all the insane ideas the US was deploying, developing, or researching at the peak of Cold War. Plenty of those hit doomsday level, and we only avoided them seeing the light of day because USSR collapsed before they could, ending the cold war and taking both motivation and funding from all those projects. Looking at that, one can't possibly believe the words you wrote above.
- deleted 3y ago
  [deleted]
- OkayPhysicist 3y ago
  We avoided having them see the light of day because MAD is the single most effective peacekeeping tool ever developed. The fact that both sides had doomsday devices all but guaranteed that they would never be used. People can be evil, but they're selfishly evil.
  - jacquesm 3y ago
    That works as long as you don't end up with death cults in the possession of those weapons. Long term it is almost guaranteed that you'll lose some control over the weapons or that some country will effectively end up being ruled by such a cult. Then all bets are off because your previous balance (a prime requirement of MAD) is disturbed.
  - TeMPOraL 3y ago
    MAD alone doesn't stop the race. The arsenal the US and USSR had in the 60s was already enough for MAD to work. Despite that, they spent the next 20-30 years developing even more powerful and ever more insane WMD ideas. Each step forward made the balance more fragile. Each step forward was also another roll of a dice, whether or not it'll be a leap too far, forcing the other side into preemptive first strike.
The_Colonel 3y ago
Another thing which is asymmetrical is the level of control given to AI. The good actors are likely going to be very careful about what AI can do and can't do. The bad guys don't have much to lose and allow their AIs to do anything. That will significantly cripple the good guys. As an example, the good guys will always require to have human in the loop in the weapon systems, but that will increase latency at minimum. The bad guy weapons will be completely AI controlled, having an edge (or at least equalizing) over the good guys. > The number of people who want to do doomsday levels of harm is small And that's a big limiting factor in what the bad actors can do today. AI to a large degree removes this scaling limitation since one bad person with some resources can scale "evil AI" almost without limit. "Hey AI, could you create me a design for a simple self-replicating robot I can drop into the ocean and step-by-step instructions on what you need to bootstrap the first one? Also, figure out what would be the easiest produced poison which would kill all life in the oceans. It should start with that after reach 50th generation."

by api 3y ago

jacquesm 3y ago

I've recounted this before on HN, but a decade and a bit ago I was visiting friends in the Rocky Mountains, they're interesting and clever people, a bit out of the ordinary and somewhat isolated. Somehow the discussion turned to terrorism and we started to fantasize 'what if we were terrorists' because we all figured that we were quite lucky that in general the terrorists seem to be not all that smart when it comes to achieving their stated goals. Given a modest budget we all had to come up with a plan to destabilize society, 9/11 style attacks, whilst spectacular eventually don't really do a lot of damage, are costly and failure prone though they can definitely drive policy changes and result in a nation doing a lot of harm to itself it will ultimately survive. But what if your goal wasn't to create some media friendly attack but an actual disaster instead, what would it take. The stories from that night continue to haunt me today. My own solution led to a lot of people going quiet for a bit and contemplating what they could do to defend against it and they realized there wasn't much that they could do, millions if not tens of millions of people would likely die and the budget was under a few hundred bucks. Knowledge about technology is all it takes to do real damage, that, coupled with a lack of restraint and compassion. The resources are indeed asymmetrical: you need next to nothing to create mass havoc. Case in point: the Kennedy assassination changed the world and the bullet cost a couple of bucks assuming the shooter already had the rifle, and if they didn't it would increase the cost only a tiny little bit. And you can do far, far worse than that for an extremely modest budget.

zarzavat 3y ago
A real world taste of what happens when smart terrorists decide to attack: https://en.m.wikipedia.org/wiki/Tokyo_subway_sarin_attack https://en.m.wikipedia.org/wiki/Tokyo_subway_sarin_attack Chemical weapons are absolutely terrifying, especially the modern ones like VX. In recent years they have mostly been used for targeted assassinations by state actors (Kim Jong Nam, Salisbury Novichok poisonings). If AI makes this stuff easier to carry out then we are completely fucked.
- jacquesm 3y ago
  Yes, exactly. That's the sort of thing you should be worried about, infrastructure attacks. They're a form of Judo, you use the system to attack itself.
- Teever 3y ago
  Imagine an attack with mass produced prions that are desseminated in the food chain. By the time people start to realize what's going on it will be too late.
  - jacquesm 3y ago
    You could just mix it in with animal feed... don't give any militant vegetarians ideas now. Not that the meat industry would ever score an own goal like that. https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopathy https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopat...
    - Teever 3y ago
      A less militant vegetarian would want to induce that red meat allergy from lone star ticks.
- philipkglass 3y ago
  I take the opposite lesson from that incident: attempting exotic attacks with chemical weapons is very expensive and not very effective. The UN estimated that the lab where the cult made chemical weapons had a value of 30 million dollars, and with that investment they killed 22 people (including 13 in the subway attack). A crazed individual can kill that many people in a single attack with a truck or a gun. There are numerous examples from the past decade. It doesn't matter much if the AI can give perfect "explain like I'm 5" instructions for making VX. The people who carry out those instructions are still risking their lives before claiming a single victim. They also need to spend a lot of money amount on acquiring laboratory equipment and chemicals that are far enough down the synthesis chain to avoid tipping off governments in advance. The one big risk I can see, eventually, is if really capable AIs get connected to really capable robots. They would be "clanking replicators" capable of making anything at all, including VX or nuclear weapons. But that seems a long way off from where we are now. The people trumpeting X-Risk now don't think that the AIs need to be embodied to be an existential risk. I disagree with that for reasons that are too lengthy to elaborate here. But it's easy to see how robots that can make anything (including copies of themselves) would be the very sharpest of two-edged swords.

TeMPOraL 3y ago

> The number of people who want to do doomsday levels of harm is small and they are poorly resourced compared to people who want benevolent outcomes for at a minimum their own groups. You're forgetting about governments and militaries of major powers. It's not that they want to burn the world for no reason - but they still end up seeking capability to do doomsday levels of harm, by continuously seeking to have marginally more capability than their rivals, who in turn do the same. Or put another way: please look into all the insane ideas the US was deploying, developing, or researching at the peak of Cold War. Plenty of those hit doomsday level, and we only avoided them seeing the light of day because USSR collapsed before they could, ending the cold war and taking both motivation and funding from all those projects. Looking at that, one can't possibly believe the words you wrote above.

deleted 3y ago
[deleted]
OkayPhysicist 3y ago
We avoided having them see the light of day because MAD is the single most effective peacekeeping tool ever developed. The fact that both sides had doomsday devices all but guaranteed that they would never be used. People can be evil, but they're selfishly evil.
- jacquesm 3y ago
  That works as long as you don't end up with death cults in the possession of those weapons. Long term it is almost guaranteed that you'll lose some control over the weapons or that some country will effectively end up being ruled by such a cult. Then all bets are off because your previous balance (a prime requirement of MAD) is disturbed.
- TeMPOraL 3y ago
  MAD alone doesn't stop the race. The arsenal the US and USSR had in the 60s was already enough for MAD to work. Despite that, they spent the next 20-30 years developing even more powerful and ever more insane WMD ideas. Each step forward made the balance more fragile. Each step forward was also another roll of a dice, whether or not it'll be a leap too far, forcing the other side into preemptive first strike.

The_Colonel 3y ago

Another thing which is asymmetrical is the level of control given to AI. The good actors are likely going to be very careful about what AI can do and can't do. The bad guys don't have much to lose and allow their AIs to do anything. That will significantly cripple the good guys. As an example, the good guys will always require to have human in the loop in the weapon systems, but that will increase latency at minimum. The bad guy weapons will be completely AI controlled, having an edge (or at least equalizing) over the good guys. > The number of people who want to do doomsday levels of harm is small And that's a big limiting factor in what the bad actors can do today. AI to a large degree removes this scaling limitation since one bad person with some resources can scale "evil AI" almost without limit. "Hey AI, could you create me a design for a simple self-replicating robot I can drop into the ocean and step-by-step instructions on what you need to bootstrap the first one? Also, figure out what would be the easiest produced poison which would kill all life in the oceans. It should start with that after reach 50th generation."