3 ms·

Don't think about it in terms of the binary lens of easy vs hard. There are probability frequencies and gradations to outcomes. What percentage of the general

Don't think about it in terms of the binary lens of easy vs hard. There are probability frequencies and gradations to outcomes. What percentage of the general population can create a respiratory virus with Covid's infectiousness and Ebola's lethality? Close to 0%. Maybe 0% if you're talking about a single human that isn't operating with the support of a team of people and equipment in a lab. So effectively 0% because an entire lab probably isn't going to do such a stupid thing, that's the domain of lone terrorists. What does that % go up to if everyone is equipped with multiple instances of autonomous AGI? We don't know. It could remain at 0% (due to real world constraints like lack of access to required equipment) or it could go up to 10% or 100%. The point is that the probability and expected magnitude of asymmetric offensive risks goes up, probably significantly. If 10% of the world can suddenly do it, that's 800 million people. For it to not go wrong, all 800 million have to individually decide to not engage in the behavior, which is a probability of (1-p)^800000000. Even if p is very small (which it is, because the large majority of people aren't insane), that number goes to zero quickly. This is the point of the asymmetry observation, you just need 1 insane person to make things bad if each person is equipped with very powerful technology, whereas in the old world large groups of people need to jointly decide to be bad which is much less likely to happen.

impossiblefork 3y ago
I don't agree. Fiddling with viruses in the way you describe can be done easily by individual researchers and PhD students. Probably many tens of thousands of people can do this. But of course it's close to 0%, but the number of people who can cook béarnaise sauce is also close to 0%. I don't think it'll go up much. Most of it is physical finesse in dealing with yeast and bacterial cultures, knowing how to cook the growth medium, knowing how to kill the wrong bacteria, knowing how to debug your procedures, etc. If they can just order plasmids etc., I think the easiest methods are actually accessible to skilled highschoolers, like the kind of people who in highschool train to be laboratory technicians. There's no way to protect oneself from any of this. One just has to accept that the biomedicine people can kill everybody and it's going to be easy for them to do this, and that it's going to be easy for them to do this forever.
- bbor 3y ago
  There's no way to protect oneself from any of this. One just has to accept that the biomedicine people can kill everybody and it's going to be easy for them to do this, and that it's going to be easy for them to do this forever. With absolutely no offense intended: this reads like dialogue from the well-meaning second-act villain in a comic book movie. You’re obviously knowledgeable about the specifics, so I’m not saying you’re wrong. I for one am optimistic we can keep biomedical safety at or near current levels
  - impossiblefork 3y ago
    Maybe I'm knowledgeable, but I obviously haven't tried these things that I'm fairly sure are relatively easy, but I can't absolutely say that it's as easy as I have said. It can be fiddly to figure out experimental procedures, and debugging is hard, so it's possible that this really only is something that PhD students and above can do, but I think it's accessible to skilled to hobbyists. But I should say my biomedicine knowledge consists of a special highschool biotechnology program, nothing more. Basically we put plasmids into bacteria and grew them, maybe we did something with yeast, we did DNA amplification and other easy experiments of that sort. This is the extent of my biomedicine knowledge. I think biomedicine people just don't want to kill everybody. The idea of biomedical safety at our current levels, that's what I object to: there isn't any. The safety consists entirely in the biomedicine people being decent. They are self-interested and intelligent, so a society genuinely shits on them, then the risk increases. After all, this stuff is really very easy. But as long as one has democracy, reasonable order, etc., then the fact that they're decent means that there is very little risk. It's the same thing with political assassinations. The politicians aren't alive because of clever guards and security arrangements, they are quite vulnerable in fact, but most people want democracy, civilisation, etc. are willing to accept even bad politicians that they oppose if that's what everybody else wants. But if you think that you're safe because of your guards and security arrangements, then you get the wrong attitude.
- hackerlight 3y ago
  > One just has to accept that the biomedicine people can kill everybody This is falling back into the binary thinking of can vs cannot. I want to re-emphasize probabilities, magnitudes and speeds. If 1 out of 1 billion people are insane enough to do it, but only 20k people can do it, that's (1-1e-9)^20000 = 0.99998 chance that it won't happen. If you invent a technology that puts this capability in the hands of 1 billion people, that probability changes to 0.36789 that it won't happen. Over a sufficient number of generations (N) of people, the probability that it won't happen goes down to 0.36789^N, a very small number. On a long enough time horizon (say, 200 years), we are dead. It's why nuclear proliferation is bad. If only 2 countries have nukes, the chances of a nuclear first strike are (1-p)^2. We probably survive because it probably won't happen. If every country has nukes, it becomes (1-p)^200 (this isn't exactly correct because superpowers have more nukes, and the probability of war between any pair of countries isn't i.i.d. with another pair of countries, among other assumptions, but it's fine for a conceptual first scratch). If every person has nukes, it's (1-p)^8billion, and we are definitely dead. This idea that decentralization is automatically good is quite silly. It is good in many areas. But decentralized nukes or decentralized gain of function virus research is categorically bad. You really have to scrutinize what it is you're making accessible.

impossiblefork 3y ago

I don't agree. Fiddling with viruses in the way you describe can be done easily by individual researchers and PhD students. Probably many tens of thousands of people can do this. But of course it's close to 0%, but the number of people who can cook béarnaise sauce is also close to 0%. I don't think it'll go up much. Most of it is physical finesse in dealing with yeast and bacterial cultures, knowing how to cook the growth medium, knowing how to kill the wrong bacteria, knowing how to debug your procedures, etc. If they can just order plasmids etc., I think the easiest methods are actually accessible to skilled highschoolers, like the kind of people who in highschool train to be laboratory technicians. There's no way to protect oneself from any of this. One just has to accept that the biomedicine people can kill everybody and it's going to be easy for them to do this, and that it's going to be easy for them to do this forever.

bbor 3y ago
There's no way to protect oneself from any of this. One just has to accept that the biomedicine people can kill everybody and it's going to be easy for them to do this, and that it's going to be easy for them to do this forever. With absolutely no offense intended: this reads like dialogue from the well-meaning second-act villain in a comic book movie. You’re obviously knowledgeable about the specifics, so I’m not saying you’re wrong. I for one am optimistic we can keep biomedical safety at or near current levels
- impossiblefork 3y ago
  Maybe I'm knowledgeable, but I obviously haven't tried these things that I'm fairly sure are relatively easy, but I can't absolutely say that it's as easy as I have said. It can be fiddly to figure out experimental procedures, and debugging is hard, so it's possible that this really only is something that PhD students and above can do, but I think it's accessible to skilled to hobbyists. But I should say my biomedicine knowledge consists of a special highschool biotechnology program, nothing more. Basically we put plasmids into bacteria and grew them, maybe we did something with yeast, we did DNA amplification and other easy experiments of that sort. This is the extent of my biomedicine knowledge. I think biomedicine people just don't want to kill everybody. The idea of biomedical safety at our current levels, that's what I object to: there isn't any. The safety consists entirely in the biomedicine people being decent. They are self-interested and intelligent, so a society genuinely shits on them, then the risk increases. After all, this stuff is really very easy. But as long as one has democracy, reasonable order, etc., then the fact that they're decent means that there is very little risk. It's the same thing with political assassinations. The politicians aren't alive because of clever guards and security arrangements, they are quite vulnerable in fact, but most people want democracy, civilisation, etc. are willing to accept even bad politicians that they oppose if that's what everybody else wants. But if you think that you're safe because of your guards and security arrangements, then you get the wrong attitude.
hackerlight 3y ago
> One just has to accept that the biomedicine people can kill everybody This is falling back into the binary thinking of can vs cannot. I want to re-emphasize probabilities, magnitudes and speeds. If 1 out of 1 billion people are insane enough to do it, but only 20k people can do it, that's (1-1e-9)^20000 = 0.99998 chance that it won't happen. If you invent a technology that puts this capability in the hands of 1 billion people, that probability changes to 0.36789 that it won't happen. Over a sufficient number of generations (N) of people, the probability that it won't happen goes down to 0.36789^N, a very small number. On a long enough time horizon (say, 200 years), we are dead. It's why nuclear proliferation is bad. If only 2 countries have nukes, the chances of a nuclear first strike are (1-p)^2. We probably survive because it probably won't happen. If every country has nukes, it becomes (1-p)^200 (this isn't exactly correct because superpowers have more nukes, and the probability of war between any pair of countries isn't i.i.d. with another pair of countries, among other assumptions, but it's fine for a conceptual first scratch). If every person has nukes, it's (1-p)^8billion, and we are definitely dead. This idea that decentralization is automatically good is quite silly. It is good in many areas. But decentralized nukes or decentralized gain of function virus research is categorically bad. You really have to scrutinize what it is you're making accessible.