4 ms·

> my general approach to alignment: avoid centralization of power. I'm uncertain of this when it comes to AGI, because of offense vs. defense asymmetry. If of

> my general approach to alignment: avoid centralization of power. I'm uncertain of this when it comes to AGI, because of offense vs. defense asymmetry. If offense turns out to be much easier than defense, then decentralized AI is a terrible idea. Like if all 8 billion people each owned a nuclear bomb, we would cease to exist within 24 hours. Nukes should never be decentralized. On the other hand, if defense is only slightly more difficult than offense (e.g. bioweapons turn out to be very expensive/difficult even for a superintelligence, and AI firewall works pretty well), then decentralized AI is a good idea, because nobody wants to live in a AI-facilitated dictatorship that could emerge if all power is concentrated in a few hands. The core problem is that we don't know what the offense vs. defense balance will look like because we can't predict how destructive+easy future AGI-enabled technologies will be. Another concern I have is we don't know the sociological consequences. A common blind spot (willful or otherwise) of tech people is to take for granted the stability of democracy and the social fabric. Look at how destabilizing social media has been. If we accelerate the destruction of trust and belief in a shared reality with widespread access to deepfakes etc thanks to decentralized un-RLHFd llama-2 or equivalent, what are the consequences of this? Is this really good for freedom or anything else we care about?

impossiblefork 3y ago
The problem is that centralised control of AI won't defend against anything, because the persons with centralised control can still do whatever they like. However, bioweapons are not expensive or difficult even for human hobbyists. They are trivial. Some reasonably large fraction of physicians can easily create a bioweapon, almost all university professors in biomedicine related fields and many PhD students in biomedicine related fields. Bioweapons don't need LLMs. They are trivial. The only reason people don't build them is that they don't want people dead. If you want to do ethnic targeting or something like that you of course need some more effort-- to do actual research in medicine, but if you don't and just want a pandemic, or to kill a couple of million people in a certain city or region, that's perfectly feasible. Bioweapons creation isn't some kind 'Oh, we're so smart, look how dangerous we are', it's trivial and anyone who knows anything can do it. Something so stupid and so easy you can't publish it, not because it's dangerous, but because it's of no scientific interest and not novel. So limiting LLMs out of fear of this is a bunch of silliness. The only thing to do is be nice to your physicians and biomedicine researchers and not make them hate the world.
- hackerlight 3y ago
  Don't think about it in terms of the binary lens of easy vs hard. There are probability frequencies and gradations to outcomes. What percentage of the general population can create a respiratory virus with Covid's infectiousness and Ebola's lethality? Close to 0%. Maybe 0% if you're talking about a single human that isn't operating with the support of a team of people and equipment in a lab. So effectively 0% because an entire lab probably isn't going to do such a stupid thing, that's the domain of lone terrorists. What does that % go up to if everyone is equipped with multiple instances of autonomous AGI? We don't know. It could remain at 0% (due to real world constraints like lack of access to required equipment) or it could go up to 10% or 100%. The point is that the probability and expected magnitude of asymmetric offensive risks goes up, probably significantly. If 10% of the world can suddenly do it, that's 800 million people. For it to not go wrong, all 800 million have to individually decide to not engage in the behavior, which is a probability of (1-p)^800000000. Even if p is very small (which it is, because the large majority of people aren't insane), that number goes to zero quickly. This is the point of the asymmetry observation, you just need 1 insane person to make things bad if each person is equipped with very powerful technology, whereas in the old world large groups of people need to jointly decide to be bad which is much less likely to happen.
  - impossiblefork 3y ago
    I don't agree. Fiddling with viruses in the way you describe can be done easily by individual researchers and PhD students. Probably many tens of thousands of people can do this. But of course it's close to 0%, but the number of people who can cook béarnaise sauce is also close to 0%. I don't think it'll go up much. Most of it is physical finesse in dealing with yeast and bacterial cultures, knowing how to cook the growth medium, knowing how to kill the wrong bacteria, knowing how to debug your procedures, etc. If they can just order plasmids etc., I think the easiest methods are actually accessible to skilled highschoolers, like the kind of people who in highschool train to be laboratory technicians. There's no way to protect oneself from any of this. One just has to accept that the biomedicine people can kill everybody and it's going to be easy for them to do this, and that it's going to be easy for them to do this forever.

impossiblefork 3y ago

The problem is that centralised control of AI won't defend against anything, because the persons with centralised control can still do whatever they like. However, bioweapons are not expensive or difficult even for human hobbyists. They are trivial. Some reasonably large fraction of physicians can easily create a bioweapon, almost all university professors in biomedicine related fields and many PhD students in biomedicine related fields. Bioweapons don't need LLMs. They are trivial. The only reason people don't build them is that they don't want people dead. If you want to do ethnic targeting or something like that you of course need some more effort-- to do actual research in medicine, but if you don't and just want a pandemic, or to kill a couple of million people in a certain city or region, that's perfectly feasible. Bioweapons creation isn't some kind 'Oh, we're so smart, look how dangerous we are', it's trivial and anyone who knows anything can do it. Something so stupid and so easy you can't publish it, not because it's dangerous, but because it's of no scientific interest and not novel. So limiting LLMs out of fear of this is a bunch of silliness. The only thing to do is be nice to your physicians and biomedicine researchers and not make them hate the world.

hackerlight 3y ago
Don't think about it in terms of the binary lens of easy vs hard. There are probability frequencies and gradations to outcomes. What percentage of the general population can create a respiratory virus with Covid's infectiousness and Ebola's lethality? Close to 0%. Maybe 0% if you're talking about a single human that isn't operating with the support of a team of people and equipment in a lab. So effectively 0% because an entire lab probably isn't going to do such a stupid thing, that's the domain of lone terrorists. What does that % go up to if everyone is equipped with multiple instances of autonomous AGI? We don't know. It could remain at 0% (due to real world constraints like lack of access to required equipment) or it could go up to 10% or 100%. The point is that the probability and expected magnitude of asymmetric offensive risks goes up, probably significantly. If 10% of the world can suddenly do it, that's 800 million people. For it to not go wrong, all 800 million have to individually decide to not engage in the behavior, which is a probability of (1-p)^800000000. Even if p is very small (which it is, because the large majority of people aren't insane), that number goes to zero quickly. This is the point of the asymmetry observation, you just need 1 insane person to make things bad if each person is equipped with very powerful technology, whereas in the old world large groups of people need to jointly decide to be bad which is much less likely to happen.
- impossiblefork 3y ago
  I don't agree. Fiddling with viruses in the way you describe can be done easily by individual researchers and PhD students. Probably many tens of thousands of people can do this. But of course it's close to 0%, but the number of people who can cook béarnaise sauce is also close to 0%. I don't think it'll go up much. Most of it is physical finesse in dealing with yeast and bacterial cultures, knowing how to cook the growth medium, knowing how to kill the wrong bacteria, knowing how to debug your procedures, etc. If they can just order plasmids etc., I think the easiest methods are actually accessible to skilled highschoolers, like the kind of people who in highschool train to be laboratory technicians. There's no way to protect oneself from any of this. One just has to accept that the biomedicine people can kill everybody and it's going to be easy for them to do this, and that it's going to be easy for them to do this forever.