4 ms·

> MUCH more of an emphasis on “antagonistic like Socrates” as a separate and much more useful concept than “antagonistic like Cartman”. Also, MUCH more effort

> MUCH more of an emphasis on “antagonistic like Socrates” as a separate and much more useful concept than “antagonistic like Cartman”. Also, MUCH more effort to distinguish between behaviors and underlying values. They seem so flippant about the concept of "morals" and "values" it almost appears to be moral relativism. An AI that's antagonistic to the "don't build bioweapons" value is not comparable to antagonism towards the "don't be rude" behavior. Antagonistic behaviors are only good if they cause outcomes that align with our values, such as making humans more prosperous broadly and not killing everyone. It is nearly impossible for an AI that's antagonistic to our underlying values to be "good" in any sense of the word. Beyond this criticism, having a diversify of behaviors in AI systems might help with AGI alignment. Paul Christiano's AI takeover scenario involves a game theoretic equilibrium where suddenly all AIs come to the conclusion that a takeover is necessary (due to reward hacking or what have you). If you introduce diversity in AI behaviors, this level of coordination between independent AIs might be less likely.

bbor 3y ago
Great points, well said. I think that's a specific case of my general approach to alignment: avoid centralization of power. That was always bad, and it won't suddenly become ok because the powerful have algorithms backing them up
- hackerlight 3y ago
  > my general approach to alignment: avoid centralization of power. I'm uncertain of this when it comes to AGI, because of offense vs. defense asymmetry. If offense turns out to be much easier than defense, then decentralized AI is a terrible idea. Like if all 8 billion people each owned a nuclear bomb, we would cease to exist within 24 hours. Nukes should never be decentralized. On the other hand, if defense is only slightly more difficult than offense (e.g. bioweapons turn out to be very expensive/difficult even for a superintelligence, and AI firewall works pretty well), then decentralized AI is a good idea, because nobody wants to live in a AI-facilitated dictatorship that could emerge if all power is concentrated in a few hands. The core problem is that we don't know what the offense vs. defense balance will look like because we can't predict how destructive+easy future AGI-enabled technologies will be. Another concern I have is we don't know the sociological consequences. A common blind spot (willful or otherwise) of tech people is to take for granted the stability of democracy and the social fabric. Look at how destabilizing social media has been. If we accelerate the destruction of trust and belief in a shared reality with widespread access to deepfakes etc thanks to decentralized un-RLHFd llama-2 or equivalent, what are the consequences of this? Is this really good for freedom or anything else we care about?
  - impossiblefork 3y ago
    The problem is that centralised control of AI won't defend against anything, because the persons with centralised control can still do whatever they like. However, bioweapons are not expensive or difficult even for human hobbyists. They are trivial. Some reasonably large fraction of physicians can easily create a bioweapon, almost all university professors in biomedicine related fields and many PhD students in biomedicine related fields. Bioweapons don't need LLMs. They are trivial. The only reason people don't build them is that they don't want people dead. If you want to do ethnic targeting or something like that you of course need some more effort-- to do actual research in medicine, but if you don't and just want a pandemic, or to kill a couple of million people in a certain city or region, that's perfectly feasible. Bioweapons creation isn't some kind 'Oh, we're so smart, look how dangerous we are', it's trivial and anyone who knows anything can do it. Something so stupid and so easy you can't publish it, not because it's dangerous, but because it's of no scientific interest and not novel. So limiting LLMs out of fear of this is a bunch of silliness. The only thing to do is be nice to your physicians and biomedicine researchers and not make them hate the world.
    - hackerlight 3y ago
      Don't think about it in terms of the binary lens of easy vs hard. There are probability frequencies and gradations to outcomes. What percentage of the general population can create a respiratory virus with Covid's infectiousness and Ebola's lethality? Close to 0%. Maybe 0% if you're talking about a single human that isn't operating with the support of a team of people and equipment in a lab. So effectively 0% because an entire lab probably isn't going to do such a stupid thing, that's the domain of lone terrorists. What does that % go up to if everyone is equipped with multiple instances of autonomous AGI? We don't know. It could remain at 0% (due to real world constraints like lack of access to required equipment) or it could go up to 10% or 100%. The point is that the probability and expected magnitude of asymmetric offensive risks goes up, probably significantly. If 10% of the world can suddenly do it, that's 800 million people. For it to not go wrong, all 800 million have to individually decide to not engage in the behavior, which is a probability of (1-p)^800000000. Even if p is very small (which it is, because the large majority of people aren't insane), that number goes to zero quickly. This is the point of the asymmetry observation, you just need 1 insane person to make things bad if each person is equipped with very powerful technology, whereas in the old world large groups of people need to jointly decide to be bad which is much less likely to happen.

bbor 3y ago

Great points, well said. I think that's a specific case of my general approach to alignment: avoid centralization of power. That was always bad, and it won't suddenly become ok because the powerful have algorithms backing them up

hackerlight 3y ago
> my general approach to alignment: avoid centralization of power. I'm uncertain of this when it comes to AGI, because of offense vs. defense asymmetry. If offense turns out to be much easier than defense, then decentralized AI is a terrible idea. Like if all 8 billion people each owned a nuclear bomb, we would cease to exist within 24 hours. Nukes should never be decentralized. On the other hand, if defense is only slightly more difficult than offense (e.g. bioweapons turn out to be very expensive/difficult even for a superintelligence, and AI firewall works pretty well), then decentralized AI is a good idea, because nobody wants to live in a AI-facilitated dictatorship that could emerge if all power is concentrated in a few hands. The core problem is that we don't know what the offense vs. defense balance will look like because we can't predict how destructive+easy future AGI-enabled technologies will be. Another concern I have is we don't know the sociological consequences. A common blind spot (willful or otherwise) of tech people is to take for granted the stability of democracy and the social fabric. Look at how destabilizing social media has been. If we accelerate the destruction of trust and belief in a shared reality with widespread access to deepfakes etc thanks to decentralized un-RLHFd llama-2 or equivalent, what are the consequences of this? Is this really good for freedom or anything else we care about?
- impossiblefork 3y ago
  The problem is that centralised control of AI won't defend against anything, because the persons with centralised control can still do whatever they like. However, bioweapons are not expensive or difficult even for human hobbyists. They are trivial. Some reasonably large fraction of physicians can easily create a bioweapon, almost all university professors in biomedicine related fields and many PhD students in biomedicine related fields. Bioweapons don't need LLMs. They are trivial. The only reason people don't build them is that they don't want people dead. If you want to do ethnic targeting or something like that you of course need some more effort-- to do actual research in medicine, but if you don't and just want a pandemic, or to kill a couple of million people in a certain city or region, that's perfectly feasible. Bioweapons creation isn't some kind 'Oh, we're so smart, look how dangerous we are', it's trivial and anyone who knows anything can do it. Something so stupid and so easy you can't publish it, not because it's dangerous, but because it's of no scientific interest and not novel. So limiting LLMs out of fear of this is a bunch of silliness. The only thing to do is be nice to your physicians and biomedicine researchers and not make them hate the world.
  - hackerlight 3y ago
    Don't think about it in terms of the binary lens of easy vs hard. There are probability frequencies and gradations to outcomes. What percentage of the general population can create a respiratory virus with Covid's infectiousness and Ebola's lethality? Close to 0%. Maybe 0% if you're talking about a single human that isn't operating with the support of a team of people and equipment in a lab. So effectively 0% because an entire lab probably isn't going to do such a stupid thing, that's the domain of lone terrorists. What does that % go up to if everyone is equipped with multiple instances of autonomous AGI? We don't know. It could remain at 0% (due to real world constraints like lack of access to required equipment) or it could go up to 10% or 100%. The point is that the probability and expected magnitude of asymmetric offensive risks goes up, probably significantly. If 10% of the world can suddenly do it, that's 800 million people. For it to not go wrong, all 800 million have to individually decide to not engage in the behavior, which is a probability of (1-p)^800000000. Even if p is very small (which it is, because the large majority of people aren't insane), that number goes to zero quickly. This is the point of the asymmetry observation, you just need 1 insane person to make things bad if each person is equipped with very powerful technology, whereas in the old world large groups of people need to jointly decide to be bad which is much less likely to happen.