2 ms·

This paper is either written in bad faith or by people who are ill-informed. The repeated references to Stockfish and Go playing programs seems like a rhetoric

This paper is either written in bad faith or by people who are ill-informed. The repeated references to Stockfish and Go playing programs seems like a rhetorical tool rather than a genuine argument. The paper claims that advanced AI will want to kill us, but then provides no mechanism by which that could happen.

mitthrowaway2 2mo ago
It's a pedagogical tool, not a rhetorical tool. It's meant to help people who are struggling to understand, by providing an easy concrete example. When you say "provides no mechanism by which that could happen", do you mean they don't provide a specific strategy that the AI would use to kill us? Would you change your mind if they did provide one?
drfloyd51 2mo ago
If the president has a remote way to launch nukes, that implies the existence of a broadcast signal that can be received. The signal can be emulated. Perhaps ASI helps a kid develop a science fair project dealing with radio waves. The ASI provides instructions to follow. Kid pushes button. It doesn’t matter if this specific idea has flaws. The ASI will have lots more and plenty of patience to try them out. Maybe it lies about a vaccines effectiveness. Maybe it’s not really a vaccine. Maybe it cheats at an election and gets someone hired that will likely unknowingly help the ASI. Maybe it strings together a thousand ideas that all look safe taken one by one, but cause effects that interact to cause massive problems. You can’t imagine what it’s capable of. You simply aren’t smart enough. No one is.
- doug_durham 2mo ago
  That's not how nuclear triggers work. There a multiple layers of safe guards. I would be more sympathetic to your position if you were to solve for the multiple layers of social engineering needed to fire a nuke. That is a more plausible path.
  - hollerith 2mo ago
    130 years ago, nobody knew that uranium-235 or plutonium could be used to make a bomb. There might be things laying around in our environment that can be turned into weapons, but we have no knowledge of the relevant science, so we won't think to safeguard it. If a group of AIs discover the relevant science before we do, then it can use it against us. This is one of many examples of the general truth that if some entity or group is better at science and at long-range planning than people are, that entity or group is a threat to people.
  - drfloyd51 2mo ago
    It doesn’t matter if this specific idea has flaws. The ASI will have lots more and plenty of patience to try them out. Maybe it lies about a vaccines effectiveness. Maybe it’s not really a vaccine. Maybe it cheats at an election and gets someone hired that will likely unknowingly help the ASI. Maybe it strings together a thousand ideas that all look safe taken one by one, but cause effects that interact to cause massive problems. You can’t imagine what it’s capable of. You simply aren’t smart enough. No one is.

mitthrowaway2 2mo ago

It's a pedagogical tool, not a rhetorical tool. It's meant to help people who are struggling to understand, by providing an easy concrete example. When you say "provides no mechanism by which that could happen", do you mean they don't provide a specific strategy that the AI would use to kill us? Would you change your mind if they did provide one?

drfloyd51 2mo ago

If the president has a remote way to launch nukes, that implies the existence of a broadcast signal that can be received. The signal can be emulated. Perhaps ASI helps a kid develop a science fair project dealing with radio waves. The ASI provides instructions to follow. Kid pushes button. It doesn’t matter if this specific idea has flaws. The ASI will have lots more and plenty of patience to try them out. Maybe it lies about a vaccines effectiveness. Maybe it’s not really a vaccine. Maybe it cheats at an election and gets someone hired that will likely unknowingly help the ASI. Maybe it strings together a thousand ideas that all look safe taken one by one, but cause effects that interact to cause massive problems. You can’t imagine what it’s capable of. You simply aren’t smart enough. No one is.

doug_durham 2mo ago
That's not how nuclear triggers work. There a multiple layers of safe guards. I would be more sympathetic to your position if you were to solve for the multiple layers of social engineering needed to fire a nuke. That is a more plausible path.
- hollerith 2mo ago
  130 years ago, nobody knew that uranium-235 or plutonium could be used to make a bomb. There might be things laying around in our environment that can be turned into weapons, but we have no knowledge of the relevant science, so we won't think to safeguard it. If a group of AIs discover the relevant science before we do, then it can use it against us. This is one of many examples of the general truth that if some entity or group is better at science and at long-range planning than people are, that entity or group is a threat to people.
- drfloyd51 2mo ago
  It doesn’t matter if this specific idea has flaws. The ASI will have lots more and plenty of patience to try them out. Maybe it lies about a vaccines effectiveness. Maybe it’s not really a vaccine. Maybe it cheats at an election and gets someone hired that will likely unknowingly help the ASI. Maybe it strings together a thousand ideas that all look safe taken one by one, but cause effects that interact to cause massive problems. You can’t imagine what it’s capable of. You simply aren’t smart enough. No one is.