4 ms·

> To "avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power" what does one do with an idea or line of research that could potentially

by dsacco 9y ago

> To "avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power" what does one do with an idea or line of research that could potentially harm humanity or unduly concentrate power? Pursue the research and publish its results freely and in their entirety.

rspeer 9y ago
I don't think you answered 3pt14159's question. What if you publish the research and nobody who can do anything about it cares?
- dsacco 9y ago
  Then what exactly was the new bad thing introduced by publishing it in the first place?
Smaug123 9y ago
Are you saying that if it were discovered to be both cheap and easy to create a super-pandemic using nothing more than household chemicals and a swab of your own snot, then you'd publish the results freely and in their entirety? If not, how catastrophic a level of harm are you willing to risk before you start advocating concealing the results?
- sfsylvester 9y ago
  This is the very same concern cyber-security researchers face every time they notice a vulnerability. Do they publish their work openly and potentially leave all those using the service vulnerable to attack? Yes, for a combination of two reasons: 1) They have obvious incentives to publish their results. 2) The more we know about vulnerabilities, the better we are at combatting ourselves against it. That second reason is why we are (overall) better off from acknowledging possible threats to our existence. In your example, the more open the study of super-pandemics is, the more open the study of combatting against super-pandemics is. The more we are aware of a threat, the better informed we are to prevent it. Yes, in your example, the threat could be highly dangerous and imminent. However, if only one individual was capable of creating a super-pandemic, the number of people who could potentially help stop it is drastically reduced. Rather than a free distribution of results which arms our society completely with the ability of preventing it.
  - Smaug123 9y ago
    "Only one individual capable of creating a super-pandemic": by hypothesis, this individual is morally good in some way. (Otherwise they wouldn't be having this internal debate about whether or not to release the information about how to produce the virus: they'd just release the virus!) The ideal scenario is that throughout the rest of time, no entity ever manufactures the virus; the second-best scenario is that an entity does manufacture the virus, but only after there is some hypothetical defence against it. You are shattering forever the dream of achieving the ideal scenario (again by hypothesis, the virus is easy to make, and history demonstrates that if something is easy and destructive, eventually someone will do it); you're pinning your hopes on the second-best scenario. You are therefore implicitly assuming that "we get better at arming ourselves against the virus" happens faster than "enemies create the virus", which is by no means a given and must be weighed on a case-by-case basis. By the way, this is the basic reason why I'm so sad about the existence of fully 3D-printed guns. There is essentially no defence against guns. The creators of the 3D-printed gun chose to spread the blueprints far and wide, to accelerate their advent. It's sad that human nature is such that this was predictable and inevitable.
- dsacco 9y ago
  > Are you saying that if it were discovered to be both cheap and easy to create a super-pandemic using nothing more than household chemicals and a swab of your own snot, then you'd publish the results freely and in their entirety? No. > If not, how catastrophic a level of harm are you willing to risk before you start advocating concealing the results? I'm not sure. Probably quite catastrophic, because it stretches the limits of credulity to seriously talk about a what if scenario where any given individual can easily create a weapon of mass destruction in their kitchen. A better question is why you think such a straightforwardly accessible method of eradicating our species won't be discovered independently despite your efforts to conceal it. How do we navigate that philosophical labyrinth? But more pointedly, I dispute that this is comparable to any specific, credible example of strong AI. Give me a credible scenario that brings us from strong AI to the annihilation of our species without handwaving about recursive self-improvement and selective idiocy and I'll reconsider my position. I'm not going to give up the chance to publish incredibly novel research because some other people like to work themselves into hysterics about a conceptually incoherent boogeyman with "AI" slapped onto it.
  - edanm 9y ago
    > Give me a credible scenario that brings us from strong AI to the annihilation of our species without handwaving about recursive self-improvement and selective idiocy and I'll reconsider my position. What kind of credible scenario are you looking for? You're obviously knowledgeable about the subject, I'm sure you've read various stories, but it's quite easy to dismiss them all as "not credible" or "just so stories". E.g. A good story is Tegmark's story at the beginning of the book Life 3.0, but again, can be easily dismissed if you don't specify acceptance criteria for the story. I mean, simple scenario - AGI gets built by a farming company, gets programmed to farm more corn, but there is no stop condition programmed in, so converts ever available land into corn fields. Is it likely? No, of course not - no specific example is likely. But why is this something that fundamentally can't happen? More importantly, can you specify criteria for a scenario which you would deem acceptable? Cause if not, you're literally saying that there is no possible way to convince you that this is a possibility, which seems to me not to be a good position on pretty much anything.

by dsacco 9y ago

rspeer 9y ago

I don't think you answered 3pt14159's question. What if you publish the research and nobody who can do anything about it cares?

dsacco 9y ago
Then what exactly was the new bad thing introduced by publishing it in the first place?

Smaug123 9y ago

Are you saying that if it were discovered to be both cheap and easy to create a super-pandemic using nothing more than household chemicals and a swab of your own snot, then you'd publish the results freely and in their entirety? If not, how catastrophic a level of harm are you willing to risk before you start advocating concealing the results?

sfsylvester 9y ago
This is the very same concern cyber-security researchers face every time they notice a vulnerability. Do they publish their work openly and potentially leave all those using the service vulnerable to attack? Yes, for a combination of two reasons: 1) They have obvious incentives to publish their results. 2) The more we know about vulnerabilities, the better we are at combatting ourselves against it. That second reason is why we are (overall) better off from acknowledging possible threats to our existence. In your example, the more open the study of super-pandemics is, the more open the study of combatting against super-pandemics is. The more we are aware of a threat, the better informed we are to prevent it. Yes, in your example, the threat could be highly dangerous and imminent. However, if only one individual was capable of creating a super-pandemic, the number of people who could potentially help stop it is drastically reduced. Rather than a free distribution of results which arms our society completely with the ability of preventing it.
- Smaug123 9y ago
  "Only one individual capable of creating a super-pandemic": by hypothesis, this individual is morally good in some way. (Otherwise they wouldn't be having this internal debate about whether or not to release the information about how to produce the virus: they'd just release the virus!) The ideal scenario is that throughout the rest of time, no entity ever manufactures the virus; the second-best scenario is that an entity does manufacture the virus, but only after there is some hypothetical defence against it. You are shattering forever the dream of achieving the ideal scenario (again by hypothesis, the virus is easy to make, and history demonstrates that if something is easy and destructive, eventually someone will do it); you're pinning your hopes on the second-best scenario. You are therefore implicitly assuming that "we get better at arming ourselves against the virus" happens faster than "enemies create the virus", which is by no means a given and must be weighed on a case-by-case basis. By the way, this is the basic reason why I'm so sad about the existence of fully 3D-printed guns. There is essentially no defence against guns. The creators of the 3D-printed gun chose to spread the blueprints far and wide, to accelerate their advent. It's sad that human nature is such that this was predictable and inevitable.
dsacco 9y ago
> Are you saying that if it were discovered to be both cheap and easy to create a super-pandemic using nothing more than household chemicals and a swab of your own snot, then you'd publish the results freely and in their entirety? No. > If not, how catastrophic a level of harm are you willing to risk before you start advocating concealing the results? I'm not sure. Probably quite catastrophic, because it stretches the limits of credulity to seriously talk about a what if scenario where any given individual can easily create a weapon of mass destruction in their kitchen. A better question is why you think such a straightforwardly accessible method of eradicating our species won't be discovered independently despite your efforts to conceal it. How do we navigate that philosophical labyrinth? But more pointedly, I dispute that this is comparable to any specific, credible example of strong AI. Give me a credible scenario that brings us from strong AI to the annihilation of our species without handwaving about recursive self-improvement and selective idiocy and I'll reconsider my position. I'm not going to give up the chance to publish incredibly novel research because some other people like to work themselves into hysterics about a conceptually incoherent boogeyman with "AI" slapped onto it.
- edanm 9y ago
  > Give me a credible scenario that brings us from strong AI to the annihilation of our species without handwaving about recursive self-improvement and selective idiocy and I'll reconsider my position. What kind of credible scenario are you looking for? You're obviously knowledgeable about the subject, I'm sure you've read various stories, but it's quite easy to dismiss them all as "not credible" or "just so stories". E.g. A good story is Tegmark's story at the beginning of the book Life 3.0, but again, can be easily dismissed if you don't specify acceptance criteria for the story. I mean, simple scenario - AGI gets built by a farming company, gets programmed to farm more corn, but there is no stop condition programmed in, so converts ever available land into corn fields. Is it likely? No, of course not - no specific example is likely. But why is this something that fundamentally can't happen? More importantly, can you specify criteria for a scenario which you would deem acceptable? Cause if not, you're literally saying that there is no possible way to convince you that this is a possibility, which seems to me not to be a good position on pretty much anything.