5 ms·

> This is not a dig or a slight against them or their work, I mean, you say that, but "nannying themselves out of existence" is nothing if not a dig against th

> This is not a dig or a slight against them or their work, I mean, you say that, but "nannying themselves out of existence" is nothing if not a dig against their work. If they are right about the threat posed by unaligned AGI, and you are right about open-source alternatives inevitably outconpeting them, then what we're seeing is the beginning of a race to the bottom that ends with the extinction of humanity when a paperclip maximizer decides it can generate more social network engagement by replacing humans with computer-brains or something. It's nothing to be glib about.

concinds 4y ago
I don't understand why the popular dichotomy is AI-killbots versus aligned-AI. A much more realistic and relevant one, which we're already facing today, is government AIs that oppress people, and open-source AIs that allow people to fight back. Haven't you seen the "AI" used for unfair sentencing? Thankfully we can fight back against that one with basic statistical analyses demonstrating bias, but imagine if that hadn't been revealed in court documents, how would we have found out? We'd have needed an AI to parse all public inmate records and analyze them, which would likely be far too time-consuming for human researchers since none of it is open/structured data. Soon enough we'll need equally-powerful open-source AIs to even be able to uncover bad-AI injustices that we lack open data about.
- PoignardAzur 4y ago
  There isn't necessarily such a dichotomy in the AI Safety field (though I feel like some people in that field do have similar all-or-nothing viewpoints). A central belief of many AI researchers is that violently unaligned AGI is as realistic as current bad-in-a-socially-familiar-way AIs. It just seems more unrealistic because we've never experienced it before, and it's only existed in fiction. (Eg, the threat of nuclear anihilation would probably have sounded incredibly fanciful to someone in the 19th century) So a lot of AI safety researchers will agree that, yes, we should worry about oppressive use of AI by government, but claim that government is only one of the many pathways through which an unaligned AGI could manifest. And not only that, that any open-source AI strong enough to compete against an unaligned AGI would likely be opaque and its alignment brittle enough that it would be an unreliable ally anyway.
Baeocystin 4y ago
I've struggled to come to a good analogy for why I feel this way. The closest I've come is to imagine a steel foundry that has developed a new alloy that is amazing. The foundry, being run by pro-social people, spend much of their effort on trying to modify the alloy so that it can only be used for good. But that is never going to work, because they are operating on the wrong level of abstraction. Even worse, by focusing on an untenable abstract, they miss the lower-hanging fruit of more effective methods of control.
Baeocystin 4y ago
I really, truly do not intend to be glib. I simply, genuinely think that closing Pandora's Box is impossible. That doesn't mean that I also think that Paperclipocalypse is inevitable, or even slightly likely. But I mean it when I wish the best for the team working on this.

concinds 4y ago

I don't understand why the popular dichotomy is AI-killbots versus aligned-AI. A much more realistic and relevant one, which we're already facing today, is government AIs that oppress people, and open-source AIs that allow people to fight back. Haven't you seen the "AI" used for unfair sentencing? Thankfully we can fight back against that one with basic statistical analyses demonstrating bias, but imagine if that hadn't been revealed in court documents, how would we have found out? We'd have needed an AI to parse all public inmate records and analyze them, which would likely be far too time-consuming for human researchers since none of it is open/structured data. Soon enough we'll need equally-powerful open-source AIs to even be able to uncover bad-AI injustices that we lack open data about.

PoignardAzur 4y ago
There isn't necessarily such a dichotomy in the AI Safety field (though I feel like some people in that field do have similar all-or-nothing viewpoints). A central belief of many AI researchers is that violently unaligned AGI is as realistic as current bad-in-a-socially-familiar-way AIs. It just seems more unrealistic because we've never experienced it before, and it's only existed in fiction. (Eg, the threat of nuclear anihilation would probably have sounded incredibly fanciful to someone in the 19th century) So a lot of AI safety researchers will agree that, yes, we should worry about oppressive use of AI by government, but claim that government is only one of the many pathways through which an unaligned AGI could manifest. And not only that, that any open-source AI strong enough to compete against an unaligned AGI would likely be opaque and its alignment brittle enough that it would be an unreliable ally anyway.

Baeocystin 4y ago

I've struggled to come to a good analogy for why I feel this way. The closest I've come is to imagine a steel foundry that has developed a new alloy that is amazing. The foundry, being run by pro-social people, spend much of their effort on trying to modify the alloy so that it can only be used for good. But that is never going to work, because they are operating on the wrong level of abstraction. Even worse, by focusing on an untenable abstract, they miss the lower-hanging fruit of more effective methods of control.

Baeocystin 4y ago

I really, truly do not intend to be glib. I simply, genuinely think that closing Pandora's Box is impossible. That doesn't mean that I also think that Paperclipocalypse is inevitable, or even slightly likely. But I mean it when I wish the best for the team working on this.