3 ms·

So we agree higher intelligence is capable. But, “Is that a bad doomsday scenario?” is the next question. And then, “Is it so bad that society should fear these

by dcow 3y ago

So we agree higher intelligence is capable. But, “Is that a bad doomsday scenario?” is the next question. And then, “Is it so bad that society should fear these intelligence tools?” And to argue that, you have to get into a real abstract realm where there is zero conclusive anything (outside of philosophical thought experiments) indicating whether we should welcome or fear our new super intelligent systems.

naasking 3y ago
You don't have to get that abstract. Absent some guarantee that intelligence implies inherent moral superiority, the precautionary principle would apply, and so we should plan for there being no association between the two. Some reasonable level of caution is thus warranted. People can disagree on what's "reasonable", but we should at least all agree that a danger could exist and that some caution should apply. We can't generalize from any association between intelligence and morality from humans because human morality and intelligence evolved, possibly even co-evolved, and these constraints simply don't apply to artificial systems any more than a bird's feathers apply to artificial flight.
- dcow 3y ago
  We all agree that there may be dragons. The “doomsday” question is whether we’re even remotely close to a dragon’s lair such that we should imminently worry about waking the dragon—risking it burning down the village. To each their own until we find dragon tracks and charred forests.
  - naasking 3y ago
    The doomsday question has two components: the mean and the error bars. Most people probably agree that a reasonable date for likely achieving AGI is maybe 30-50 years from now. But because we don't know what we don't know, the error bars are so wide that they encompass tomorrow (we could be one small generalization trick away!), and that's the danger.

3 ms·

So we agree higher intelligence is capable. But, “Is that a bad doomsday scenario?” is the next question. And then, “Is it so bad that society should fear these

by dcow 3y ago

naasking 3y ago
You don't have to get that abstract. Absent some guarantee that intelligence implies inherent moral superiority, the precautionary principle would apply, and so we should plan for there being no association between the two. Some reasonable level of caution is thus warranted. People can disagree on what's "reasonable", but we should at least all agree that a danger could exist and that some caution should apply. We can't generalize from any association between intelligence and morality from humans because human morality and intelligence evolved, possibly even co-evolved, and these constraints simply don't apply to artificial systems any more than a bird's feathers apply to artificial flight.
- dcow 3y ago
  We all agree that there may be dragons. The “doomsday” question is whether we’re even remotely close to a dragon’s lair such that we should imminently worry about waking the dragon—risking it burning down the village. To each their own until we find dragon tracks and charred forests.
  - naasking 3y ago
    The doomsday question has two components: the mean and the error bars. Most people probably agree that a reasonable date for likely achieving AGI is maybe 30-50 years from now. But because we don't know what we don't know, the error bars are so wide that they encompass tomorrow (we could be one small generalization trick away!), and that's the danger.