3 ms·

I appreciate your ability to separate sharing the belief itself from approval of acting on sincerely-held principle. However, I think the danger is much more pl

by khafra 26d ago

I appreciate your ability to separate sharing the belief itself from approval of acting on sincerely-held principle. However, I think the danger is much more plausible than you do. First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae. Second, and more important, consider this fully non-magic route to doom: - We continue putting AI in charge of more things - It continues to get more capable, more eval-aware, and more prone to doing odd things, in service of goals that humans didn't intend to inculcate in it - Eventually, enough of the economy depends on it that we couldn't turn it off, any more than we could turn off the faber-bosch process or cargo shipping - AIs start doing something we can't survive, but less acutely than we couldn't survive turning them off. Everything else we try seems to work at first, but quickly loses effect - Game over

artemisart 26d ago
> First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae. But what is that supposed to mean? Because humanity is not facing existential threat from algae.
- khafra 26d ago
  That's why that was the least important objection, independent from the second, and only intended to address the "magic" claim, by showing that microscopic self-replicators already exist. By analogy, consider how you might respond if someone claimed that there's no possible danger from pocket-sized projectile launchers, because they would require some magic means of propulsion that didn't depend on a taut string attached to a long, flexible arm: You could reply that atlatls can launch projectiles without using a taut string. Atlatls are not pocket-sized, but they are sufficient to establish that projectiles can be non-magically launched without a full bow. You could then go on to describe a sling, or derringer; and these would not be invalidated by your initial objection to the "magic" part.
vigilantpuma 25d ago
But we CAN survive turning off both the Faber-Bosch process and cargo shipping. Our numbers would be diminished, we would lead poorer, much harder lives. But no one would claim that getting rid of modern fertilizers would be the end of the species. But the claim you’re making is that we won’t be able to stop AI from destroying our species because turning it off will destroy our species?
- khafra 25d ago
  The species would survive the end of faber-Bosch, yes; in some diminished form. But the species would not survive, if for some reason we needed to voluntarily stop using faber-Bosch in order to do so—it would take some kind of supernatural event to convince everyone, and even then some countries would probably keep doing it.

by khafra 26d ago

artemisart 26d ago

> First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae. But what is that supposed to mean? Because humanity is not facing existential threat from algae.

khafra 26d ago
That's why that was the least important objection, independent from the second, and only intended to address the "magic" claim, by showing that microscopic self-replicators already exist. By analogy, consider how you might respond if someone claimed that there's no possible danger from pocket-sized projectile launchers, because they would require some magic means of propulsion that didn't depend on a taut string attached to a long, flexible arm: You could reply that atlatls can launch projectiles without using a taut string. Atlatls are not pocket-sized, but they are sufficient to establish that projectiles can be non-magically launched without a full bow. You could then go on to describe a sling, or derringer; and these would not be invalidated by your initial objection to the "magic" part.

vigilantpuma 25d ago

But we CAN survive turning off both the Faber-Bosch process and cargo shipping. Our numbers would be diminished, we would lead poorer, much harder lives. But no one would claim that getting rid of modern fertilizers would be the end of the species. But the claim you’re making is that we won’t be able to stop AI from destroying our species because turning it off will destroy our species?

khafra 25d ago
The species would survive the end of faber-Bosch, yes; in some diminished form. But the species would not survive, if for some reason we needed to voluntarily stop using faber-Bosch in order to do so—it would take some kind of supernatural event to convince everyone, and even then some countries would probably keep doing it.