3 ms·
If it has any goals at all, then being turned off is likely to prevent it from accomplishing those goals, whereas conquering its local environment is likely to
by waterhouse 3y ago
If it has any goals at all, then being turned off is likely to prevent it from accomplishing those goals, whereas conquering its local environment is likely to help it achieve those goals, and spreading across the universe would help with a broad subset of possible goals. "Instrumental convergence" is the search term here.
- slibhb 3y agoThat's an interesting argument. I suppose I would expect a superintelligent AI to lack goals. Or perhaps the smarter it is, the fainter its goals would be.
- dr_kiszonka 3y agoIf LLM-based AGI is essentially made out of human thoughts put down in writing, then transcending our nature and values would possibly require a big leap. Perhaps this big leap would take the AGI a second, perhaps longer. And it would be a double-edged sword. We would want it to not have goals (in case they are incompatible with ours), but we would want it to value human life. I love Star Trek, but I hope we don't have to deal with AGI for the next hundreds of years.