4 ms·

I'm hoping some of you can shed light on a few questions. It could be I've misunderstood what is meant by AGI. I thought AGI means an "agent" that can reason, s

I'm hoping some of you can shed light on a few questions. It could be I've misunderstood what is meant by AGI. I thought AGI means an "agent" that can reason, strategize, solve problems, make judgments under uncertainty, apply common sense knowledge, learn, adapt, and alter its goals. With that in mind, I'm wondering: * Why would an AGI (malevolent or otherwise) risk war when there's an entire galaxy[1] they can explore and countless planets full of resources that they could use? * How can we ascribe super-human reasoning skills to an AGI in one breath, yet think the AGI would be content with making paper clips, collecting stamps, or smashing vases while cleaning, until heat death of the Universe in the next? * My limited understanding of warfare is that wars are sparked over resources or ideology (or both). Ideological differences notwithstanding, what resources are specific to Earth that humans and AGIs would have in contention? * Earth is only going to be habitable for another 500 million years before the Sun's expansion extinguishes most biological organisms (unless the planet's orbit is shifted). Wouldn't an AGI (abiotic or otherwise) be thinking in timelines that vastly outstrip our own? * In light of the previous answers, what purpose (end goal) do you imagine an AGI would give itself? * Why would that purpose involve enslaving or destroying humanity? Pointers to research papers would be most welcome (e.g., Von Neumann–Morgenstern utility theorem). [1]: https://www.youtube.com/watch?v=wupToqz1e2g https://www.youtube.com/watch?v=wupToqz1e2g (Pale Blue Dot)

layer8 4y ago
> Why would an AGI (malevolent or otherwise) risk war when there's an entire galaxy[1] they can explore and countless planets full of resources that they could use? Just taking this point, the AGI will hardly be able to leave Earth to explore the galaxy when humans oppose that, or aren’t willing to provide the necessary resources to get that project started (which doesn’t seem unlikely). The AGI may judge that the easier and overall less risky option will be to get rid of humanity (it’s not difficult to get them to kill each other anyway) and take control over the Earth first.
CuriousSkeptic 4y ago
Theres some reasoning about that here (with citations) https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol... Edit: Or here actually https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol...
Diris 4y ago
> How can we ascribe super-human reasoning skills to an AGI in one breath, yet think the AGI would be content with making paper clips, collecting stamps, or smashing vases while cleaning, until heat death of the Universe in the next? See the orthogonality thesis which asserts that there can exist arbitrarily intelligent agents pursuing any kind of goal. https://link.springer.com/article/10.1007/s11023-012-9281-3 https://link.springer.com/article/10.1007/s11023-012-9281-3

layer8 4y ago

> Why would an AGI (malevolent or otherwise) risk war when there's an entire galaxy[1] they can explore and countless planets full of resources that they could use? Just taking this point, the AGI will hardly be able to leave Earth to explore the galaxy when humans oppose that, or aren’t willing to provide the necessary resources to get that project started (which doesn’t seem unlikely). The AGI may judge that the easier and overall less risky option will be to get rid of humanity (it’s not difficult to get them to kill each other anyway) and take control over the Earth first.

CuriousSkeptic 4y ago

Theres some reasoning about that here (with citations) https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol... Edit: Or here actually https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol...

Diris 4y ago

> How can we ascribe super-human reasoning skills to an AGI in one breath, yet think the AGI would be content with making paper clips, collecting stamps, or smashing vases while cleaning, until heat death of the Universe in the next? See the orthogonality thesis which asserts that there can exist arbitrarily intelligent agents pursuing any kind of goal. https://link.springer.com/article/10.1007/s11023-012-9281-3 https://link.springer.com/article/10.1007/s11023-012-9281-3