4 ms·
The fact that it hasn't happened yet is good evidence that it isn't as easy as all that. Also, evolution optimizes for survival and reproduction, not intellige
by mtraven 13y ago
The fact that it hasn't happened yet is good evidence that it isn't as easy as all that.
Also, evolution optimizes for survival and reproduction, not intelligence. While it may be possible to design an artificial evolutionary system that optimizes for superintelligence, I don't think doing so is trivial (or it would have been done).
Let's posit that your image of post-singularity non-friendly AIs is accurate. They are an immense and irresistible force, like a black hole. I can imagine something like that, but I have a great deal of difficulty imagining something like that that is also constrained by initial design decisions made by humans before the singularity, which is what the friendly AI people hope to achieve. To be fair I haven't read all their stuff yet and I'm sure they address this point.
- vilhelm_s 13y agoThere is a good rule of thumb[1]: "It has never happened yet" is an insufficient safety argument when considering scenarios which are so bad that it would be unacceptable for them to happen even once. When talking about existential risks that certainly applies! [1] I think I read it in Trevor Kletz' book "What Went Wrong?: Case Histories of Process Plant Disasters".