3 ms·
> Ironically paperclip-maximizing-robots are exactly the thing that the rationalists are so worried about. They are a group of people who missed, and then disav
by cycomanic 1y ago
> Ironically paperclip-maximizing-robots are exactly the thing that the rationalists are so worried about. They are a group of people who missed, and then disavowed, and now advocate disavowing, this "personal" morality, and unsurprisingly they view the world in a lens that doesn't include personality morality, which means mostly being worried about problems of the same sort. But it provokes a strong negative reaction from everyone who thinks about the world in terms of that personal duty to safety.
I head not read any rationalist writing in a long time (and I didn't know about Scott's proximity), but the whole time I was reqding the article I was thinking the same thing you just wrote... "why are they afraid of AI, i.e. the ultimate rationalist taking over the world", maybe something deep inside of them has the same reaction to their own theories as you so eloquently put above.
- jay_kyburz 1y agoI don't read these rationalist essays either, but you don't need to be a deep thinker to understand why any rational person would be afraid of AI and the singularity. The AI will do what its programmed to do, but its programmers morality may not match my own. What more scary is that it may be developed with the morality of a corporation rather than a person. (That is to say, no morals at all.) I think its perfectly justifiable to be scared of a very powerful being with no morals stomping around!
- wat10000 1y agoThose corporations are already superhuman entities with morals that don’t match ours. They do cause a lot of problems. Maybe it’s better to figure out how to fix that real, current problem rather than hypothetical future ones.
- jay_kyburz 1y agoYeah, the AI will be a tool of a corp. So in effect we are talking about limiting corporate power.
- Viliam1234 1y ago> the AI will be a tool of a corp. Up to a certain level. Behind that level, the corp will be a tool of the AI. And even further that way, the corp will no longer be necessary.
- Twey 1y agoThis parallel has been drawn. Charlie Stross [0] in particular thinks the main difference is that pre-digital AIs are capable of behaving much faster, so that other entities (countries, lawmakers, …) have time to react to them. [0]: https://www.antipope.org/charlie/blog-static/2018/01/dude-you-broke-the-future.html https://www.antipope.org/charlie/blog-static/2018/01/dude-yo... Similar claims can be made about any structure of humans that exhibits gestalt intelligence, e.g. nations, stock markets, etc.
- wat10000 1y agoIt's quite likely I got it from that!