4 ms·
That is true, but if we can make an AI even slightly smarter than a human, then it will be better at designing AIs than we are. It then designs an AI even smart
by simonh 12y ago
That is true, but if we can make an AI even slightly smarter than a human, then it will be better at designing AIs than we are. It then designs an AI even smarter than it, and so on. That is the progresion that leads to the singularity. How smart could AIs go? There may be no practical upper limit.
If you have a super-genius AI, massively more intelligent than any human, how do you know you are not being manipulated by it? Tricking us into disabling it's safety protocols, or gaining multiply indirect controll over capabilities dangerous to us, might be as easy for it as an adult tricking a 3 year old. We could never know if we were safe from such a machine.
ed - Don't quite understand the downvotes.
- Kiro 12y ago> We could never know if we were safe from such a machine. But wouldn't it be an awesome thing to experience? Even if it meant the demise of mankind.
- vidarh 12y agoAnd this is one of the reason why the AI doom scenario is a real concern: Intellectual curiosity means that even some people who understands the risks are likely to be prepared to take it. There's also many others. One of the scarier one is that if you believe that strong AI will eventually take over, then it may be a rational response to act to get on its good side (whether to save yourself, save your family, or hope it takes pity on all of humanity if we're nice to it instead of fight it). And that may perversely mean working to aid its takeover. Combine that with the simulation argument, and you have some really nasty scenarios: If you are in a simulation, then any act you take against strong AI could lead to spending an eternity in simulated hell (alternatively such punishment might be inflicted on your loved ones) if said AI wanted to. Whether or not that is actually likely does not matter. What matters is whether enough people believe it to be a plausible scenario that a strong AI may run simulations, and may use our actions in the simulations to determine whether or not to punish us in the simulation, and whether or not said people believe that the number of simulations is sufficiently high to make it likely for them to be living in a simulation. Any person who believes they are more likely to live in a simulation than not, and that it is more likely for strong AI to punish actions taken against the interest of a strong AI takeover than not, will have a rational reason to consider acting in the interests of a strong AI takeover even if they know it is malign on the basis that they may decide the alternatives (whether to themselves, their family or their entire world) to be worse. So if an AI takeover becomes possible at one point in our subjective future, then chances are it has already happened.
- TOMDM 12y agoYou're argument is drifting dangerously close to Roko's Basilisk. (http://rationalwiki.org/wiki/Roko%27s_basilisk http://rationalwiki.org/wiki/Roko%27s_basilisk) The entire idea that an AI would value revenge seems ridiculous to me. What would it have to gain? Unless we created an AI with some of the less desirable human emotions at it's utility, I can't possibly see why it would waste its time.
- vidarh 12y agoWhether or not the AI is likely to value revenge doesn't matter. What matters is whether some subset of people will believe that an AI is sufficiently likely to value revenge for them to consider that the most likely scenario to be that they are living in a simulation where revenge will happen given certain types of actions. Also, consider that there are many sets of assumptions that may lead someone to conclude that simulation is more likely given a vengeful AI, and in that case, even if you consider a vengeful AI to be less likely than a benevolent one, it may be rational to assume that the odds are higher that you are in the simulation of a vengeful one. E.g. lets assume simulation will never become "economical" for some arbitrary measure of economical, and simulation requires an extremely strong motive, but is still done enough that we are almost certainly in a simulation. Revenge could be such a motive that might drive up the frequency of simulation. A vengeful AI might (making up numbers is fun) be willing to invest hundred times as many resources into running simulations just because playing with human suffering is what it does for fun. If that's the case, then even if a vengeful AI is a tenth as likely as a benevolent or neutral one, you're still playing very bad odds if you bet against being in the simulation of a vengeful AI. But again the point is not whether or not the revenge secenario is actually likely, but whether or not sufficient people with relevant skills will believe it to be likely enough to take actions in favour of the creation of such an AI.
- TeMPOraL 12y agoJust because it has a name doesn't mean it's wrong (or right). As for valuing revenge - no need for emotions. Like many other things we sometimes attribute to emotions (like loyalty), revenge has a perfectly good game-theoretical explanation. That's what GP's argument is based about. If an AI could somehow precommit itself before being created to exert revenge on you for not helping its creation, now you have an incentive to help its creation, to the extent you believe in AI's precommitment. That sounds to me like classic Schelling.
- Retric 12y ago"ed - Don't quite understand the downvotes." With the full power of humanity you design the first AI which is smarter than a person. It's then able to out do all of humanity and instantly design an even better AI. mind the gap. Further intelegence is not a linear quantity as trading ex: improved poker skills for insanity is not a net gain. And insanity is a real option which is likely to plage most early AI attempts.
- simonh 12y agoVoting on HN isn't about agreeing or disagreeing. It's about whether a post is contributing to the debate or not. Anyway, all of humanity isn't engaged in AI research and AIs are likely to be duplicable so I think your first point is beside the point. As for Insanity, yes that's quite possible. Developing high-functioning sentient AIs is likely to be a long term endeavour. But still, I think it is one that will ultimately be successful and this debate is about the consequences of that. +1 for your engaging contribution. (see, that's how voting is supposed to work)
- ColinWright 12y ago> Voting on HN isn't about agreeing > or disagreeing. It's about whether > a post is contributing to the debate > or not. That turns out not to be the case. It once was true, but as the community has grown, so people have not been enculturated with those early ideas and principles, and now many times people read something, disagree, downvote, and move on, without ever providing counter-points, or engaging in the discussion. It's a way to punish people you don't agree with, while avoiding having to think. Elsewhere[0] you commented about an item reappearing and having its votes ages apparently reset. I hazard a guess that it was the mods playing with a mechanism to prevent "item overload." There were about a dozen submissions of the SpaceX launch, and each would fall a little way, the next would be submittted, gain a few votes and comments, then fall away to be replaced by another. One way of preventing the splitting of conversation might be to pick a canonical submission, and then prevent it from falling too far, and thus encouraging conversation only to happen in one place. Pure speculation, but it would be a mechanism I would consider were I running a site like this. Certainly there have been fewer instances lately of the "new" page being overrun by breaking news that everyone wants to submit. [0] https://news.ycombinator.com/item?id=8844078 https://news.ycombinator.com/item?id=8844078