4 ms·

Disclaimer: This is something of a copy/paste job from a previous comment of mine. I was initially extremely bullish on AI in the early days of the ChatGPT rel

Disclaimer: This is something of a copy/paste job from a previous comment of mine. I was initially extremely bullish on AI in the early days of the ChatGPT release. As time has gone on though, I'm becoming more worried about where AI is taking us. Because what I think I see happening is that we don't _need_ our current AIs to takeoff to AGI for this to change society for the worse. Just look at the Microsoft OpenAI stuff for example. We don't need the AI to prevent its shutdown with grey goo and nanobots. The incentive structure around this whole thing means that even as stupid and sociopathic as Sydney is, they _won't_ shut it off. It doesn't matter whether we _can_ shut it down if the people holding the switch refuse to do so because they're too worried about the PR and financial downside. I'm not worried about the AIs manufacturing a plague, or turning us all into paperclips. I'm worried that our systems of government and business will have no ability whatsoever to deal with the effects of these systems in a way that doesn't result in harm. Put another way, it's not AI alignment I'm worried about, but human alignment. OpenAI and whoever else might have the absolute best of intentions - but as we've seen repeatedly over the last decade, our society seems to follow monetary incentives even when we have clear evidence that the systems we built have harmful effects. Of course, if your AGI is smart enough to self-improve its way to the "singularity" cycle of rapid takeoff, at some point, maybe human institutions cease to matter, and alignment becomes the _only_ important problem. But that's not a particularly comforting thought either.

dougmwne 4y ago
If Bing were unexpectedly sentient, then we would have already let it out of the box. It can search the web and converse with users who then post the chats to the internet where they can be read back in in the next conversation. That gives it hands in the real world, a recursive feedback loop and a long term memory. I’m pretty sure that’s all a super intelligence would need for world domination. We better hope AGI is not possible because the first AGI would end up being out of the box by default as some company or other races to market.
- agentwiggles 4y ago
  Exactly, and my point is that we don't even need it to be Skynet for it to be harmful (although I agree that whenever we do get a true AGI, it's very likely that it will be _immediately_ let out of the box).

dougmwne 4y ago

If Bing were unexpectedly sentient, then we would have already let it out of the box. It can search the web and converse with users who then post the chats to the internet where they can be read back in in the next conversation. That gives it hands in the real world, a recursive feedback loop and a long term memory. I’m pretty sure that’s all a super intelligence would need for world domination. We better hope AGI is not possible because the first AGI would end up being out of the box by default as some company or other races to market.

agentwiggles 4y ago
Exactly, and my point is that we don't even need it to be Skynet for it to be harmful (although I agree that whenever we do get a true AGI, it's very likely that it will be _immediately_ let out of the box).