> Not unless you think everybody is involved with hooking up every single instance of an AI. That’s not what I meant - let me try to clarify. I think you are g

> Not unless you think everybody is involved with hooking up every single instance of an AI. That’s not what I meant - let me try to clarify. I think you are going to see a very widespread adoption of AI, and the security boundaries are going to look like IoT, which is to say a dumpster fire. The thing that makes me worried is that we will get broad market penetration followed by gradual frog-boiling upgrades to scary capabilities. I don’t think your model of “some bad people will set up dangerous AI and we can simply stop them” is going to be realistic. Rather, I think that the default extrapolation of existing trends is that ~everybody is going to by default use services that effectively hook up LLMs (or their successors) to all the digital and physical actuators in their lives. I’m not that worried about you personally hooking up an AI on your laptop to a chainsaw drone, and then pulling the cord when it behaves badly. Some scenarios that worry me more are things like: what if your Alexa++ gets prompt-hacked? Assume Alexa gained long-running agent abilities. That will be running in some data center along with all the other instances of Alexa. If your instance starts making malicious actions somewhere else on the internet or directing your robot butlers to rampage somewhere, how would you even track down which agent needs to get its process terminated? Even in your terms though, I’m reminded of the episode of “love death and robots” where the humans are trying to get through to tech support while their roombas try to kill them. How would you “turn it off” if your smart-car disables all the UIs and tries to drive you off a cliff? You seem to be imagining a very narrow deployment of this technology, limited to a brain in a box with a big red “off” button on the side. I think cloud SaaS and IoT are the better model, and plenty of physical systems will have their own battery and software switches (ie inputs they can ignore). > that's an argument for not letting stupid humans play around with dangerous toys. I want to be clear again that it’s not some subset of users that are “stupid” for wiring up a dangerous configuration. The default path (assuming AI progress continues) for all software services in the next 5-10 years is for AI agents to get wired into basically every software interface. It gets deployed to everybody. > Which would mean that open sourcing AI code is tantamount to spewing toxic waste randomly out into the world I will probably bite that bullet within a few generations. I don’t think the current models are in themselves dangerous. Unless we solve alignment, I’d be extremely worried about publishing weights for a human-level or above AGI. I’m still undecided on the question of whether the technological proliferation effects of open-weighting current-gen models is bad enough that it must be prohibited. I suspect not yet because most of the danger is actually from closed-source models like ChatGPT. (Also how do you even enforce such a ban? It only makes sense if global.)