It is a little bit dismissive, which I apologize for, but my intent is not to mock the people worried about AGI as much as to point out that the conversations a

It is a little bit dismissive, which I apologize for, but my intent is not to mock the people worried about AGI as much as to point out that the conversations around AGI in mainstream contexts tend to be reductive, inaccurate, or oversimplified, and I'm frustrated by how that coverage distracts from the more immediate problems. We are building products on a fundamentally insecure piece of technology that multiple researchers are starting to suggest might be impossible to secure, and basically everyone in the industry is completely ignoring that and just launching their products anyway, because they've already invested too much VC money into the hype cycle and are unwilling to hit the brakes and admit there might be critical unsolved problems. It's not so much, "look at these Rationalists, they're so silly that's not a real concern" as much as "look at these Rationalists, they're so worried about the future that they don't realize that their concerns about AGI have basically been coopted into propaganda for OpenAI to distract from the fact that GPT has fundamental security concerns that have to be solved before people can build products on top of this." The EA/Rationalist community doesn't seem to realize that their talk about paperclip maximizers is part of the reason why ordinary people don't believe security researchers when they're told just how gullible and unpredictable LLMs are. They are throwing away short-term concerns to talk about long-term concerns that are poorly understood and difficult to prepare for, and in the process they are increasing the speed of irresponsible AI development and making even their own concerns harder to address. OpenAI loves to have conversations about AGI. It loves to do that kind of security research. It's not as fond of talking about prompt injection, because the news articles that get generated when it talks about paying humans to solve captchas are much better press coverage than the news articles talking about how reading the wrong web page can cause it to start phishing the user. There is a kind of hijacking of EA fear that's happening here, where every opinion piece by Elieser about bombing chipset production companies actually increases people's confidence in the current products and makes the casual reader believe that LLMs are smarter and more secure and more human-like and reliable than they actually are, and increases the likelyhood that some VC firm runs out and starts wiring them into police robots. And I think that very often the Rationalist/EA response to this kind of critique is "yes, those problems are bad, but AGI is worse. It's existential." And my response to that would be that if you're worried about AGI, you're picking arguably the worst strategy possible to try and prevent it. The demonstrable security/privacy risks around LLMs are more than enough to call for caution and pause around their deployment in real-world applications; you don't need to resort to having conversations that (fairly or not) come across to an average person as basically sci-fi storytelling. Especially not if those conversations are (arguably) making people more confident about LLMs than they otherwise would be. On top of that, if you want to talk actual solutions to those existential concerns, solving prompt injection is absolutely a necessary pre-step towards achieving AI alignment. If you can't solve prompt injection, you can't "align" an AI by pretty much any definition of alignment. So it seems to me that those more "short-term" problems are pretty important to talk about and probably should be prioritized even if you're worried primarily about the long-term risks, because there's no plausible solution to those long-term risks that doesn't involve addressing the short-term risks as well.