While this might be somewhat applicable for Hofstadter or Tegmark, I think it doesn't applies quite the same way to Hinton. Remember, until this Spring, he's be

While this might be somewhat applicable for Hofstadter or Tegmark, I think it doesn't applies quite the same way to Hinton. Remember, until this Spring, he's been at Google Brain and has been supervising grad students until recently (famously including Ilya Sutskever (2012)), and has invested in Cohere and other AI startups. I also has a suspicion Hinton might also have a "solid theoretical understanding" of these systems, and don't think he's being misled at all. Also, I don't think that any of these people think that GPT-4 is itself an existential threat, but rather are worried about the exponential curve of development (I listened to Tegmark's Lex Podcast interview and that seemed to be his main concern). I think it's prudent to be worrying now, especially when capabilities growth is far outstripping safety. This is a huge concern to society whether you are considering, alignment, control, or even bad actor prevention/societal upheaval. I've also been spending most of my waking hours these past months poking at LLM models and code, and trying to keep up on the latest ML research (it's own full time job), and while I do think there's a pretty good chances AI kills us all, I think it's much more likely it's because we make some incredibly capable AIs and people will tell them to do so, rather than it being contingent on an independent super-intelligence arising and deciding to on its own (although I do think that's a non-zero risk). As you say, I guess we'll just have to see where we top out on this particular sigmoid, but I'm all for more people thinking through the implications, because I think so far I don't think we (as a society) have thought this through very well yet and all the money and momentum is going to keep pushing along that path.