6 ms·

I read Hofstadter's GEB and Tegmark's Our Mathematical Universe, and of course I developed a rather fond admiration of these brilliant minds. For some reason, b

I read Hofstadter's GEB and Tegmark's Our Mathematical Universe, and of course I developed a rather fond admiration of these brilliant minds. For some reason, both of them have developed a profound aversion and fear of what they consider an existential threat. I have a solid theoretical understanding of these systems, and I spent 15 years studying, building, and deploying them at scale and for diverse use cases. The past 6 months, I spent my days pushing ChatGPT and GPT-4 to their limits. Yet, I don't share at all the fear of Hofstadter, Tegmark or Hinton. A part of me thinks that they have one thing in common: they are old and somewhat recluse thinkers. No matter how brilliant they are, they might be misled by the _appearance_ of intelligence that LLMs project. Another part of me thinks that they are vastly wiser than I'll ever be, so I should also be worried... Future will tell, I guess.

zone411 3y ago
Do you have some specific reasons why we shouldn't be worried? It's not about about what GPT-4 can do now.
climatologist 3y ago
I think they are not good at expressing what exactly they fear. Both understand that people are embedded in culture and tools which act as cognitive aids and guides. Their fear is that people will start trusting these artificial systems too much and delegating too many important decisions to systems that have no understanding of what it means to be human. This is already happening with children that grew up with iPhones and ubiquitous internet connectivity. Their mental habits are markedly different from the people that did not have these affordances. I don't think this is an existential threat but I also understand why they are afraid because I've seen what happens to kids who have their phones taken away. These systems are extremely brittle and prone to all sorts of weird failures so as people start relying on them more and more the probability of catastrophic failures also starts to creep up. All it takes is a single grid failure to show how brittle the whole thing really is and I think that's what they're failing to properly express.
- notpachet 3y ago
  > Their fear is that people will start trusting these artificial systems too much and delegating too many important decisions to systems that have no understanding of what it means to be human In the interview, Hofstadter says what he's afraid of in explicit terms: > It's not clear whether that will mean the end of humanity in the sense of the systems we've created destroying us. It's not clear if that's the case, but it's certainly conceivable. If not, it also just renders humanity a very small phenomenon compared to something else that is far more intelligent and will become incomprehensible to us, as incomprehensible to us as we are to cockroaches. It's not about whether we'll become dependent on AI. It's that AI will become independent of us. Completely different problem. Not saying I agree with that viewpoint per se, but I don't think you're accurately representing what his fears are.
frankfrank13 3y ago
Agreed 100%. He's brilliant, but he's not a programmer. IMO he's been tricked, in a way that Chompsky was not.
- FrustratedMonky 3y ago
  "programmers" are also fallible. I've seen enough programmers get heads down pounding out code, and be completely out of touch on what they are building. If they can lose the big picture on simple apps, then it is not a stretch to think they could lose track on what is consciousness, or what is human. I am also a programmer. But it does get tiring on HN to give so much credence to 'programmers'. Just because someone can debug some JavaScript doesn't make them an expert. I really doubt that many people here have traced out these algorithms and 'know' what is happening.
- kilolima 3y ago
  He's fluent in Lisp, so that would qualify him as a programmer, unless you think that only Silicon Valley wage slaves can be "programmers". https://en.m.wikipedia.org/wiki/Metamagical_Themas https://en.m.wikipedia.org/wiki/Metamagical_Themas
nadam 3y ago
I think they are used to that these fields progress relatively slowly. Also there are some things these systems already know that they though would come much later. Hofstadter even says this in the interview. In the 90s, early 2000s I also thought that AGI will come one day, but it will take centuries. Nowadays I think it can be just decades. The progress is very fast compared to my old worldview. I don't think they are mislead at all. I don't think they mischaracterize the current capabalities of these systems. They just think that if progress is this fast (compared to their previous estimates), AGI can come soon, where soon is 5-10-20 years. Younger guys, who are more used to the state of the art, and did not live when things currently possible seemed far far away are less impressed.
Loquebantur 3y ago
Consider the comparatively simple case of these LLMs providing decent enough chatbots to fool a majority. If you deploy them on social media in a coordinated fashion, you can easily sway public opinion. You only need to train them to adhere to psyops techniques not sufficiently well known nor easily detectable. Of which there are many.
- mikeg8 3y ago
  This assumes there will still be centralized social media sites to deploy to which seems less likely based on how Twitter, Reddit, etc are rapidly changing. The effect of propaganda on say FB will be diminished as less and less young users join the platform.
mlyle 3y ago
I think there's a number of just crappy scenarios that can come with LLMs and other generative AI: - Further trashing our public discourse: making truth even more uncertain and valuable information even harder to find. We're not doing great with social media, and it's easy to envision that generative AI could make it twice as bad. - Kneecapping creative work by commoditizing perhaps half of it. There's going to be a lot of bodies fighting over the scraps that remain. - Fostering learned helplessness. I think you need to be a good writer and thinker to fully use LLMs' capabilities. But a whole lot of kids are looking at machines "writing perfectly" and think they don't need to learn anything. We don't need any further progress for these things to happen. Further progress may be even scarier, but the above is scary enough.
- zerocrates 3y ago
  Yeah these effects seem hugely foreseeable (and/or are currently happening already) but tend to not be the type of thing "AI risk" people are talking about.
  - jurgenaut23 3y ago
    Exactly! I am positive there will be a ton of negative impact, like with most (all?) of tech. And it might be worse than anything we’ve seen (although beating the Snapchat algo might prove tricky). But that’s NOT what Tegmark and Hofstadter are talking about… their concern is existential and somewhat philosophical in the case of Hofstadter, as if GPT-4 questioned his very nature. To me, that doesn’t make sense.
ilaksh 3y ago
I have been focused on ChatGPT and other generative AI also since last year. They are intelligent. Hofstadter does seem to be possibly mistakenly assigning GPT-4 more animal characteristics than it really has, like a subjective stream of consciousness, but he is correct when he anticipates that these systems will shortly eclipse our intelligence. No, GPT-4 does not have many characteristics of most animals, such as high bandwidth senses, detailed spatial-temporal world models, emotions, fast adaptation, survival instinct, etc. It isn't alive. But that doesn't mean that it doesn't have intelligence. We will continue to make these systems fully multimodal, more intelligent, more robust, much, much faster, and increasingly more animal-like. Even with say another 30% improvement in the IQ and without any of the animalness, we must anticipate multimodal operations and vast increases in efficiency in the next 5-10 years for large models. When it can be operated continuously outputting and reasoning and acting at 50-100 times human speed and genius level, that is dangerous. Because it means that the only way for humans to compete is to deploy these models and let them make the decisions. Because interrupting them to figure out what the hell they are doing and try to direct them means your competitors race ahead the equivalent of weeks. And researchers are focused on making more and more animal-like systems. This combined with hyperspeed and genius-level intelligence will definitely be dangerous. Having said all of that, I also think that these technologies are the best hope that humanity has for significantly addressing our severe problems. But we will shortly be walking a fine line.
reducesuffering 3y ago
It’s pretty amazing how so many titans people respect are warning the alarm bells: Hofstadter, Altman, Hassabis, Bengio, Hinton, Tegmark, Sutskever, etc. but the implications are too scary for most people here to accept. So one by one the most common copium response is to downplay each about-facer's achievements. I like the odds betting on the intellectual prowess of that first group. On the other hand, crypto king Marc Andreesen is on your side! But I empathize, because like Hofstadter says: "And my whole intellectual edifice, my system of beliefs... It's a very traumatic experience when some of your most core beliefs about the world start collapsing. And especially when you think that human beings are soon going to be eclipsed. It felt as if not only are my belief systems collapsing, but it feels as if the entire human race is going to be eclipsed and left in the dust soon." Don't Look Up https://twitter.com/kristjanmoore/status/1663860424100413440 https://twitter.com/kristjanmoore/status/1663860424100413440
lhl 3y ago
While this might be somewhat applicable for Hofstadter or Tegmark, I think it doesn't applies quite the same way to Hinton. Remember, until this Spring, he's been at Google Brain and has been supervising grad students until recently (famously including Ilya Sutskever (2012)), and has invested in Cohere and other AI startups. I also has a suspicion Hinton might also have a "solid theoretical understanding" of these systems, and don't think he's being misled at all. Also, I don't think that any of these people think that GPT-4 is itself an existential threat, but rather are worried about the exponential curve of development (I listened to Tegmark's Lex Podcast interview and that seemed to be his main concern). I think it's prudent to be worrying now, especially when capabilities growth is far outstripping safety. This is a huge concern to society whether you are considering, alignment, control, or even bad actor prevention/societal upheaval. I've also been spending most of my waking hours these past months poking at LLM models and code, and trying to keep up on the latest ML research (it's own full time job), and while I do think there's a pretty good chances AI kills us all, I think it's much more likely it's because we make some incredibly capable AIs and people will tell them to do so, rather than it being contingent on an independent super-intelligence arising and deciding to on its own (although I do think that's a non-zero risk). As you say, I guess we'll just have to see where we top out on this particular sigmoid, but I'm all for more people thinking through the implications, because I think so far I don't think we (as a society) have thought this through very well yet and all the money and momentum is going to keep pushing along that path.
billti 3y ago
I don't think many people are worried that what we have now is a major risk. The major concern is the implications of the trajectory we are now on. If you look at what ChatGPT and Midjourney and the like can do now compared to just a couple of years ago, it's pretty incredible. If you extrapolate the next few similar jumps in capability, and assume that won't be 20 years away, then what AI is going to be capable of before even my kids leave college is going to be mind-boggling, and in some possible futures not in a good way. I remember seeing this talk from Sam Harris nearly 6 years ago and it logically making a lot of sense back then (https://youtu.be/8nt3edWLgIg https://youtu.be/8nt3edWLgIg). The past couple of years have made this all the more prescient. (Worth a watch if you have 15 mins).
esafak 3y ago
LLMs are not going to do anything but the next class of models may well. They will have a physical understanding of the world, like us. Slap some sensors and a body on it and you will have something that might look scary.

zone411 3y ago

Do you have some specific reasons why we shouldn't be worried? It's not about about what GPT-4 can do now.

climatologist 3y ago

I think they are not good at expressing what exactly they fear. Both understand that people are embedded in culture and tools which act as cognitive aids and guides. Their fear is that people will start trusting these artificial systems too much and delegating too many important decisions to systems that have no understanding of what it means to be human. This is already happening with children that grew up with iPhones and ubiquitous internet connectivity. Their mental habits are markedly different from the people that did not have these affordances. I don't think this is an existential threat but I also understand why they are afraid because I've seen what happens to kids who have their phones taken away. These systems are extremely brittle and prone to all sorts of weird failures so as people start relying on them more and more the probability of catastrophic failures also starts to creep up. All it takes is a single grid failure to show how brittle the whole thing really is and I think that's what they're failing to properly express.

notpachet 3y ago
> Their fear is that people will start trusting these artificial systems too much and delegating too many important decisions to systems that have no understanding of what it means to be human In the interview, Hofstadter says what he's afraid of in explicit terms: > It's not clear whether that will mean the end of humanity in the sense of the systems we've created destroying us. It's not clear if that's the case, but it's certainly conceivable. If not, it also just renders humanity a very small phenomenon compared to something else that is far more intelligent and will become incomprehensible to us, as incomprehensible to us as we are to cockroaches. It's not about whether we'll become dependent on AI. It's that AI will become independent of us. Completely different problem. Not saying I agree with that viewpoint per se, but I don't think you're accurately representing what his fears are.

frankfrank13 3y ago

Agreed 100%. He's brilliant, but he's not a programmer. IMO he's been tricked, in a way that Chompsky was not.

FrustratedMonky 3y ago
"programmers" are also fallible. I've seen enough programmers get heads down pounding out code, and be completely out of touch on what they are building. If they can lose the big picture on simple apps, then it is not a stretch to think they could lose track on what is consciousness, or what is human. I am also a programmer. But it does get tiring on HN to give so much credence to 'programmers'. Just because someone can debug some JavaScript doesn't make them an expert. I really doubt that many people here have traced out these algorithms and 'know' what is happening.
kilolima 3y ago
He's fluent in Lisp, so that would qualify him as a programmer, unless you think that only Silicon Valley wage slaves can be "programmers". https://en.m.wikipedia.org/wiki/Metamagical_Themas https://en.m.wikipedia.org/wiki/Metamagical_Themas

nadam 3y ago

I think they are used to that these fields progress relatively slowly. Also there are some things these systems already know that they though would come much later. Hofstadter even says this in the interview. In the 90s, early 2000s I also thought that AGI will come one day, but it will take centuries. Nowadays I think it can be just decades. The progress is very fast compared to my old worldview. I don't think they are mislead at all. I don't think they mischaracterize the current capabalities of these systems. They just think that if progress is this fast (compared to their previous estimates), AGI can come soon, where soon is 5-10-20 years. Younger guys, who are more used to the state of the art, and did not live when things currently possible seemed far far away are less impressed.

Loquebantur 3y ago

Consider the comparatively simple case of these LLMs providing decent enough chatbots to fool a majority. If you deploy them on social media in a coordinated fashion, you can easily sway public opinion. You only need to train them to adhere to psyops techniques not sufficiently well known nor easily detectable. Of which there are many.

mikeg8 3y ago
This assumes there will still be centralized social media sites to deploy to which seems less likely based on how Twitter, Reddit, etc are rapidly changing. The effect of propaganda on say FB will be diminished as less and less young users join the platform.

mlyle 3y ago

I think there's a number of just crappy scenarios that can come with LLMs and other generative AI: - Further trashing our public discourse: making truth even more uncertain and valuable information even harder to find. We're not doing great with social media, and it's easy to envision that generative AI could make it twice as bad. - Kneecapping creative work by commoditizing perhaps half of it. There's going to be a lot of bodies fighting over the scraps that remain. - Fostering learned helplessness. I think you need to be a good writer and thinker to fully use LLMs' capabilities. But a whole lot of kids are looking at machines "writing perfectly" and think they don't need to learn anything. We don't need any further progress for these things to happen. Further progress may be even scarier, but the above is scary enough.

zerocrates 3y ago
Yeah these effects seem hugely foreseeable (and/or are currently happening already) but tend to not be the type of thing "AI risk" people are talking about.
- jurgenaut23 3y ago
  Exactly! I am positive there will be a ton of negative impact, like with most (all?) of tech. And it might be worse than anything we’ve seen (although beating the Snapchat algo might prove tricky). But that’s NOT what Tegmark and Hofstadter are talking about… their concern is existential and somewhat philosophical in the case of Hofstadter, as if GPT-4 questioned his very nature. To me, that doesn’t make sense.

ilaksh 3y ago

I have been focused on ChatGPT and other generative AI also since last year. They are intelligent. Hofstadter does seem to be possibly mistakenly assigning GPT-4 more animal characteristics than it really has, like a subjective stream of consciousness, but he is correct when he anticipates that these systems will shortly eclipse our intelligence. No, GPT-4 does not have many characteristics of most animals, such as high bandwidth senses, detailed spatial-temporal world models, emotions, fast adaptation, survival instinct, etc. It isn't alive. But that doesn't mean that it doesn't have intelligence. We will continue to make these systems fully multimodal, more intelligent, more robust, much, much faster, and increasingly more animal-like. Even with say another 30% improvement in the IQ and without any of the animalness, we must anticipate multimodal operations and vast increases in efficiency in the next 5-10 years for large models. When it can be operated continuously outputting and reasoning and acting at 50-100 times human speed and genius level, that is dangerous. Because it means that the only way for humans to compete is to deploy these models and let them make the decisions. Because interrupting them to figure out what the hell they are doing and try to direct them means your competitors race ahead the equivalent of weeks. And researchers are focused on making more and more animal-like systems. This combined with hyperspeed and genius-level intelligence will definitely be dangerous. Having said all of that, I also think that these technologies are the best hope that humanity has for significantly addressing our severe problems. But we will shortly be walking a fine line.

reducesuffering 3y ago

It’s pretty amazing how so many titans people respect are warning the alarm bells: Hofstadter, Altman, Hassabis, Bengio, Hinton, Tegmark, Sutskever, etc. but the implications are too scary for most people here to accept. So one by one the most common copium response is to downplay each about-facer's achievements. I like the odds betting on the intellectual prowess of that first group. On the other hand, crypto king Marc Andreesen is on your side! But I empathize, because like Hofstadter says: "And my whole intellectual edifice, my system of beliefs... It's a very traumatic experience when some of your most core beliefs about the world start collapsing. And especially when you think that human beings are soon going to be eclipsed. It felt as if not only are my belief systems collapsing, but it feels as if the entire human race is going to be eclipsed and left in the dust soon." Don't Look Up https://twitter.com/kristjanmoore/status/1663860424100413440 https://twitter.com/kristjanmoore/status/1663860424100413440

lhl 3y ago

While this might be somewhat applicable for Hofstadter or Tegmark, I think it doesn't applies quite the same way to Hinton. Remember, until this Spring, he's been at Google Brain and has been supervising grad students until recently (famously including Ilya Sutskever (2012)), and has invested in Cohere and other AI startups. I also has a suspicion Hinton might also have a "solid theoretical understanding" of these systems, and don't think he's being misled at all. Also, I don't think that any of these people think that GPT-4 is itself an existential threat, but rather are worried about the exponential curve of development (I listened to Tegmark's Lex Podcast interview and that seemed to be his main concern). I think it's prudent to be worrying now, especially when capabilities growth is far outstripping safety. This is a huge concern to society whether you are considering, alignment, control, or even bad actor prevention/societal upheaval. I've also been spending most of my waking hours these past months poking at LLM models and code, and trying to keep up on the latest ML research (it's own full time job), and while I do think there's a pretty good chances AI kills us all, I think it's much more likely it's because we make some incredibly capable AIs and people will tell them to do so, rather than it being contingent on an independent super-intelligence arising and deciding to on its own (although I do think that's a non-zero risk). As you say, I guess we'll just have to see where we top out on this particular sigmoid, but I'm all for more people thinking through the implications, because I think so far I don't think we (as a society) have thought this through very well yet and all the money and momentum is going to keep pushing along that path.

billti 3y ago

I don't think many people are worried that what we have now is a major risk. The major concern is the implications of the trajectory we are now on. If you look at what ChatGPT and Midjourney and the like can do now compared to just a couple of years ago, it's pretty incredible. If you extrapolate the next few similar jumps in capability, and assume that won't be 20 years away, then what AI is going to be capable of before even my kids leave college is going to be mind-boggling, and in some possible futures not in a good way. I remember seeing this talk from Sam Harris nearly 6 years ago and it logically making a lot of sense back then (https://youtu.be/8nt3edWLgIg https://youtu.be/8nt3edWLgIg). The past couple of years have made this all the more prescient. (Worth a watch if you have 15 mins).

esafak 3y ago

LLMs are not going to do anything but the next class of models may well. They will have a physical understanding of the world, like us. Slap some sensors and a body on it and you will have something that might look scary.