4 ms·

The Less Wrong community isn't a community of careful consideration and logical reasoning. It's a cult that believes their own beliefs so fervently they would b

The Less Wrong community isn't a community of careful consideration and logical reasoning. It's a cult that believes their own beliefs so fervently they would burn the earth for it. Small communities like this inevitably become so insular they recycle apocalyptic beliefs. There's no evidence that stochastic parrots are evidence of true AI. There's no evidence that humans can, by practice, subordinate emotions and become purely rational thought machines as Less Wrong people or "shape rotators" believe. Even such titles are counter-evidence of such beliefs. The human condition is so bound up in feelings, it's unclear where thinking begins and feelings end. Thoughts arise from contact, contact which produces feelings. Gut microbes are as wrapped up in cognition as the brain itself. Any out-group like this is bound to be hopelessly entrapped in its own cognitive fallacies. Yudkowsky is no exception. EDIT: Before anyone says but Yudkowsky already gets ahead of your argument with > None of this danger depends on whether or not AIs are or can be conscious; it’s intrinsic to the notion of powerful cognitive systems that optimize hard and calculate outputs that meet sufficiently complicated outcome criteria. It's one sentence that handwaves away what GPT-4 and its ilk are, which is not AI. It isn't synthetic intelligence because intelligence isn't what's happening on a fundamental level.

mitthrowaway2 4y ago
There are lots of weird and crazy people reading and writing on Less Wrong, for sure. Do you think that, without gut microbes and emotions, AI will never become intelligent enough to be existentially dangerous? Are those essential to its ability to outsmart humans?
- emptysongglass 4y ago
  I think we're asking the wrong questions. We're assuming that a. what we're developing (an associative propability machine) is in any way related to the pursuit of real artificial intelligence. We're assuming that b. motive, desire, passion, some motive force is inhered into the thing. I, personally, don't think we've seen anything that can be dangerous independent of its wielding by a badly acting human. That is to say, human agency and acting force is required on the other side of the LLM to do bad actions, like any other tool. I don't know what real AI that threatens humans with an extinction level event looks like. It's not large language models. I don't know if humans are capable of generating it. I think anthropomorphization is really this massive veil that hangs over all of us. We can't get outside of it (our essential fallacy) so we're cursed to see it everywhere.
  - mitthrowaway2 4y ago
    I'm fairly certain that an LLM alone will not independently start taking actions and pursuing goals and causing harm. That's not how LLMs work, as you and I both know. I'm not confident I can say the same about, say: an LLM calling itself from within an infinite loop in which the LLM is asked to predict actions that will [eg. add money into a bank account], and then to write code that executes those actions, then to examine the results of those actions and update its plan. (This takes no additional breakthrough in AI architecture, only an increase in the reliability of GPT for it to become practical. It is basically a summary of the AutoGPT project). I don't know if LLMs are anywhere close to any part of what goes on inside the human thought process. But I am sure that evolution was able to stumble upon the human thought process. In 2003 I was quite confident that was forever beyond the reach of humans to program into a computer, but in 2023 I am far less certain. I think human cognition is gradually becoming less of a mystery.
NumberWangMan 4y ago
Wait, but we don't actually even know what is happening inside GPT-4 on a fundamental level, to produce the output we see. We don't even know what is happening inside our own brains, really. How do the neurons turning on and off produce reasoning and consciousness? So in the same way, how can you say that GPT-4 is not AI, definitively, or at least a primordial form of one? Clearly, there's some arrangement of neurons that produces intelligence. Maybe it's just a "stochastic parrot", but one can probably make a similarly dismissive-sounding and yet accurate description of how humans cognate. Sometimes quantity has its own quality. Maybe a big enough stochastic parrot becomes smart.
- emptysongglass 4y ago
  > Wait, but we don't actually even know what is happening inside GPT-4 on a fundamental level, to produce the output we see. This just isn't true. All of this research descends from transformer research that came from Google in 2017. We know exactly how they work. There's nothing surprising in what's going on. I spent a bit of time looking for a decent intro to all this because I wanted to at least provide a resource for people who are terrified of LLMs. I think this is a pretty decent one [1] [1] https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
  - NumberWangMan 4y ago
    My understanding was that we're not sure of how some of the things that look like reasoning are coming about, but I'll gladly check it out. I may be wrong. I'm not particularly terrified of GPT-4, but a more of what comes after. Maybe GPT-6 or 8, or another breakthrough non-LLM AI. I'm not sure if reading it will do much more than push my estimate of real AGI out a couple of years, but thanks anyway, I appreciate it.
  - NumberWangMan 4y ago
    Ok, after reading through it, I don't think I was speaking inaccurately. Here are some quotes from the blog post (which was very neat, by the way, thanks again) > And it’s part of the lore of neural nets that—in some sense—so long as the setup one has is “roughly right” it’s usually possible to home in on details just by doing sufficient training, without ever really needing to “understand at an engineering level” quite how the neural net has ended up configuring itself. > What determines this structure? Ultimately it’s presumably some “neural net encoding” of features of human language. But as of now, what those features might be is quite unknown. In effect, we’re “opening up the brain of ChatGPT” (or at least GPT-2) and discovering, yes, it’s complicated in there, and we don’t understand it—even though in the end it’s producing recognizable human language. This is the kind of thing I'm referring to. Even though we can look at pictures of the neuron activations, we don't really know what it's doing, any more than you can look at a picture of a brain scan of a person speaking and know why they decided to say those particular words. We know at a low level how the network works, of course, because we coded it. It's the emergent behavior that we don't understand, and that's the bit that's scary because that's where the AI risk lives. It's like how we understand particle physics pretty well, and chemistry to some degree, but biology is a massively complicated jungle that we've barely scratched the surface of. Maybe there's something we could develop analogous to using an MRI as a lie detector, for these networks. But as far as I know, this is still an unsolved problem, and apparently really hard for networks that are smarter than you are: https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H...

mitthrowaway2 4y ago

There are lots of weird and crazy people reading and writing on Less Wrong, for sure. Do you think that, without gut microbes and emotions, AI will never become intelligent enough to be existentially dangerous? Are those essential to its ability to outsmart humans?

emptysongglass 4y ago
I think we're asking the wrong questions. We're assuming that a. what we're developing (an associative propability machine) is in any way related to the pursuit of real artificial intelligence. We're assuming that b. motive, desire, passion, some motive force is inhered into the thing. I, personally, don't think we've seen anything that can be dangerous independent of its wielding by a badly acting human. That is to say, human agency and acting force is required on the other side of the LLM to do bad actions, like any other tool. I don't know what real AI that threatens humans with an extinction level event looks like. It's not large language models. I don't know if humans are capable of generating it. I think anthropomorphization is really this massive veil that hangs over all of us. We can't get outside of it (our essential fallacy) so we're cursed to see it everywhere.
- mitthrowaway2 4y ago
  I'm fairly certain that an LLM alone will not independently start taking actions and pursuing goals and causing harm. That's not how LLMs work, as you and I both know. I'm not confident I can say the same about, say: an LLM calling itself from within an infinite loop in which the LLM is asked to predict actions that will [eg. add money into a bank account], and then to write code that executes those actions, then to examine the results of those actions and update its plan. (This takes no additional breakthrough in AI architecture, only an increase in the reliability of GPT for it to become practical. It is basically a summary of the AutoGPT project). I don't know if LLMs are anywhere close to any part of what goes on inside the human thought process. But I am sure that evolution was able to stumble upon the human thought process. In 2003 I was quite confident that was forever beyond the reach of humans to program into a computer, but in 2023 I am far less certain. I think human cognition is gradually becoming less of a mystery.

NumberWangMan 4y ago

Wait, but we don't actually even know what is happening inside GPT-4 on a fundamental level, to produce the output we see. We don't even know what is happening inside our own brains, really. How do the neurons turning on and off produce reasoning and consciousness? So in the same way, how can you say that GPT-4 is not AI, definitively, or at least a primordial form of one? Clearly, there's some arrangement of neurons that produces intelligence. Maybe it's just a "stochastic parrot", but one can probably make a similarly dismissive-sounding and yet accurate description of how humans cognate. Sometimes quantity has its own quality. Maybe a big enough stochastic parrot becomes smart.

emptysongglass 4y ago
> Wait, but we don't actually even know what is happening inside GPT-4 on a fundamental level, to produce the output we see. This just isn't true. All of this research descends from transformer research that came from Google in 2017. We know exactly how they work. There's nothing surprising in what's going on. I spent a bit of time looking for a decent intro to all this because I wanted to at least provide a resource for people who are terrified of LLMs. I think this is a pretty decent one [1] [1] https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
- NumberWangMan 4y ago
  My understanding was that we're not sure of how some of the things that look like reasoning are coming about, but I'll gladly check it out. I may be wrong. I'm not particularly terrified of GPT-4, but a more of what comes after. Maybe GPT-6 or 8, or another breakthrough non-LLM AI. I'm not sure if reading it will do much more than push my estimate of real AGI out a couple of years, but thanks anyway, I appreciate it.
- NumberWangMan 4y ago
  Ok, after reading through it, I don't think I was speaking inaccurately. Here are some quotes from the blog post (which was very neat, by the way, thanks again) > And it’s part of the lore of neural nets that—in some sense—so long as the setup one has is “roughly right” it’s usually possible to home in on details just by doing sufficient training, without ever really needing to “understand at an engineering level” quite how the neural net has ended up configuring itself. > What determines this structure? Ultimately it’s presumably some “neural net encoding” of features of human language. But as of now, what those features might be is quite unknown. In effect, we’re “opening up the brain of ChatGPT” (or at least GPT-2) and discovering, yes, it’s complicated in there, and we don’t understand it—even though in the end it’s producing recognizable human language. This is the kind of thing I'm referring to. Even though we can look at pictures of the neuron activations, we don't really know what it's doing, any more than you can look at a picture of a brain scan of a person speaking and know why they decided to say those particular words. We know at a low level how the network works, of course, because we coded it. It's the emergent behavior that we don't understand, and that's the bit that's scary because that's where the AI risk lives. It's like how we understand particle physics pretty well, and chemistry to some degree, but biology is a massively complicated jungle that we've barely scratched the surface of. Maybe there's something we could develop analogous to using an MRI as a lie detector, for these networks. But as far as I know, this is still an unsolved problem, and apparently really hard for networks that are smarter than you are: https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H...