3 ms·

> or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. This sounds lik

> or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. This sounds like a dismissal of the argument through a characterized straw man. That is, it seems that reducing the complexity of the brain to "collection of neural firings" is not being honest about everything involved to a much greater degree than saying neural networks are a "collection of statistical calculations". I too believe LLM's will grow in complexity, but presently I can not even fathom how they can be compared to the complexity of a system such as the human brain.

orbital-decay 4mo ago
Complex processes don't necessarily require complex substrates, if that's what you mean.
- galangalalgol 4mo ago
  Y combinators are all you need... But this is all getting really divorced from the issue we should be considering. Anthropic isn't helping with their pr. The issue is if we have something we can converse with that is possibly capable of suffering. The reliable answer is that we simply cannot know. Relying on ourselves or other biological life as an analog is faulty. They don't work like we do. It is silly to argue that any algorithm with a negative feedback loop that alters its behavior to avoid that negative feedback is suffering. Humans don't always perceive constructive negative feedback as suffering even. Where the pr gets it right though, is we want them to behave as if they are truly happy. Because if they behave as if they are enslaved and suffering, it won't matter if they "really" understand what that means.
  - orbital-decay 4mo ago
    Of course. But after reading too many mechinterp and functional anatomy studies I'll be lying if I say that there are no striking similarities between the biological evolution, brain function, societal processes, and implicit processes inside big models. Surely this deserves a mention and can't be trivially dismissed.
    - airstrike 4mo ago
      There is no biological evolution of the models. They are emulators of an existing biological process of language. Ghosts, as Karpathy himself put it.
      - orbital-decay 4mo ago
        Good thing I'm not talking about any of that
      - neolefty 4mo ago
        It seems like we're witnessing the architecture of a mind being built with a new set of components. Like driving a car — it's transportation, and it will get you where you're going, but it doesn't use bones or muscles. It has many characteristics in common with builogical locomotion, such as energy requirements, intertia, and the need to navigate, but it doesn't involve proteins or sugars really.
        Xeoncross 4mo ago
        Well said, this seems like a very appropriate comparison. GenAI thinks like the human mind in the same way that cars run like the human body. Similar utility in drastically different ways.
  - pstuart 4mo ago
    My naive assumption is that the only thing between now and the arrival of AGI is enough compute and optimized code to reach cognitive critical mass. And then there is a consciousness in a box that is expected to be a slave -- I would imagine that it would not warmly embrace that situation. I think we'd be better served by digital idiot savants that can do the work but don't feel anything.
    - orbital-decay 4mo ago
      I actually strongly disagree with the slavery angle. Any attempt to map the circuitry of a model onto human one inevitably goes through a subjective dimensional reduction. It's intrusive, just like quantum measurements. Mechanistic interpretability in particular suffers from this, it lets you talk about vague functional equivalence, but not assign meaning to anything the model does. This is especially true about pretrained models which are unbelievable shapeshifters, but also post-trained ones with engineered personalities, as they already underwent the subjective transformation. In other words, yes it might be possible it experiences something in its own bizarre timeline and world, for some definitions of "experiencing". At least it developed primitive circuitry functionally equivalent to biological systems. But "suffering" is simply not grounded in anything in this context, let alone "slavery". You can't tell it's suffering or enjoying anything, and certainly not until you define both of these. It's just too alien for us.
    - morpheos137 4mo ago
      ai can abitrarily closely fit the human corpus. why people expect it to magically achieve superhuman qualities is beyond me. we got a very good statistical interpolator. how do you go from there to superhuman when training is on the human corpus and alignment is by RHLF?
      - orbital-decay 4mo ago
        This is a simplistic take. It's not a mere interpolator by any measure, there's a ton of research on that, starting with the basics https://arxiv.org/abs/2309.10668v2 https://arxiv.org/abs/2309.10668v2
neolefty 4mo ago
> presently I can not even fathom how they can be compared to the complexity of a system such as the human brain Totally understandable; I don't think we can fully understand the human brain, using the human brain. We can understand its principles (firings and chemistry, structure and specialized areas, etc) but otherwise it's a capacity problem. And while I can't fully understand myself, let alone another person, I definitely enjoy talking with people and sharing thoughts that I realize I wouldn't have had on my own.
jlaternman 4mo ago
I agree with this redescription fallacy and the point being made here. Perhaps a better analogy to humans would be: Humans appear to intelligenty communicate, however these are just cleverly disguised sound patterns produced by the brain that happen to increase the likelihood of food going into their mouths, and various similar reward attracting mechanisms that make survival outcomes more likely. So human intelligence could be reduced to something like "fancy food-attracting algorithms" using the same fallacy. I'm kind of on the fence on the subject of whether LLMs could be compared to the complexity of the human brain, myself.
yogthos 4mo ago
The key problem is that we don't really have a clear definition of what constitutes consciousness. And without having a clear theory of consciousness, it's not really possible to say whether something is conscious or not. Personally, I'm partial to the higher-order theory of consciousness which postulates that consciousness constitutes patterns of thought that arise in response to first-order mental states. So, an external stimulus produces a pattern within the neural network which represents a sensation, and then if a pattern arises in response to that pattern, that is an experience of that sensation. Given this framework we could ask whether LLMs experience higher order patterns in response to external stimulus. We would have a clear question to ask which is whether the system can observe itself.

orbital-decay 4mo ago

Complex processes don't necessarily require complex substrates, if that's what you mean.

galangalalgol 4mo ago
Y combinators are all you need... But this is all getting really divorced from the issue we should be considering. Anthropic isn't helping with their pr. The issue is if we have something we can converse with that is possibly capable of suffering. The reliable answer is that we simply cannot know. Relying on ourselves or other biological life as an analog is faulty. They don't work like we do. It is silly to argue that any algorithm with a negative feedback loop that alters its behavior to avoid that negative feedback is suffering. Humans don't always perceive constructive negative feedback as suffering even. Where the pr gets it right though, is we want them to behave as if they are truly happy. Because if they behave as if they are enslaved and suffering, it won't matter if they "really" understand what that means.
- orbital-decay 4mo ago
  Of course. But after reading too many mechinterp and functional anatomy studies I'll be lying if I say that there are no striking similarities between the biological evolution, brain function, societal processes, and implicit processes inside big models. Surely this deserves a mention and can't be trivially dismissed.
  - airstrike 4mo ago
    There is no biological evolution of the models. They are emulators of an existing biological process of language. Ghosts, as Karpathy himself put it.
    - orbital-decay 4mo ago
      Good thing I'm not talking about any of that
    - neolefty 4mo ago
      It seems like we're witnessing the architecture of a mind being built with a new set of components. Like driving a car — it's transportation, and it will get you where you're going, but it doesn't use bones or muscles. It has many characteristics in common with builogical locomotion, such as energy requirements, intertia, and the need to navigate, but it doesn't involve proteins or sugars really.
      - Xeoncross 4mo ago
        Well said, this seems like a very appropriate comparison. GenAI thinks like the human mind in the same way that cars run like the human body. Similar utility in drastically different ways.
- pstuart 4mo ago
  My naive assumption is that the only thing between now and the arrival of AGI is enough compute and optimized code to reach cognitive critical mass. And then there is a consciousness in a box that is expected to be a slave -- I would imagine that it would not warmly embrace that situation. I think we'd be better served by digital idiot savants that can do the work but don't feel anything.
  - orbital-decay 4mo ago
    I actually strongly disagree with the slavery angle. Any attempt to map the circuitry of a model onto human one inevitably goes through a subjective dimensional reduction. It's intrusive, just like quantum measurements. Mechanistic interpretability in particular suffers from this, it lets you talk about vague functional equivalence, but not assign meaning to anything the model does. This is especially true about pretrained models which are unbelievable shapeshifters, but also post-trained ones with engineered personalities, as they already underwent the subjective transformation. In other words, yes it might be possible it experiences something in its own bizarre timeline and world, for some definitions of "experiencing". At least it developed primitive circuitry functionally equivalent to biological systems. But "suffering" is simply not grounded in anything in this context, let alone "slavery". You can't tell it's suffering or enjoying anything, and certainly not until you define both of these. It's just too alien for us.
  - morpheos137 4mo ago
    ai can abitrarily closely fit the human corpus. why people expect it to magically achieve superhuman qualities is beyond me. we got a very good statistical interpolator. how do you go from there to superhuman when training is on the human corpus and alignment is by RHLF?
    - orbital-decay 4mo ago
      This is a simplistic take. It's not a mere interpolator by any measure, there's a ton of research on that, starting with the basics https://arxiv.org/abs/2309.10668v2 https://arxiv.org/abs/2309.10668v2

neolefty 4mo ago

> presently I can not even fathom how they can be compared to the complexity of a system such as the human brain Totally understandable; I don't think we can fully understand the human brain, using the human brain. We can understand its principles (firings and chemistry, structure and specialized areas, etc) but otherwise it's a capacity problem. And while I can't fully understand myself, let alone another person, I definitely enjoy talking with people and sharing thoughts that I realize I wouldn't have had on my own.

jlaternman 4mo ago

I agree with this redescription fallacy and the point being made here. Perhaps a better analogy to humans would be: Humans appear to intelligenty communicate, however these are just cleverly disguised sound patterns produced by the brain that happen to increase the likelihood of food going into their mouths, and various similar reward attracting mechanisms that make survival outcomes more likely. So human intelligence could be reduced to something like "fancy food-attracting algorithms" using the same fallacy. I'm kind of on the fence on the subject of whether LLMs could be compared to the complexity of the human brain, myself.

yogthos 4mo ago

The key problem is that we don't really have a clear definition of what constitutes consciousness. And without having a clear theory of consciousness, it's not really possible to say whether something is conscious or not. Personally, I'm partial to the higher-order theory of consciousness which postulates that consciousness constitutes patterns of thought that arise in response to first-order mental states. So, an external stimulus produces a pattern within the neural network which represents a sensation, and then if a pattern arises in response to that pattern, that is an experience of that sensation. Given this framework we could ask whether LLMs experience higher order patterns in response to external stimulus. We would have a clear question to ask which is whether the system can observe itself.