4 ms·

ChatGPT is a glorified word predictor. It isn’t sentient. It doesn’t know what it’s saying, and yes, you can coax it into admitting that it wants to take over t

by calf 4y ago

ChatGPT is a glorified word predictor. It isn’t sentient. It doesn’t know what it’s saying, and yes, you can coax it into admitting that it wants to take over the world or saying hurtful things (although it was specially conditioned during training to try to suppress such output). It’s simply stringing words together using an expansive statistical model built from billions of sentences. Is this true though? The public debate albeit poorly explained by many, is whether the emergent behaviors users are seeing are caused by emergent algorithms and structures arising in the neural network. So for example some scientists claim that they can find fragments of syntax trees or grammars that the neural network emergently constructs. That would point to higher-level phenomena going on inside ChatGPT and its ilk, than merely statistics and predictions. I'm curious as to the answer but it's not implausible to me that there's stuff happening on two levels of abstraction at the same time. Analogous to hardware/software abstraction, nobody says a Mac running Safari is a glorified Boolean circuit. I don't know the answer but it's not implausible, or maybe I don't know enough about machine learning to understand the author's quote above.

missingdays 4y ago
Of course Mac running Safari is a glorified boolean circuit. It's programmed in a way to show the pixels on the screen in a way that you understand, and take inputs from your mouse and keyboard in a way you understand
- jstanley 4y ago
  It is true that a Mac running Safari is just a glorified boolean circuit. It is also true that a human being is just a glorified bunch of subatomic particles. It's not a useful way to understand what's going on. There is a higher-level structure built out of the lower-level pieces, and it is the higher-level structure that is important and not the lower-level pieces. It would be possible to make a web browser out of a different type of logic circuit. It's the higher-level structure of the browser that matters, and not the fact that it is built out of boolean logic. Similarly, with ChatGPT, it is the higher-level structures (whatever they may be) that matter, and not the low-level details of the neural network. The higher-level structures could be far too complex for us to understand.
- swid 4y ago
  I’m sure you don’t think it follows a human is a glorified rock? Both objects are processed by the universe under the same rules, and understanding how consciousness arises directly from physics is likely intractable.
scotty79 4y ago
It's pretty much the truth. What the ChatGPT is good at is "keeping in mind" various associations between words that occurred in the session so far. To keep those associations some internal structure bound to get conjured. It doesn't mean the transformer understands anything or can do any kind of reasoning, despite the fact that it can mimic a bit how reasoning output looks like and even get it right sometimes if the context is fairly close to something it seen in the training data.
feanaro 4y ago
> That would point to higher-level phenomena going on inside ChatGPT and its ilk, than merely statistics and predictions. No, it wouldn't, because nothing in "higher-level phenomena" precludes it being caused by statistics and predictions.
- calf 4y ago
  I don't understand that. In computer science everyone learned that computation is best described and explained at several levels of abstraction. E.g., HW/SW interface; machine code vs C++; RTL vs architecture, the list of levels of abstractions goes on and on. So what is the reason for not appropriately extending this idea to analyzing whatever a neural network is doing?
  - unlikelymordant 4y ago
    I look at in the following way: understanding something by abstracting over lower level details doesn't mean the abstraction is how things actually work, the extra layer of abstraction may just be a nice way of thinking about something that makes thinking about it easier. But in the end the true mechanics are the sum of low level details. In general abstractions are not perfect, hence 'leaky abstractions'.
    - calf 4y ago
      That's a philosophical argument, whereas people are implicitly denying that ChatGPT can have higher level abstraction at all as if it were valid to claim that Stroustrup's contribution to C++ was ultimately just a way to write a lot of assembly code, so let's forever only think in assembly terms. (As a separate tangent, I don't accept the philosophy that abstractions are merely human niceties or conveniences. They are information theoretic models of reality and can be tested and validated, after all, even the bottom level of reality is an abstraction. The very argument used to deny the primacy of abstractions itself requires conceptual abstractions, leading to a circular logic. But then, I'm not a philosopher so what do I know.)
      - 4y ago
m3kw9 4y ago
It isn’t sentient because it stops when there is no input. If OpenAI were implementing a continuous feedback loop of not just textual info, but one that can sense the world like we do(visual, spatial, audio, tactile) and it keeps improves as it interacts with other people and world. It gets closer.
- m3kw9 4y ago
  Humans also has continuous input from our body itself all the organs, cellular etc all giving the brain continuous input
- abecedarius 4y ago
  Careful. You just proved that a human whole brain emulation is not sentient.
tarsinge 4y ago
I don’t see the problem with LLM having a world model and superhuman intelligence without sentience. It seems very unlikely sentience comes from computation and that it’s not a physical property: why would only certain Boolean or mathematical operations in a certain order make your pen, piece of paper or CPU see colors or hear sounds? That the operations you do follow a complex plan is irrelevant. Conversely if you do enough mindfulness you can attain a state of complete empty mind / "no computation" while still being physically there, and it would not be surprising some animals with a very limited intelligence and world model have sentience. The burden of proof is on sentience not being some kind of fundamental property of matter or EM fields for me.

by calf 4y ago

missingdays 4y ago

Of course Mac running Safari is a glorified boolean circuit. It's programmed in a way to show the pixels on the screen in a way that you understand, and take inputs from your mouse and keyboard in a way you understand

jstanley 4y ago
It is true that a Mac running Safari is just a glorified boolean circuit. It is also true that a human being is just a glorified bunch of subatomic particles. It's not a useful way to understand what's going on. There is a higher-level structure built out of the lower-level pieces, and it is the higher-level structure that is important and not the lower-level pieces. It would be possible to make a web browser out of a different type of logic circuit. It's the higher-level structure of the browser that matters, and not the fact that it is built out of boolean logic. Similarly, with ChatGPT, it is the higher-level structures (whatever they may be) that matter, and not the low-level details of the neural network. The higher-level structures could be far too complex for us to understand.
swid 4y ago
I’m sure you don’t think it follows a human is a glorified rock? Both objects are processed by the universe under the same rules, and understanding how consciousness arises directly from physics is likely intractable.

scotty79 4y ago

It's pretty much the truth. What the ChatGPT is good at is "keeping in mind" various associations between words that occurred in the session so far. To keep those associations some internal structure bound to get conjured. It doesn't mean the transformer understands anything or can do any kind of reasoning, despite the fact that it can mimic a bit how reasoning output looks like and even get it right sometimes if the context is fairly close to something it seen in the training data.

feanaro 4y ago

> That would point to higher-level phenomena going on inside ChatGPT and its ilk, than merely statistics and predictions. No, it wouldn't, because nothing in "higher-level phenomena" precludes it being caused by statistics and predictions.

calf 4y ago
I don't understand that. In computer science everyone learned that computation is best described and explained at several levels of abstraction. E.g., HW/SW interface; machine code vs C++; RTL vs architecture, the list of levels of abstractions goes on and on. So what is the reason for not appropriately extending this idea to analyzing whatever a neural network is doing?
- unlikelymordant 4y ago
  I look at in the following way: understanding something by abstracting over lower level details doesn't mean the abstraction is how things actually work, the extra layer of abstraction may just be a nice way of thinking about something that makes thinking about it easier. But in the end the true mechanics are the sum of low level details. In general abstractions are not perfect, hence 'leaky abstractions'.
  - calf 4y ago
    That's a philosophical argument, whereas people are implicitly denying that ChatGPT can have higher level abstraction at all as if it were valid to claim that Stroustrup's contribution to C++ was ultimately just a way to write a lot of assembly code, so let's forever only think in assembly terms. (As a separate tangent, I don't accept the philosophy that abstractions are merely human niceties or conveniences. They are information theoretic models of reality and can be tested and validated, after all, even the bottom level of reality is an abstraction. The very argument used to deny the primacy of abstractions itself requires conceptual abstractions, leading to a circular logic. But then, I'm not a philosopher so what do I know.)
    - 4y ago

m3kw9 4y ago

It isn’t sentient because it stops when there is no input. If OpenAI were implementing a continuous feedback loop of not just textual info, but one that can sense the world like we do(visual, spatial, audio, tactile) and it keeps improves as it interacts with other people and world. It gets closer.

m3kw9 4y ago
Humans also has continuous input from our body itself all the organs, cellular etc all giving the brain continuous input
abecedarius 4y ago
Careful. You just proved that a human whole brain emulation is not sentient.

tarsinge 4y ago

I don’t see the problem with LLM having a world model and superhuman intelligence without sentience. It seems very unlikely sentience comes from computation and that it’s not a physical property: why would only certain Boolean or mathematical operations in a certain order make your pen, piece of paper or CPU see colors or hear sounds? That the operations you do follow a complex plan is irrelevant. Conversely if you do enough mindfulness you can attain a state of complete empty mind / "no computation" while still being physically there, and it would not be surprising some animals with a very limited intelligence and world model have sentience. The burden of proof is on sentience not being some kind of fundamental property of matter or EM fields for me.