4 ms·

Should we worry if it achieves sentience? Any reason why we shouldn't?

nomel 4y ago
I would assume consciousness/sentience requires significant feedback loops, so the "thought process" can "keep going". I don't think any of these models have real feedback loops.
- bioemerl 4y ago
  They do during training, but not when you use them
Planktonne 4y ago
No more than you should worry about your phone, your toaster, or a boulder suddenly becoming self-aware. Extremely theoretically, a consciousness could spontaneously form at any point [1]. In practice, there is no reason to worry about this - it's not a likely event. There is nothing about this model (or any other one in existence) that increases the likelihood of it achieving sentience when compared to anything else ever. [1] https://en.wikipedia.org/wiki/Boltzmann_brain https://en.wikipedia.org/wiki/Boltzmann_brain
MichaelCollins 4y ago
Assuming the premise of this achieving sentience... > Should we worry if it achieves sentience? No, because it's easy to kill.
- taeric 4y ago
  So should we worry if it learns to propagate itself? :)
  - MichaelCollins 4y ago
    I don't really buy the whole "it teaches itself to hack itself out" idea. I think a sentient AI would be able to introspect and manipulate its operating environment if it was specifically designed to have that capability. Another scenario might be that somebody installs the sentient AI inside an drone submarine with nuclear armed SLBMs. You can contrive scenarios where the AI is given capabilities that make it hard to kill. But baring such scenarios, if the program scares you then just SIGKILL it.
    - taeric 4y ago
      I meant that solely as a joke. I can kind of buy that someday systems can replicate themselves. But I don't see that on any near horizon; such that I agree with you on this.
kumarvvr 4y ago
Depends on what you define sentience as. Is it the presence of a soul, or the presence of capability to learn stuff it is not trained for?
ctoth 4y ago
Probably, for ethical reasons. but we should worry a lot more if it achieves sapience. I'm also pretty sure that anything (sapience, sentience, whatever) requires closing the loop so the perceiver can perceive itself perceiving, though the idea that I am merely a single Boltzmannesque forward-pass of a language model will certainly run in the back of my head as I'm trying to drift off to sleep tonight.

I would assume consciousness/sentience requires significant feedback loops, so the "thought process" can "keep going". I don't think any of these models have real feedback loops.

bioemerl 4y ago
They do during training, but not when you use them

Planktonne 4y ago

No more than you should worry about your phone, your toaster, or a boulder suddenly becoming self-aware. Extremely theoretically, a consciousness could spontaneously form at any point [1]. In practice, there is no reason to worry about this - it's not a likely event. There is nothing about this model (or any other one in existence) that increases the likelihood of it achieving sentience when compared to anything else ever. [1] https://en.wikipedia.org/wiki/Boltzmann_brain https://en.wikipedia.org/wiki/Boltzmann_brain

MichaelCollins 4y ago

Assuming the premise of this achieving sentience... > Should we worry if it achieves sentience? No, because it's easy to kill.

taeric 4y ago
So should we worry if it learns to propagate itself? :)
- MichaelCollins 4y ago
  I don't really buy the whole "it teaches itself to hack itself out" idea. I think a sentient AI would be able to introspect and manipulate its operating environment if it was specifically designed to have that capability. Another scenario might be that somebody installs the sentient AI inside an drone submarine with nuclear armed SLBMs. You can contrive scenarios where the AI is given capabilities that make it hard to kill. But baring such scenarios, if the program scares you then just SIGKILL it.
  - taeric 4y ago
    I meant that solely as a joke. I can kind of buy that someday systems can replicate themselves. But I don't see that on any near horizon; such that I agree with you on this.

kumarvvr 4y ago

Depends on what you define sentience as. Is it the presence of a soul, or the presence of capability to learn stuff it is not trained for?

ctoth 4y ago

Probably, for ethical reasons. but we should worry a lot more if it achieves sapience. I'm also pretty sure that anything (sapience, sentience, whatever) requires closing the loop so the perceiver can perceive itself perceiving, though the idea that I am merely a single Boltzmannesque forward-pass of a language model will certainly run in the back of my head as I'm trying to drift off to sleep tonight.