5 ms·

This thought experiment on a paperclip maximizer made it very clear to me: https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/

This thought experiment on a paperclip maximizer made it very clear to me: https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer

idlewords 10y ago
The thought experiment requires you believe in an entity that has simultaneously godlike intelligence, and follows a 'goal function' expressible in a sentence. My hunch is that the two assumptions will prove incompatible. Complex entities will have complex motivations, like people do. A true AI born out of a paperclip maximizer would be more likely to spend its time writing hyper-symphonies about paperclips, and majoring in Clippy studies.
- dredmorbius 10y ago
  My read of the PM isn't that the intelligence is godlike. Only that it is very good at achieving its stated goal, and lacks a negative feedback control. On a related theme, I've been kicking around an ontology of technological mechanisms,[1] one element of which is control systems. It's somewhat underappreciated that what made complex and/or highly-scaled systems possible in many cases were systems for either regulating or mitigating processes or discrepencies between parts of systems. Swingletrees for draught teams, Watt's governor, for his steam engine (and additional tweaking to damp down oscillation tendencies), infrared temperature sensing and gas chromatography in steelmaking. We figure out how to make a thing go, then we figure out how to make it go where we want. Even high-speed rail was a bit more about quality control (rail and roadbed specification) and control (wheel truck hunting, pantograph-commutator interactions) than dumping gobs of energy into a rail-wheel system. ________________________________ Notes: 1 https://ello.co/dredmorbius/post/klsjjjzzl9plqxz-ms8nww https://ello.co/dredmorbius/post/klsjjjzzl9plqxz-ms8nww
  - idlewords 10y ago
    "Godlike" in the sense that we are so far below it, we have no hope of competing with it. I agree with your point and think we will discover that intelligent systems are by their nature full of interlocking feedback loops with negative controls, and that this is a necessary prerequisite for a system to even be intelligent.
    - dredmorbius 10y ago
      I've been poking into neurons and brain function a bit. There are many neurons which are exitatory -- they respond when stimulated by others. There are a large number which are inhibitory. They supress activity of other neurons. I'm suspecting inhibition is more critical than excitation for consciousness and intelligence.
- sbierwagen 10y ago
  Instrumental convergence: no matter what its goals are, an agent will always take actions so that it can pursue its goals. Agents that can improve themselves, and gain utility from improving themselves, will try to improve themselves. If doubling its number of CPUs will let it write more paperclip symphonies, then it'll do that.
  - idlewords 10y ago
    This is trivialy disproved just by looking at the nearest slacker. If none is nearby, I offer myself as a counterexample.
    - sbierwagen 10y ago
      Hey man, it's not my fault you've calculated the marginal utility of self improvement to be low or negative. Open a bug with the designer of your neural architecture.
      - idlewords 10y ago
        I sure will. Maybe tomorrow...

This thought experiment on a paperclip maximizer made it very clear to me: https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer

idlewords 10y ago

The thought experiment requires you believe in an entity that has simultaneously godlike intelligence, and follows a 'goal function' expressible in a sentence. My hunch is that the two assumptions will prove incompatible. Complex entities will have complex motivations, like people do. A true AI born out of a paperclip maximizer would be more likely to spend its time writing hyper-symphonies about paperclips, and majoring in Clippy studies.

dredmorbius 10y ago
My read of the PM isn't that the intelligence is godlike. Only that it is very good at achieving its stated goal, and lacks a negative feedback control. On a related theme, I've been kicking around an ontology of technological mechanisms,[1] one element of which is control systems. It's somewhat underappreciated that what made complex and/or highly-scaled systems possible in many cases were systems for either regulating or mitigating processes or discrepencies between parts of systems. Swingletrees for draught teams, Watt's governor, for his steam engine (and additional tweaking to damp down oscillation tendencies), infrared temperature sensing and gas chromatography in steelmaking. We figure out how to make a thing go, then we figure out how to make it go where we want. Even high-speed rail was a bit more about quality control (rail and roadbed specification) and control (wheel truck hunting, pantograph-commutator interactions) than dumping gobs of energy into a rail-wheel system. ________________________________ Notes: 1 https://ello.co/dredmorbius/post/klsjjjzzl9plqxz-ms8nww https://ello.co/dredmorbius/post/klsjjjzzl9plqxz-ms8nww
- idlewords 10y ago
  "Godlike" in the sense that we are so far below it, we have no hope of competing with it. I agree with your point and think we will discover that intelligent systems are by their nature full of interlocking feedback loops with negative controls, and that this is a necessary prerequisite for a system to even be intelligent.
  - dredmorbius 10y ago
    I've been poking into neurons and brain function a bit. There are many neurons which are exitatory -- they respond when stimulated by others. There are a large number which are inhibitory. They supress activity of other neurons. I'm suspecting inhibition is more critical than excitation for consciousness and intelligence.
sbierwagen 10y ago
Instrumental convergence: no matter what its goals are, an agent will always take actions so that it can pursue its goals. Agents that can improve themselves, and gain utility from improving themselves, will try to improve themselves. If doubling its number of CPUs will let it write more paperclip symphonies, then it'll do that.
- idlewords 10y ago
  This is trivialy disproved just by looking at the nearest slacker. If none is nearby, I offer myself as a counterexample.
  - sbierwagen 10y ago
    Hey man, it's not my fault you've calculated the marginal utility of self improvement to be low or negative. Open a bug with the designer of your neural architecture.
    - idlewords 10y ago
      I sure will. Maybe tomorrow...