5 ms·

> By default AI will optimize the universe to the greatest extent possible towards it's values. Hmm, I'm not sure I understand this line of thinking. Intellige

> By default AI will optimize the universe to the greatest extent possible towards it's values. Hmm, I'm not sure I understand this line of thinking. Intelligence is complex, soft, and can even conflict with itself. Why do you envision artificial intelligence as if it were a pure optimization algorithm with high-level problem solving abilities? It seems to me that a fuzzy metaheuristic with lots of competing goals would be a better comparison and this does not lend itself to the "paperclip factory" quite as readily.

TheOtherHobbes 11y ago
"Siri - make me a Dyson sphere." The irony about these conversations is that humans are already far more of a threat to humanity now than an AI might be, possibly, at some undetermined time in the future.
- JoeAltmaier 11y ago
  Not true! AI's are thinking and acting at the speed of light. We put them in charge of our infrastructure all the time (internet, phone systems, stock trading, medical records and on and on). Already they have tremendous potential to do us harm, at a rate far outstripping any given humans. They only lack the will, so far.
  - juliangregorian 11y ago
    Rule-based systems are hardly strong AI. I don't even know what you mean by "thinking and acting at the speed of light". If you have more information I'd love to hear it.
    - JoeAltmaier 11y ago
      Programmed trading on the stock market doesn't take human response times. In fact, its designed to react and communicate trades as near the speed of light as possible. AIs (that don't exist yet) are going to be in charge of controlling electronic systems (examples given). They will behave with a time-constant that is unrelated to human reaction times. They will be able to do harm, and compound that harm, at rates mere humans will be powerless to react/
      - juliangregorian 11y ago
        Got it. Hyperbole and unfounded fearmongering. Thanks for clarifying.
        JoeAltmaier 11y ago
        Only if you're afraid? I'm not. The issue was just this: AIs will operate quickly. Faster than we can react. That's the nature of electronic life vs biological. So it would be good to be ready for that.
        juliangregorian 11y ago
        And I suppose you have suggestions for how we might "be ready"?
        JoeAltmaier 11y ago
        What's with all the hate? Wishing that AIs were tame, slow, manageable tools is all fine and good; but I just pointed out that there's a very high speed-limit to how fast they'll be able to think. Why waste your venom on me?
        juliangregorian 11y ago
        This isn't a sci-fi authors message board. If all you want to do is listen to the sound of your own voice you should go elsewhere.
tgflynn 11y ago
I think the best arguments for why strong AI might be dangerous are: 1) The main approach to creating weak AI that actually works is to define some objective function and train some sufficiently complex mathematical model to maximize that objective. Hence it seems likely that if strong AI is created it will follow this paradigm and therefore such a strong AI will care about nothing other than how to maximize its objective. Such a strong AI would be nothing like a human mind so anthropomorphised thinking is of little use in reasoning about it. 2) Given a sufficiently powerful optimization algorithm (say, for example, if P == NP and an efficient algorithm for an NP hard problem were discovered) one can in fact imagine the above paradigm producing an incredibly powerful intelligence, one that could discover many sub-goals for its fundamental optimization objective which the creator of the fundamental objective could not foresee. Further it could be exceedingly effective at achieving its sub-goals, much like modern chess programs are exceedingly effective at winning at chess, but in a far more general domain. There are however reasons to doubt such a danger. 1) If P != NP then although it should in theory be possible to achieve roughly human equivalent AI, the sort of truly superhuman AI that some fear may not in fact be possible. In other words it may be that if P != NP there is some limit to how powerful an intelligence can become due to the combinatorial explosion that occurs when one tries to expand the range of possibilities that are considered. 2) Even if no such limits exist there are probably ways to maintain control over even a superintelligent AI. A powerful optimizer would be exceedingly good at finding solutions within its own model of the world but that is only part of what is needed to construct a super AI that actually interacts with the world. You would also need a process for creating and updating the world model based on empirical data. This type of architecture would probably give you several powerful levers for controlling the AI and you could always let the optimizer run on the world model to check on its behavior before allowing it to act on the real world.
Houshalter 11y ago
Even if you start off with messy complicated AI, it could modify itself to become a more efficient optimizer. And it would do that because it would be beneficial for whatever set of values it has, no matter how complicated. Regardless I don't know why you think a messy AI is safe. Having a really complicated utility function doesn't mean it is going to result in anything desirable for humans.

TheOtherHobbes 11y ago

"Siri - make me a Dyson sphere." The irony about these conversations is that humans are already far more of a threat to humanity now than an AI might be, possibly, at some undetermined time in the future.

JoeAltmaier 11y ago
Not true! AI's are thinking and acting at the speed of light. We put them in charge of our infrastructure all the time (internet, phone systems, stock trading, medical records and on and on). Already they have tremendous potential to do us harm, at a rate far outstripping any given humans. They only lack the will, so far.
- juliangregorian 11y ago
  Rule-based systems are hardly strong AI. I don't even know what you mean by "thinking and acting at the speed of light". If you have more information I'd love to hear it.
  - JoeAltmaier 11y ago
    Programmed trading on the stock market doesn't take human response times. In fact, its designed to react and communicate trades as near the speed of light as possible. AIs (that don't exist yet) are going to be in charge of controlling electronic systems (examples given). They will behave with a time-constant that is unrelated to human reaction times. They will be able to do harm, and compound that harm, at rates mere humans will be powerless to react/
    - juliangregorian 11y ago
      Got it. Hyperbole and unfounded fearmongering. Thanks for clarifying.
      - JoeAltmaier 11y ago
        Only if you're afraid? I'm not. The issue was just this: AIs will operate quickly. Faster than we can react. That's the nature of electronic life vs biological. So it would be good to be ready for that.
        juliangregorian 11y ago
        And I suppose you have suggestions for how we might "be ready"?
        JoeAltmaier 11y ago
        What's with all the hate? Wishing that AIs were tame, slow, manageable tools is all fine and good; but I just pointed out that there's a very high speed-limit to how fast they'll be able to think. Why waste your venom on me?
        juliangregorian 11y ago
        This isn't a sci-fi authors message board. If all you want to do is listen to the sound of your own voice you should go elsewhere.

tgflynn 11y ago

I think the best arguments for why strong AI might be dangerous are: 1) The main approach to creating weak AI that actually works is to define some objective function and train some sufficiently complex mathematical model to maximize that objective. Hence it seems likely that if strong AI is created it will follow this paradigm and therefore such a strong AI will care about nothing other than how to maximize its objective. Such a strong AI would be nothing like a human mind so anthropomorphised thinking is of little use in reasoning about it. 2) Given a sufficiently powerful optimization algorithm (say, for example, if P == NP and an efficient algorithm for an NP hard problem were discovered) one can in fact imagine the above paradigm producing an incredibly powerful intelligence, one that could discover many sub-goals for its fundamental optimization objective which the creator of the fundamental objective could not foresee. Further it could be exceedingly effective at achieving its sub-goals, much like modern chess programs are exceedingly effective at winning at chess, but in a far more general domain. There are however reasons to doubt such a danger. 1) If P != NP then although it should in theory be possible to achieve roughly human equivalent AI, the sort of truly superhuman AI that some fear may not in fact be possible. In other words it may be that if P != NP there is some limit to how powerful an intelligence can become due to the combinatorial explosion that occurs when one tries to expand the range of possibilities that are considered. 2) Even if no such limits exist there are probably ways to maintain control over even a superintelligent AI. A powerful optimizer would be exceedingly good at finding solutions within its own model of the world but that is only part of what is needed to construct a super AI that actually interacts with the world. You would also need a process for creating and updating the world model based on empirical data. This type of architecture would probably give you several powerful levers for controlling the AI and you could always let the optimizer run on the world model to check on its behavior before allowing it to act on the real world.

Houshalter 11y ago

Even if you start off with messy complicated AI, it could modify itself to become a more efficient optimizer. And it would do that because it would be beneficial for whatever set of values it has, no matter how complicated. Regardless I don't know why you think a messy AI is safe. Having a really complicated utility function doesn't mean it is going to result in anything desirable for humans.