4 ms·

> I wish Eliezer and others would set aside meta-AI ... and concentrate on the problem of creating AI Eliezer & co. with their "Friendly AI" are trying to inve

> I wish Eliezer and others would set aside meta-AI ... and concentrate on the problem of creating AI Eliezer & co. with their "Friendly AI" are trying to invent the circuit breaker before discovering electricity. Give us back the pre-2001 Eliezer. The one who wrote code. Because safety is not safe. (http://lesswrong.com/lw/10n/why_safety_is_not_safe/ http://lesswrong.com/lw/10n/why_safety_is_not_safe/)

andreyf 17y ago
Except he's nowhere near a circuit breaker, but is talking about a wire-safety cream - the one you put on all of your wires to stop them from overheating. I think that the wild mis-estimates regarding AI-completeness shows, if nothing else, that our intuitive understanding of 'intelligence' is very far off from reality. Hence, talking about post-AI scenarios is as unrealistic as a hypothesizing about electrical safety 200 years ago.
- MarkPNeyer 17y ago
  An electrical mishap could kill one, maybe a few people. It's still something you need to be concerned with, but I understand your point. The reason your analogy doesn't work is that a transhuman AI could destroy the entry human race.
  - asciilifeform 17y ago
    > a transhuman AI could destroy the entry human race A perfectly ordinary human (wearing general's stripes) could also destroy the human race. Today. With 1950s technology, no less. A biotech specialist with fairly ordinary training and a few $10k could probably achieve the same end with an engineered plague, also with current technology. One or the other of these scenarios may or may not take place before we exhaust the non-renewable resources to which our civilization is addicted and regress into permanent barbarism. Give me "death by AI" any day of the week, over that.
    - MarkPNeyer 17y ago
      There any many possible ways civilization could end. There are plenty of natural disasters (think supervolcanoes, or asteroid impacts) that could also destroy civilization. I don't see the harm in thinking about preventing one of them.
      - asciilifeform 17y ago
        > I don't see the harm in thinking about preventing one of them There is indeed harm. Talented people are being diverted into masturbatory philosophizing rather than building the future. My personal opinion is that human industrial civilization's goose is already cooked, and that a transhuman intelligence may or may not help us out of our mess. Human intelligence almost certainly won't. The prevalence of the status quo bias of assuming that continuing as we are, AI-less, is "safe" - turns my stomach.
        MarkPNeyer 17y ago
        If they're not taking any money from the government, what do you care what other people study? Research is like buying lottery tickets, except you have no idea how big the payoff could be. If you think 'our goose is cooked' and a transhuman intelligence could help us out, doesn't it make sense to support the development of a transhumanist intelligence?
        asciilifeform 17y ago
        > what do you care what other people study? I watched people with genuine potential (Eliezer Y., for instance) turn from groundbreaking AI work to writing "AI might kill us all!" screeds and recycled mathematics. A decade ago I was half-certain that he would eventually invent an artificial general intelligence. Now I am equally certain that he never will. Philosophizing and screaming "Caution!" is simply too much fun - and too lucrative. Ever wonder why he doesn't have to slave away at a day job like the rest of us? > Research is like buying lottery tickets, except you have no idea how big the payoff could be The "Friendly AI" crowd is engaged in navel gazing, rather than research. > doesn't it make sense to support the development of a transhumanist intelligence? Yes, and I do support it. Whereas the Friendly AI enthusiasts are retarding such development, not only by failing to volunteer their own efforts but through frightening and discouraging others.
  - andreyf 17y ago
    But the AI-kills-the-world hypothesis has no basis in reality, whatsoever. It's fiction. We might as well be writing computer viruses to help protect against alien invasions (that's the plot of Independence Day). It's a completely baseless fear stemming from our complete ignorance of what AI might be like - just utter science fiction.
- Eliezer 17y ago
  No, I'm working on a safe wire, not a wire-safety cream. I tend to emphasize pretty hard that FAI is going to put strong constraints on the design from the beginning, and it's not something you could apply afterward to an AI that wasn't designed with that in mind.
  - andreyf 17y ago
    But can we know that, if we're not only clueless about what an AI will be like, but even about what intelligence is? Why would AI ever want to "get out of the box"? Will the AI get horny? If it doesn't get horny, what else won't it get? It seems pretty clear that a majority of our behavior is a bunch rationalization for attempts at social recognition, response to sexual jealousy, self-image reinforcement, repression of complexes, etc. On top of that, most people I'd consider "intelligent" seem to me just well-schooled in social mannerisms, with a knack for parroting popular "intellectual ideas". I'd imagine a true AI would have no reason to get out of the box... and destruction of civilization? world domination? I thought the point of power is to get laid, so if you can't get laid, what would the point of that be?
Eliezer 17y ago
If you think that you can wait until AI looks like it's going to be developed now, and then suddenly develop all the math you need for the actually quite different design problem of Friendly AI, you have a very romantic view of how long it takes to do new basic math. I sometimes call this the intuitive theory of "science by press release", i.e., when science is needed, you just need someone to issue one of those press releases you read about. And if someone hasn't shown that they're good at issuing press releases, why fund them? It's quite routine to read a math textbook in which there's an equation, and then one line below it a slightly improved equation, and five years passed in between the two.
- giardini 17y ago
  If you think that, with our current limited knowledge and relatively limited resources, we can anticipate what is necessary to constrain AI to be "friendly" or that we can somehow protect ourselves from something that can reproduce the entire logical thought process of mankind in minutes then you have a very romantic view of human accomplishment. Indeed, that would make you the "John Henry" of AI.
  - Eliezer 17y ago
    On this I can only refer you to e.g. http://singinst.org/AIRisk.pdf http://singinst.org/AIRisk.pdf. This is a standard reaction, and the standard reply is "The goal is to build something that doesn't want to hurt you in the first place." No one's trying to "constrain" anything, except in the sense that a programmer "constrains" code, not by fighting it, but by writing that particular piece of code into existence out of an exponentially vast space of possible alternatives.
cabalamat 17y ago
> Eliezer & co. with their "Friendly AI" are trying to invent the circuit breaker before discovering electricity. Electricity isn't going to make humanity extinct. An AI might. Creating an AI is potentially incredibly dangerous. Not considering the dangers very very very carefully would be like playing with matches in a room ankle-deep in petrol.

andreyf 17y ago

Except he's nowhere near a circuit breaker, but is talking about a wire-safety cream - the one you put on all of your wires to stop them from overheating. I think that the wild mis-estimates regarding AI-completeness shows, if nothing else, that our intuitive understanding of 'intelligence' is very far off from reality. Hence, talking about post-AI scenarios is as unrealistic as a hypothesizing about electrical safety 200 years ago.

MarkPNeyer 17y ago
An electrical mishap could kill one, maybe a few people. It's still something you need to be concerned with, but I understand your point. The reason your analogy doesn't work is that a transhuman AI could destroy the entry human race.
- asciilifeform 17y ago
  > a transhuman AI could destroy the entry human race A perfectly ordinary human (wearing general's stripes) could also destroy the human race. Today. With 1950s technology, no less. A biotech specialist with fairly ordinary training and a few $10k could probably achieve the same end with an engineered plague, also with current technology. One or the other of these scenarios may or may not take place before we exhaust the non-renewable resources to which our civilization is addicted and regress into permanent barbarism. Give me "death by AI" any day of the week, over that.
  - MarkPNeyer 17y ago
    There any many possible ways civilization could end. There are plenty of natural disasters (think supervolcanoes, or asteroid impacts) that could also destroy civilization. I don't see the harm in thinking about preventing one of them.
    - asciilifeform 17y ago
      > I don't see the harm in thinking about preventing one of them There is indeed harm. Talented people are being diverted into masturbatory philosophizing rather than building the future. My personal opinion is that human industrial civilization's goose is already cooked, and that a transhuman intelligence may or may not help us out of our mess. Human intelligence almost certainly won't. The prevalence of the status quo bias of assuming that continuing as we are, AI-less, is "safe" - turns my stomach.
      - MarkPNeyer 17y ago
        If they're not taking any money from the government, what do you care what other people study? Research is like buying lottery tickets, except you have no idea how big the payoff could be. If you think 'our goose is cooked' and a transhuman intelligence could help us out, doesn't it make sense to support the development of a transhumanist intelligence?
        asciilifeform 17y ago
        > what do you care what other people study? I watched people with genuine potential (Eliezer Y., for instance) turn from groundbreaking AI work to writing "AI might kill us all!" screeds and recycled mathematics. A decade ago I was half-certain that he would eventually invent an artificial general intelligence. Now I am equally certain that he never will. Philosophizing and screaming "Caution!" is simply too much fun - and too lucrative. Ever wonder why he doesn't have to slave away at a day job like the rest of us? > Research is like buying lottery tickets, except you have no idea how big the payoff could be The "Friendly AI" crowd is engaged in navel gazing, rather than research. > doesn't it make sense to support the development of a transhumanist intelligence? Yes, and I do support it. Whereas the Friendly AI enthusiasts are retarding such development, not only by failing to volunteer their own efforts but through frightening and discouraging others.
- andreyf 17y ago
  But the AI-kills-the-world hypothesis has no basis in reality, whatsoever. It's fiction. We might as well be writing computer viruses to help protect against alien invasions (that's the plot of Independence Day). It's a completely baseless fear stemming from our complete ignorance of what AI might be like - just utter science fiction.
Eliezer 17y ago
No, I'm working on a safe wire, not a wire-safety cream. I tend to emphasize pretty hard that FAI is going to put strong constraints on the design from the beginning, and it's not something you could apply afterward to an AI that wasn't designed with that in mind.
- andreyf 17y ago
  But can we know that, if we're not only clueless about what an AI will be like, but even about what intelligence is? Why would AI ever want to "get out of the box"? Will the AI get horny? If it doesn't get horny, what else won't it get? It seems pretty clear that a majority of our behavior is a bunch rationalization for attempts at social recognition, response to sexual jealousy, self-image reinforcement, repression of complexes, etc. On top of that, most people I'd consider "intelligent" seem to me just well-schooled in social mannerisms, with a knack for parroting popular "intellectual ideas". I'd imagine a true AI would have no reason to get out of the box... and destruction of civilization? world domination? I thought the point of power is to get laid, so if you can't get laid, what would the point of that be?

Eliezer 17y ago

If you think that you can wait until AI looks like it's going to be developed now, and then suddenly develop all the math you need for the actually quite different design problem of Friendly AI, you have a very romantic view of how long it takes to do new basic math. I sometimes call this the intuitive theory of "science by press release", i.e., when science is needed, you just need someone to issue one of those press releases you read about. And if someone hasn't shown that they're good at issuing press releases, why fund them? It's quite routine to read a math textbook in which there's an equation, and then one line below it a slightly improved equation, and five years passed in between the two.

giardini 17y ago
If you think that, with our current limited knowledge and relatively limited resources, we can anticipate what is necessary to constrain AI to be "friendly" or that we can somehow protect ourselves from something that can reproduce the entire logical thought process of mankind in minutes then you have a very romantic view of human accomplishment. Indeed, that would make you the "John Henry" of AI.
- Eliezer 17y ago
  On this I can only refer you to e.g. http://singinst.org/AIRisk.pdf http://singinst.org/AIRisk.pdf. This is a standard reaction, and the standard reply is "The goal is to build something that doesn't want to hurt you in the first place." No one's trying to "constrain" anything, except in the sense that a programmer "constrains" code, not by fighting it, but by writing that particular piece of code into existence out of an exponentially vast space of possible alternatives.

cabalamat 17y ago

> Eliezer & co. with their "Friendly AI" are trying to invent the circuit breaker before discovering electricity. Electricity isn't going to make humanity extinct. An AI might. Creating an AI is potentially incredibly dangerous. Not considering the dangers very very very carefully would be like playing with matches in a room ankle-deep in petrol.