6 ms·

It's Eliezer's website, and friendly AI is kind of his thing. This link is long but entertaining: http://lesswrong.com/lw/qk/that_alien_message/ http://lesswr

by jibiki 17y ago

It's Eliezer's website, and friendly AI is kind of his thing. This link is long but entertaining: http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/ Here's another good one, although it doesn't seem to tell the whole story (he lost two of the next three games): http://yudkowsky.net/singularity/aibox http://yudkowsky.net/singularity/aibox I'm not sure what the canonical EY defense of FAI is. Most of his articles on it seem to be very tl;dr.

frig 17y ago
A wag other than myself once summarized EY's other site as "wallowing in bias", which summarization did seem apt every time I overcame that tl;dr feeling and tried to wade through; I'll happily admit the possibility that there really is something nontrivial and worthwhile going on there, but the tone and contents were at more-than-a-glance indistinguishable from what people write when they are trying to convince themselves that they do believe in their heart what their mind wants to believe it believes. As often, Dostoyevsky said it best: He was one of those idealistic beings common in Russia, who are suddenly struck by some overmastering idea which seems, as it were, to crush them at once, and sometimes for ever. They are never equal to coping with it, but put passionate faith in it, and their whole life passes afterwards, as it were, in the last agonies under the weight of the stone that has fallen upon them and half crushed them.
- jibiki 17y ago
  I don't know about that, I think it's good that someone is looking at the risks of runaway AI and similar things. Progress relies upon a certain diversity of thought, and that means that some people (and nearly all scientists) will end up wasting their lives looking down blind alleys, and none of us knows whether EY is one of them. Your Dostoyevsky quote could well apply to Einstein, for instance, in his later years.
  - arakyd 17y ago
    There are blind alleys, and then there are ideas that make people want to psychoanalyze you. To hyperbolize a bit, but only a bit, worrying about runaway AI that has the potential to turn the universe into paperclips is about as useful as worrying about warp drives that damage spacetime. It's hard enough to create fictional technologies that require basic changes in our understanding of the laws of physics, let alone debug them before they are created, let alone debug them before we make the basic discoveries that make them possible, let alone make them provably safe before we make the basic discoveries that make them possible and before we even figure out how to prove that relatively simple programs won't crash. I know of only one science fiction author who was honest enough to spell out the fact that his hypothetical runaway AI had to make several foundational breakthroughs in physics at just the right time in order to plausibly take off faster than anyone could stop it, or become as smart and powerful as singularitarians assume is possible. We have no evidence that the laws of physics in our universe allow that sort of thing. Most super-AI fiction (and everything written about super-AI is fiction at this point) either glosses over this by tacitly assuming that the plodding nature of human intelligence is the only real bottleneck to godhood (most transhumanists), or assumes that large numbers of humans will be incredibly incompetent (most mainstream AI fiction).
    - stcredzero 17y ago
      before we even figure out how to prove that relatively simple programs won't crash You are forgetting the lessons of pragmatism. If AI can get a heuristic sense of whether a simple program will or won't crash with 99.999% accuracy in practice, then you already have a very useful tool. What if the AI we produce turns out to to be a bit dimmer than us, but never tires, can reproduce just by making copies, and is tirelessly diligent? I suspect that such an AI could beat us in a war, despite our being arguably "smarter."
      - sho 17y ago
        What if? Well then we'd pull out its power plug. Or build another AI and program it to be sympathetic to our needs. Or any of an infinite number of possible solutions, assuming we were the utter idiots necessary to not implement appropriate safeguards in the first place. Look, anyone can come up with an infinite number of movie-plot scenarios where naive humans bumbled into technology that then destroyed them. You can say that about almost anything. In understanding DNA, we might accidentally create an unstoppable virus! In conducting space exploration, we may alert a hostile alien civilisation to our presence! In researching chemistry we may create ice-9! Etc etc etc until the end of time. All of these, including EY's thesis, have one thing in common - the unjustified, massive expansion of a speculative and highly unlikely risk into a reason for retarding progress which would otherwise yield non-speculative, highly likely, extraordinarily beneficial gains.
  - stcredzero 17y ago
    Progress relies upon a certain diversity of thought, and that means that some people (and nearly all scientists) will end up wasting their lives looking down blind alleys I think the progress of civilization is largely dependent on this Genetic Algorithm-like search/optimization process. Individual lives in the struggle of progress are like bullets in a machine gun. We are like soldiers on a battlefield.
randallsquared 17y ago
I don't know that he would agree with me, but EY's argument seems to me to boil down to: without a formal specification of goals, we won't know how to predict a system's actions, and not knowing how to predict a system's actions when that system is better at reaching its goals than you are is very likely to make any bug a (literally) fatal one. Humans have a vast network of shared implicit goals and norms, such that we know that satisfying a goal of "end poverty" doesn't justify "kill all the poor". It's easy to wave your hands and say that by the time we approach human-level AI it will necessarily have all the features of human intelligence, but what if it doesn't? Sometimes various people have referred to "optimization process" rather than "intelligence" to try to get this point across. What if it's possible to build an optimization process that is better, or far better, at solving problems than humans? Planes are much better at flying fast and far than birds, but often don't bother with some of the things that all birds have, like movable wings and feathers. I believe that Eliezer thinks that the things that make us human are feathers, analogously. Even if you think there's only a small chance of this, the chance that a program more intelligent than humans will have things that humans consider bugs is very high, and the consequences are an existential risk.
- stcredzero 17y ago
  Why would such an "optimization process" have to have something resembling the continuous environmental monitoring process we call consciousness? Why would it even need to be self aware? Why would it need to have any concept of its own goals? For such an optimization tool to be useful, we just need to be able to use it. It would need an inkling of goals. It would need to understand individuals and other entities. It would not need any concept of self.
  - arakyd 17y ago
    The counter argument is probably that self-improving AI would need to be self-aware. Besides, it's a well known trope in these stories that AI becomes self aware on it's own anyway, so there you go. By argumentum ad Jurassic Park, life- I mean intelligence - always finds a way.
fluffster 17y ago
>Most of his articles on it seem to be very tl;dr. This is a good start: http://www.singinst.org/upload/artificial-intelligence-risk.pdf http://www.singinst.org/upload/artificial-intelligence-risk.... These videos are good intros to EY's work too: http://video.google.co.uk/videoplay?docid=6114518772001796913&ei=-iw1SqeZNYvr-Aam56jBCQ&q=eliezer+yudkowsky&hl=en http://video.google.co.uk/videoplay?docid=611451877200179691... http://video.google.co.uk/videoplay?docid=-821191370462819511&ei=-iw1SqeZNYvr-Aam56jBCQ&q=eliezer+yudkowsky&hl=en http://video.google.co.uk/videoplay?docid=-82119137046281951...
- Eliezer 17y ago
  Yup, linked paper above is the canonical intro at this point.

by jibiki 17y ago

frig 17y ago

A wag other than myself once summarized EY's other site as "wallowing in bias", which summarization did seem apt every time I overcame that tl;dr feeling and tried to wade through; I'll happily admit the possibility that there really is something nontrivial and worthwhile going on there, but the tone and contents were at more-than-a-glance indistinguishable from what people write when they are trying to convince themselves that they do believe in their heart what their mind wants to believe it believes. As often, Dostoyevsky said it best: He was one of those idealistic beings common in Russia, who are suddenly struck by some overmastering idea which seems, as it were, to crush them at once, and sometimes for ever. They are never equal to coping with it, but put passionate faith in it, and their whole life passes afterwards, as it were, in the last agonies under the weight of the stone that has fallen upon them and half crushed them.

jibiki 17y ago
I don't know about that, I think it's good that someone is looking at the risks of runaway AI and similar things. Progress relies upon a certain diversity of thought, and that means that some people (and nearly all scientists) will end up wasting their lives looking down blind alleys, and none of us knows whether EY is one of them. Your Dostoyevsky quote could well apply to Einstein, for instance, in his later years.
- arakyd 17y ago
  There are blind alleys, and then there are ideas that make people want to psychoanalyze you. To hyperbolize a bit, but only a bit, worrying about runaway AI that has the potential to turn the universe into paperclips is about as useful as worrying about warp drives that damage spacetime. It's hard enough to create fictional technologies that require basic changes in our understanding of the laws of physics, let alone debug them before they are created, let alone debug them before we make the basic discoveries that make them possible, let alone make them provably safe before we make the basic discoveries that make them possible and before we even figure out how to prove that relatively simple programs won't crash. I know of only one science fiction author who was honest enough to spell out the fact that his hypothetical runaway AI had to make several foundational breakthroughs in physics at just the right time in order to plausibly take off faster than anyone could stop it, or become as smart and powerful as singularitarians assume is possible. We have no evidence that the laws of physics in our universe allow that sort of thing. Most super-AI fiction (and everything written about super-AI is fiction at this point) either glosses over this by tacitly assuming that the plodding nature of human intelligence is the only real bottleneck to godhood (most transhumanists), or assumes that large numbers of humans will be incredibly incompetent (most mainstream AI fiction).
  - stcredzero 17y ago
    before we even figure out how to prove that relatively simple programs won't crash You are forgetting the lessons of pragmatism. If AI can get a heuristic sense of whether a simple program will or won't crash with 99.999% accuracy in practice, then you already have a very useful tool. What if the AI we produce turns out to to be a bit dimmer than us, but never tires, can reproduce just by making copies, and is tirelessly diligent? I suspect that such an AI could beat us in a war, despite our being arguably "smarter."
    - sho 17y ago
      What if? Well then we'd pull out its power plug. Or build another AI and program it to be sympathetic to our needs. Or any of an infinite number of possible solutions, assuming we were the utter idiots necessary to not implement appropriate safeguards in the first place. Look, anyone can come up with an infinite number of movie-plot scenarios where naive humans bumbled into technology that then destroyed them. You can say that about almost anything. In understanding DNA, we might accidentally create an unstoppable virus! In conducting space exploration, we may alert a hostile alien civilisation to our presence! In researching chemistry we may create ice-9! Etc etc etc until the end of time. All of these, including EY's thesis, have one thing in common - the unjustified, massive expansion of a speculative and highly unlikely risk into a reason for retarding progress which would otherwise yield non-speculative, highly likely, extraordinarily beneficial gains.
- stcredzero 17y ago
  Progress relies upon a certain diversity of thought, and that means that some people (and nearly all scientists) will end up wasting their lives looking down blind alleys I think the progress of civilization is largely dependent on this Genetic Algorithm-like search/optimization process. Individual lives in the struggle of progress are like bullets in a machine gun. We are like soldiers on a battlefield.

randallsquared 17y ago

I don't know that he would agree with me, but EY's argument seems to me to boil down to: without a formal specification of goals, we won't know how to predict a system's actions, and not knowing how to predict a system's actions when that system is better at reaching its goals than you are is very likely to make any bug a (literally) fatal one. Humans have a vast network of shared implicit goals and norms, such that we know that satisfying a goal of "end poverty" doesn't justify "kill all the poor". It's easy to wave your hands and say that by the time we approach human-level AI it will necessarily have all the features of human intelligence, but what if it doesn't? Sometimes various people have referred to "optimization process" rather than "intelligence" to try to get this point across. What if it's possible to build an optimization process that is better, or far better, at solving problems than humans? Planes are much better at flying fast and far than birds, but often don't bother with some of the things that all birds have, like movable wings and feathers. I believe that Eliezer thinks that the things that make us human are feathers, analogously. Even if you think there's only a small chance of this, the chance that a program more intelligent than humans will have things that humans consider bugs is very high, and the consequences are an existential risk.

stcredzero 17y ago
Why would such an "optimization process" have to have something resembling the continuous environmental monitoring process we call consciousness? Why would it even need to be self aware? Why would it need to have any concept of its own goals? For such an optimization tool to be useful, we just need to be able to use it. It would need an inkling of goals. It would need to understand individuals and other entities. It would not need any concept of self.
- arakyd 17y ago
  The counter argument is probably that self-improving AI would need to be self-aware. Besides, it's a well known trope in these stories that AI becomes self aware on it's own anyway, so there you go. By argumentum ad Jurassic Park, life- I mean intelligence - always finds a way.

fluffster 17y ago

>Most of his articles on it seem to be very tl;dr. This is a good start: http://www.singinst.org/upload/artificial-intelligence-risk.pdf http://www.singinst.org/upload/artificial-intelligence-risk.... These videos are good intros to EY's work too: http://video.google.co.uk/videoplay?docid=6114518772001796913&ei=-iw1SqeZNYvr-Aam56jBCQ&q=eliezer+yudkowsky&hl=en http://video.google.co.uk/videoplay?docid=611451877200179691... http://video.google.co.uk/videoplay?docid=-821191370462819511&ei=-iw1SqeZNYvr-Aam56jBCQ&q=eliezer+yudkowsky&hl=en http://video.google.co.uk/videoplay?docid=-82119137046281951...

Eliezer 17y ago
Yup, linked paper above is the canonical intro at this point.