18 ms·

The Myth of AI – Jaron Lanier

139 points by discreteevent 11y ago

circlefavshape 11y ago
Am I missing something or does the first reply (by George Church) completely ignore everything the original article says?
- JetSetWilly 11y ago
  I don't think you are, he seems to respond with the predictable hyperbolic future-boosting expected of futurologists, "singulatarians" etc. Everything is all "exponential growth" and inevitable transformation right-around-the-corner. Except it isn't. And this ignored every point that jaron was trying to make and falls into all the traps he was trying to point out. It is an oblivious comment. I must say, I don't find most of the contributors to edge particular insightful.
- AndyMcConachie 11y ago
  I agree. It seems he doesn't really get it.
- natch 11y ago
  Not only the first reply. Going through the replies, most of them do not seem to have read. Pretty disappointed with the Edge here. It's clear that the participants are just exploiting it as a platform to put forward their own ideas.
VLM 11y ago
"There has been a domineering subculture—that's been the most wealthy, prolific, and influential subculture in the technical world—that for a long time has not only promoted the idea that there's an equivalence between algorithms and life, and certain algorithms and people, but a historical determinism that we're inevitably making computers that will be smarter and better than us and will take over from us." On a different discussion site I recently pointed out there is a similar religious belief or religious dogma in the idea of the self driving car, that went over like a lead balloon. People acting in a religious manner and expressing religious belief don't like to have that pointed out to them. Probably vestigial monotheism, if they act religiously at their traditional church on Sundays they don't like it pointed out that they worship a different altar online, or the atheists get really wound up when they are called out for joining the atheist movement but acting as deacons of the church of the self driving car of the future. Anyway just saying its a line of reasoning, that while true, can only lead to unpleasantness. Like discussion of scientific racial / gender differences or discussions about IQ, all you're gonna get is social signalling screaming "see no evil".
- randallsquared 11y ago
  While I don't agree with Lanier, being an AI skeptic isn't quite the same as saying that self driving cars can't happen: we don't yet have a good understanding of either consciousness or tractable general optimization algorithms, so it's still possible that there's something that will make it impossible. Unlikely, perhaps, but possible. Also, your sarcasm about the self driving car might have been too subtle. ;)
  - ghaff 11y ago
    What you see with respect to self-driving cars IMO is a lot of people who SO want them to exist that they assign timelines that are so optimistic they'd be rolling on the floor laughing if applied to projects they had direct familiarity with. Of course, it's also fueled by a lot of hype and hand-wavy statements about exponential growth and the like. Like many other aspects of AI, I'm not sure just about anyone thinks we need to have strong AI to make self-driving cars work under the vast majority of conditions. We "just" need to create automation that can handle a reasonable number of corner cases without flipping out or grinding to a halt. But that's a debate about engineering timelines not something more philosophical.
    - randallsquared 11y ago
      I'm a little confused. We have self-driving cars. Of course there are still corner cases and bugs and regulatory issues, but cars have driven for fair distances on existing roads without a human driving. I had assumed, therefore, that the point @VLM was making was that, like self-driving cars, we already have AI by many of the standards of earlier AI researchers (the "when it works, it's not called AI any longer" argument).
      - ghaff 11y ago
        I was reading "self-driving cars" as "Level 4 autonomy" under general conditions. Which IMO is many decades away. But we're in violent agreement about the "when it works, it's not called AI any longer" argument.
moron4hire 11y ago
God, what a lucid, piercing look at reality and the human condition he has. The whole part on translators, just brilliant.
kfk 11y ago
Well, at a logical level, it's one of the biggest existential threats, if 1. it can be done; 2. humans are intelligent enough to pull it off. There is a non 0 chance of AI (and related existential threat), as there is a non 0 chance of a meteorite coming down and wiping all of us off the Earth. But my guess is that most of us would say "YES" to a meteorite deviation project, while "Meh" to any attempt to better understand the AI problem.
- conjectures 11y ago
  Compared to nuclear war, superbugs, volcanoes, meteorites or catastrophic climate change, naughty-AI is a very distant threat. It's like worrying the geneticists are going to breed Ridley Scott's alien sometime in the relevant future. The world is actually pretty messed up with problems right now; some of which we could actually do something meaningful about. Continuing the naughty-AI bunfight isn't one of them.
  - Houshalter 11y ago
    I think it's far more of a threat than any of those things. Nuclear war isn't inevitable, and wouldn't destroy the entire world, just the countries at war. Even a ridiculous world wide nuclear war would leave tons of survivors. Volcanoes and meteorites are so unlikely they aren't even worth mentioning. Superbugs would have to be intentionally engineered by a malicious group with a lot of resources, and so are very unlikely. AI is inevitable. It's very likely to happen in our lifetime. And it's very likely to kill us all. And explaining every reason why I believe those things could fill an entire book, so I seriously recommend reading Superintelligence by Nick Bostrom.
    - stepvhen 11y ago
      >And it's very likely to kill us all. This is absolutely absurd. Could you elaborate on that instead of just pointing to a book?
      - Houshalter 11y ago
tsunamifury 11y ago
I think Jaron knows what many engineers who have worked on AI know: That the external promise of AI is far more inflated than the capability of the actual technology. It's simply that when the current technology gets just complicated enough for the above-average person to no longer understand, everyone starts assigning magical properties and expectations to it. This results in short term over-valuations, which inevitably lead to disappointment [1]. Jaron has observed this several times and rightly seems to be tired of repeating the same naive cycle. As I'm just now entering my second cycle and watching tech repeat itself again -- I'm beginning to understand his weariness. [1] https://hbr.org/2015/12/the-overvaluation-trap https://hbr.org/2015/12/the-overvaluation-trap
- ryanmarsh 11y ago
  Relevant: Clark's Third Law "Any sufficiently advanced technology is indistinguishable from magic."
- api 11y ago
  This drives a long cycle of AI booms and AI winters. Right now we seem to be in another boom. When the limits of deep learning and other new techniques become apparent, we'll be back in another AI winter.
  - mcguire 11y ago
    My prediction: One or both of two problems will be the straw that breaks the "wait, this doesn't work as well as you've hyped it to" camel's back. 1. Explanation of why a particular decision was made. (I was incredibly happy to see the Toyota article this morning that mentioned this very subject.) 2. Flexibility in the face of change. (Or is there a better response to a model with declining effectiveness than "start over".)
- blazespin 11y ago
  Actually, it's not as inflated as you might imagine. For example, extend Deep Minds work on video games to a fully simulated 3D environment of a factory. Now, have Deep Mind learn to 'play the factory' to produce goods (starting off with the current process as the baseline). Suddenly, the potential gets pretty exciting. Assuming of course, the AI doesn't get control of the Factory by accident and does something terrible in the name of improving productivity.
apsec112 11y ago
This article seems very confused. Most importantly, it doesn't distinguish between what AI can do right now, and what the theoretical limit of AI is, in 50 or 100 or 500 years. Everyone agrees that AI today, and in the near future, doesn't even vaguely resemble a "person". That doesn't imply that much more powerful, "person-like" AI will forever be impossible. Airplanes weren't a practical means of travel in 1910, but by 1960 it was a different story, and some people in 1910 had already realized that plane travel was coming. Secondly, it does a lot of handwaving about "religion". I am an atheist, and think most religious beliefs are irrational. However, that doesn't mean that every belief that "looks like" religion (in some vague, poorly-defined way) is irrational. The Aztec religion was false, but Hernan Cortes and his army of men with guns was very real. It would have been stupid for Aztec atheists to ignore Cortes because "that sounded like religion". The right question to ask is "is this claim supported by the evidence?", not "how much like religion does this claim sound?".
- stepvhen 11y ago
  Religion itself does a lot of hand waving, and people tend to talk about the "existential threat" of AI with a lot of hand waving. Its an easy comparison to make. Also, a claim, by itself, can't be a religion. It would need people to attach to it with a certain fervor. >it doesn't distinguish between what AI can do right now, and what the theoretical limit of AI is, in 50 or 100 or 500 years. AI, by itself, cannot interact with the physical world, past bit-flipping registers on a cpu (a negligible action). For AI to be a threat, in my opinion, it would need to be coupled with some form of physical manifestation that could, in some fashion, not be under the control of a human. Until that happens, we can just unplug the machine. And we are already seeing the intense difficulty of robotics, making good robots, and interfacing with the real world. Historically, everything with AI is much, much harder than originally thought.
  - drakenot 11y ago
    I wouldn't underestimate the effects that just bit-flipping registers on a cpu can have, even today. In the future, with increased automation, the effects of controlling those bits will be even more severe.
waterlesscloud 11y ago
"It's not so much a rise of evil as a rise of nonsense. It's a mass incompetence, as opposed to Skynet from the Terminator movies. That's what this type of AI turns into." That's the danger. I think Lanier is a lot closer to Musk on that point than he imagines, though.
FreedomToCreate 11y ago
There is a point in the article which the author kind of glosses over that is really important. A lot of the bots (Siri, Cortana, Slack bot, etc...) are refined through free data provided by the masses. These systems get smarter at what they do, essentially for free. And his second point, that once an AI understands it users preferences, it ability to change slows down because unlike when it started, it ignores virgin data, and only follows what hit has been taught. That is a huge potential issue. Seems like a topic ripe for researchers in Machine learning to jump on.
- tsunamifury 11y ago
  He might have seemed to gloss over it because he's written an entire book about it called "Who Own's the Future".
jerf 11y ago
It is, in some sense, a very simple problem. When and if we advance to the point that AI algorithms are themselves capable of writing AI algorithms, what happens then? Quantify (albeit vaguely) the rate at which AI algorithms can improve AI algorithms. If that number turns out to be roughly as slow as human beings, there's no significant problem. If it turns out that number is greatly larger than human beings, that's a problem. It is not particularly clear than the number is large; hard problem is hard. No sarcasm; arguably "general intelligence" is the hardest possibly engineering problem. Adding a bit more intelligence to the problem when it will already be getting worked on by a lot of humans may not move the needle much to speak of at all. But it is not particularly clear that that rate is small either, though; humans are smart, yet at the same time, really dumb in a lot of ways. We have terrible working memory (7 +/- 2 items is absurdly small). We have lots of irrational biases. We have lots of things to do in our lives that are not "thinking" (eating, sleeping, etc.), and the vast majority of our brain is dedicated to those problems, not rational thought. We are terrible at manipulating vast symbol systems without taking immense shortcuts, which inevitably completely color our manipulations. What happens when something that lacks those restrictions is turned loose on the problem of writing AI algorithms? Heck if I know. I'm a human too. I honestly think that those people who are utterly convinced the rate must necessarily be slow are just as wrong as those who are utterly convinced it must necessarily be fast. We really don't know, but it is hardly an invalid concern. So far, AI algorithms have not written very many AI algorithms. I've seen some toys where one AI algorithm is hooked up to tune another one, but I'm not sure if any of them have ever been practically useful. (All I know is all the ones I've personally seen have amounted to toys.) And the system as a whole is generally constrained by what the final AI algorithm can output anyhow; if the domain and range of the AI function are immutably fixed, there's not much the AI-meta-system can do to "escape" and do anything terribly nasty. I do think we're still a ways away from this being an issue. AI is currently still very much constrained to problems a great deal simpler than "writing AI algorithms", which is right up there with the hardest possible things that human intelligences are currently capable of, and that only to a rare few highly trained and highly talented individuals. We're easily decades away from this problem. But when the day comes, well, I wouldn't bet on the self-improving AI experiencing multiple orders-of-magnitude improvement in mere minutes, but I'd sure hate to bet the future of the species that it's not possible. It is not unreasonable to be concerned that the knee could be very steep. We'll know more as we get closer.
nickpsecurity 11y ago
Great article covering a lot of good topics. Especially the overpromise->winter effect and the fact that lots of whats in AI's are nonsense. My favorite part was this though: "The truth is that the part that causes the problem is the actuator. It's the interface to physicality. " BOOM! That was my exact argument in counterpoints to Superintelligence risks. I thought it was ridiculous to worry about what it thought when you could easily control what it did at the interface. I also pointed out that high assurance security already has decades of work dealing with this exact problem and pretty effectively. So, anyone worried about that sort of things should focus on securing the interface that would be used in various domains to catch issues. Now, that's not to say a superintelligence can't break an evil scheme down into a series of safe actions that result in catastrophe. There's possibilities there. Just that all methods for handling them can and should be at the interface. And can be implemented by verifiable, dumb algorithms.
- Houshalter 11y ago
  Once AI is invented, what's to keep anyone in the world from building one? And not keeping it in a box? And that's assuming the box even works, and the AI can't hack out of it, or trick you into letting it out.
  - nickpsecurity 11y ago
    My AI that works in concert with surveillance state to hunt their AI's. Plus, anything or anyone too smart will have to be registered and monitored. Basically, the reaction to mutant powers on X-men.
cygnus_a 11y ago
I think there are two valid existential concerns regarding AI: 1) Widespread use of advanced AI in robotics/production has the ability to bankrupt the working populace, at which point something closer to pure socialism may take over; not so much an annihilation scenario. 2) AI is weaponizable -- at a certain point, given other technological advances in addition to AI, a relatively small amount of like-minded people will be able to produce unique weapons of mass destruction.
- amag 11y ago
  1) Or we enter a post-scarcity era where there will be an abundance of food produced by autonomous machines. No one "needs" to work in order to feed themselves. 2) There are a lot of things that are weaponizable - today.
  - cygnus_a 11y ago
    1) yeah! socialism!! 2) exactly! like a quadcopter with a handgun attached :)
- CuriousSkeptic 11y ago
  More likely, pure socialism will not take over. While not an annihilation scenario, I can very much see a decimation scenario unfolding as a war between the elite and those no longer useful... Edit: In support of this scenario. Consider the refugee crisis and the response from the various countries. Even Sweden, ostensibly one of the most welcoming (and socialistic) countries, started to close its borders as soon as the situation started to get uncomfortable.
natch 11y ago
The Edge has lost it. Jaron's essay was a fantastic read, highly recommend it, but the respondents are not engaging with it at all. They are just using the Edge as a platform to promote their own ideas. Most of them clearly did not read what he wrote. Even though I respect most of the participants, the platform is not eliciting their best.
freyr 11y ago
> There's always been a question about whether a program is something alive or not since it intrinsically has some kind of autonomy at the very least, or it wouldn't be a program. Huh? A program, by definition, has no autonomy. Given its inputs, it performs the appropriate sequence of actions it has been programmed to perform. At no point is it governing its own behavior. Can anybody name a single realizable program that exhibits autonomy? Even machine learning algorithms, which adapt to their input, are deterministic and incapable of self-governance. Given identical initial conditions and inputs, the machine learning program will generate identical deterministic output.
- jtolmar 11y ago
  We live in a deterministic universe. You'll need a different definition of autonomy if you want anything to have it.
  - deleted 11y ago
    [deleted]
nickpsecurity 11y ago
Another thing I'll add is many people forget that the only known super-intelligence, savant humans, still takes years of training data and introspection to get the intelligence to function in society. Takes even more to function better esp accounting for human nature. So, I call bullshit on the idea there's a startup anywhere that's just going to turn on an AI we can't handle. We'll likely see it coming a mile away or the isolation it operated in will do it in when it faces human strategists.
robotcookies 11y ago
I don't think this is as much about AI as it may seem. I mean, yes, it's about AI. But what it's really about is a human tendency which is to create things and meaning when we don't necessarily have the evidence of them. What this is really about is religion. More specifically, we have people who are the most non-religious group out there (like scientists) and they dismiss formal religion only to recreate another version of it: "A core of technically proficient, digitally-minded people reject traditional religions and superstitions. They set out to come up with a better, more scientific framework. But then they re-create versions of those old religious superstitions!" And I see this more often than just in computer science. It happens all the time in economics.

139 points by discreteevent 11y ago

Am I missing something or does the first reply (by George Church) completely ignore everything the original article says?

JetSetWilly 11y ago
I don't think you are, he seems to respond with the predictable hyperbolic future-boosting expected of futurologists, "singulatarians" etc. Everything is all "exponential growth" and inevitable transformation right-around-the-corner. Except it isn't. And this ignored every point that jaron was trying to make and falls into all the traps he was trying to point out. It is an oblivious comment. I must say, I don't find most of the contributors to edge particular insightful.
AndyMcConachie 11y ago
I agree. It seems he doesn't really get it.
natch 11y ago
Not only the first reply. Going through the replies, most of them do not seem to have read. Pretty disappointed with the Edge here. It's clear that the participants are just exploiting it as a platform to put forward their own ideas.

VLM 11y ago

"There has been a domineering subculture—that's been the most wealthy, prolific, and influential subculture in the technical world—that for a long time has not only promoted the idea that there's an equivalence between algorithms and life, and certain algorithms and people, but a historical determinism that we're inevitably making computers that will be smarter and better than us and will take over from us." On a different discussion site I recently pointed out there is a similar religious belief or religious dogma in the idea of the self driving car, that went over like a lead balloon. People acting in a religious manner and expressing religious belief don't like to have that pointed out to them. Probably vestigial monotheism, if they act religiously at their traditional church on Sundays they don't like it pointed out that they worship a different altar online, or the atheists get really wound up when they are called out for joining the atheist movement but acting as deacons of the church of the self driving car of the future. Anyway just saying its a line of reasoning, that while true, can only lead to unpleasantness. Like discussion of scientific racial / gender differences or discussions about IQ, all you're gonna get is social signalling screaming "see no evil".

randallsquared 11y ago
While I don't agree with Lanier, being an AI skeptic isn't quite the same as saying that self driving cars can't happen: we don't yet have a good understanding of either consciousness or tractable general optimization algorithms, so it's still possible that there's something that will make it impossible. Unlikely, perhaps, but possible. Also, your sarcasm about the self driving car might have been too subtle. ;)
- ghaff 11y ago
  What you see with respect to self-driving cars IMO is a lot of people who SO want them to exist that they assign timelines that are so optimistic they'd be rolling on the floor laughing if applied to projects they had direct familiarity with. Of course, it's also fueled by a lot of hype and hand-wavy statements about exponential growth and the like. Like many other aspects of AI, I'm not sure just about anyone thinks we need to have strong AI to make self-driving cars work under the vast majority of conditions. We "just" need to create automation that can handle a reasonable number of corner cases without flipping out or grinding to a halt. But that's a debate about engineering timelines not something more philosophical.
  - randallsquared 11y ago
    I'm a little confused. We have self-driving cars. Of course there are still corner cases and bugs and regulatory issues, but cars have driven for fair distances on existing roads without a human driving. I had assumed, therefore, that the point @VLM was making was that, like self-driving cars, we already have AI by many of the standards of earlier AI researchers (the "when it works, it's not called AI any longer" argument).
    - ghaff 11y ago
      I was reading "self-driving cars" as "Level 4 autonomy" under general conditions. Which IMO is many decades away. But we're in violent agreement about the "when it works, it's not called AI any longer" argument.

moron4hire 11y ago

God, what a lucid, piercing look at reality and the human condition he has. The whole part on translators, just brilliant.

kfk 11y ago

Well, at a logical level, it's one of the biggest existential threats, if 1. it can be done; 2. humans are intelligent enough to pull it off. There is a non 0 chance of AI (and related existential threat), as there is a non 0 chance of a meteorite coming down and wiping all of us off the Earth. But my guess is that most of us would say "YES" to a meteorite deviation project, while "Meh" to any attempt to better understand the AI problem.

conjectures 11y ago
Compared to nuclear war, superbugs, volcanoes, meteorites or catastrophic climate change, naughty-AI is a very distant threat. It's like worrying the geneticists are going to breed Ridley Scott's alien sometime in the relevant future. The world is actually pretty messed up with problems right now; some of which we could actually do something meaningful about. Continuing the naughty-AI bunfight isn't one of them.
- Houshalter 11y ago
  I think it's far more of a threat than any of those things. Nuclear war isn't inevitable, and wouldn't destroy the entire world, just the countries at war. Even a ridiculous world wide nuclear war would leave tons of survivors. Volcanoes and meteorites are so unlikely they aren't even worth mentioning. Superbugs would have to be intentionally engineered by a malicious group with a lot of resources, and so are very unlikely. AI is inevitable. It's very likely to happen in our lifetime. And it's very likely to kill us all. And explaining every reason why I believe those things could fill an entire book, so I seriously recommend reading Superintelligence by Nick Bostrom.
  - stepvhen 11y ago
    >And it's very likely to kill us all. This is absolutely absurd. Could you elaborate on that instead of just pointing to a book?
    - Houshalter 11y ago

tsunamifury 11y ago

I think Jaron knows what many engineers who have worked on AI know: That the external promise of AI is far more inflated than the capability of the actual technology. It's simply that when the current technology gets just complicated enough for the above-average person to no longer understand, everyone starts assigning magical properties and expectations to it. This results in short term over-valuations, which inevitably lead to disappointment [1]. Jaron has observed this several times and rightly seems to be tired of repeating the same naive cycle. As I'm just now entering my second cycle and watching tech repeat itself again -- I'm beginning to understand his weariness. [1] https://hbr.org/2015/12/the-overvaluation-trap https://hbr.org/2015/12/the-overvaluation-trap

ryanmarsh 11y ago
Relevant: Clark's Third Law "Any sufficiently advanced technology is indistinguishable from magic."
api 11y ago
This drives a long cycle of AI booms and AI winters. Right now we seem to be in another boom. When the limits of deep learning and other new techniques become apparent, we'll be back in another AI winter.
- mcguire 11y ago
  My prediction: One or both of two problems will be the straw that breaks the "wait, this doesn't work as well as you've hyped it to" camel's back. 1. Explanation of why a particular decision was made. (I was incredibly happy to see the Toyota article this morning that mentioned this very subject.) 2. Flexibility in the face of change. (Or is there a better response to a model with declining effectiveness than "start over".)
blazespin 11y ago
Actually, it's not as inflated as you might imagine. For example, extend Deep Minds work on video games to a fully simulated 3D environment of a factory. Now, have Deep Mind learn to 'play the factory' to produce goods (starting off with the current process as the baseline). Suddenly, the potential gets pretty exciting. Assuming of course, the AI doesn't get control of the Factory by accident and does something terrible in the name of improving productivity.

apsec112 11y ago

This article seems very confused. Most importantly, it doesn't distinguish between what AI can do right now, and what the theoretical limit of AI is, in 50 or 100 or 500 years. Everyone agrees that AI today, and in the near future, doesn't even vaguely resemble a "person". That doesn't imply that much more powerful, "person-like" AI will forever be impossible. Airplanes weren't a practical means of travel in 1910, but by 1960 it was a different story, and some people in 1910 had already realized that plane travel was coming. Secondly, it does a lot of handwaving about "religion". I am an atheist, and think most religious beliefs are irrational. However, that doesn't mean that every belief that "looks like" religion (in some vague, poorly-defined way) is irrational. The Aztec religion was false, but Hernan Cortes and his army of men with guns was very real. It would have been stupid for Aztec atheists to ignore Cortes because "that sounded like religion". The right question to ask is "is this claim supported by the evidence?", not "how much like religion does this claim sound?".

stepvhen 11y ago
Religion itself does a lot of hand waving, and people tend to talk about the "existential threat" of AI with a lot of hand waving. Its an easy comparison to make. Also, a claim, by itself, can't be a religion. It would need people to attach to it with a certain fervor. >it doesn't distinguish between what AI can do right now, and what the theoretical limit of AI is, in 50 or 100 or 500 years. AI, by itself, cannot interact with the physical world, past bit-flipping registers on a cpu (a negligible action). For AI to be a threat, in my opinion, it would need to be coupled with some form of physical manifestation that could, in some fashion, not be under the control of a human. Until that happens, we can just unplug the machine. And we are already seeing the intense difficulty of robotics, making good robots, and interfacing with the real world. Historically, everything with AI is much, much harder than originally thought.
- drakenot 11y ago
  I wouldn't underestimate the effects that just bit-flipping registers on a cpu can have, even today. In the future, with increased automation, the effects of controlling those bits will be even more severe.

waterlesscloud 11y ago

"It's not so much a rise of evil as a rise of nonsense. It's a mass incompetence, as opposed to Skynet from the Terminator movies. That's what this type of AI turns into." That's the danger. I think Lanier is a lot closer to Musk on that point than he imagines, though.

FreedomToCreate 11y ago

There is a point in the article which the author kind of glosses over that is really important. A lot of the bots (Siri, Cortana, Slack bot, etc...) are refined through free data provided by the masses. These systems get smarter at what they do, essentially for free. And his second point, that once an AI understands it users preferences, it ability to change slows down because unlike when it started, it ignores virgin data, and only follows what hit has been taught. That is a huge potential issue. Seems like a topic ripe for researchers in Machine learning to jump on.

tsunamifury 11y ago
He might have seemed to gloss over it because he's written an entire book about it called "Who Own's the Future".

jerf 11y ago

It is, in some sense, a very simple problem. When and if we advance to the point that AI algorithms are themselves capable of writing AI algorithms, what happens then? Quantify (albeit vaguely) the rate at which AI algorithms can improve AI algorithms. If that number turns out to be roughly as slow as human beings, there's no significant problem. If it turns out that number is greatly larger than human beings, that's a problem. It is not particularly clear than the number is large; hard problem is hard. No sarcasm; arguably "general intelligence" is the hardest possibly engineering problem. Adding a bit more intelligence to the problem when it will already be getting worked on by a lot of humans may not move the needle much to speak of at all. But it is not particularly clear that that rate is small either, though; humans are smart, yet at the same time, really dumb in a lot of ways. We have terrible working memory (7 +/- 2 items is absurdly small). We have lots of irrational biases. We have lots of things to do in our lives that are not "thinking" (eating, sleeping, etc.), and the vast majority of our brain is dedicated to those problems, not rational thought. We are terrible at manipulating vast symbol systems without taking immense shortcuts, which inevitably completely color our manipulations. What happens when something that lacks those restrictions is turned loose on the problem of writing AI algorithms? Heck if I know. I'm a human too. I honestly think that those people who are utterly convinced the rate must necessarily be slow are just as wrong as those who are utterly convinced it must necessarily be fast. We really don't know, but it is hardly an invalid concern. So far, AI algorithms have not written very many AI algorithms. I've seen some toys where one AI algorithm is hooked up to tune another one, but I'm not sure if any of them have ever been practically useful. (All I know is all the ones I've personally seen have amounted to toys.) And the system as a whole is generally constrained by what the final AI algorithm can output anyhow; if the domain and range of the AI function are immutably fixed, there's not much the AI-meta-system can do to "escape" and do anything terribly nasty. I do think we're still a ways away from this being an issue. AI is currently still very much constrained to problems a great deal simpler than "writing AI algorithms", which is right up there with the hardest possible things that human intelligences are currently capable of, and that only to a rare few highly trained and highly talented individuals. We're easily decades away from this problem. But when the day comes, well, I wouldn't bet on the self-improving AI experiencing multiple orders-of-magnitude improvement in mere minutes, but I'd sure hate to bet the future of the species that it's not possible. It is not unreasonable to be concerned that the knee could be very steep. We'll know more as we get closer.

nickpsecurity 11y ago

Great article covering a lot of good topics. Especially the overpromise->winter effect and the fact that lots of whats in AI's are nonsense. My favorite part was this though: "The truth is that the part that causes the problem is the actuator. It's the interface to physicality. " BOOM! That was my exact argument in counterpoints to Superintelligence risks. I thought it was ridiculous to worry about what it thought when you could easily control what it did at the interface. I also pointed out that high assurance security already has decades of work dealing with this exact problem and pretty effectively. So, anyone worried about that sort of things should focus on securing the interface that would be used in various domains to catch issues. Now, that's not to say a superintelligence can't break an evil scheme down into a series of safe actions that result in catastrophe. There's possibilities there. Just that all methods for handling them can and should be at the interface. And can be implemented by verifiable, dumb algorithms.

Houshalter 11y ago
Once AI is invented, what's to keep anyone in the world from building one? And not keeping it in a box? And that's assuming the box even works, and the AI can't hack out of it, or trick you into letting it out.
- nickpsecurity 11y ago
  My AI that works in concert with surveillance state to hunt their AI's. Plus, anything or anyone too smart will have to be registered and monitored. Basically, the reaction to mutant powers on X-men.

cygnus_a 11y ago

I think there are two valid existential concerns regarding AI: 1) Widespread use of advanced AI in robotics/production has the ability to bankrupt the working populace, at which point something closer to pure socialism may take over; not so much an annihilation scenario. 2) AI is weaponizable -- at a certain point, given other technological advances in addition to AI, a relatively small amount of like-minded people will be able to produce unique weapons of mass destruction.

amag 11y ago
1) Or we enter a post-scarcity era where there will be an abundance of food produced by autonomous machines. No one "needs" to work in order to feed themselves. 2) There are a lot of things that are weaponizable - today.
- cygnus_a 11y ago
  1) yeah! socialism!! 2) exactly! like a quadcopter with a handgun attached :)
CuriousSkeptic 11y ago
More likely, pure socialism will not take over. While not an annihilation scenario, I can very much see a decimation scenario unfolding as a war between the elite and those no longer useful... Edit: In support of this scenario. Consider the refugee crisis and the response from the various countries. Even Sweden, ostensibly one of the most welcoming (and socialistic) countries, started to close its borders as soon as the situation started to get uncomfortable.

natch 11y ago

The Edge has lost it. Jaron's essay was a fantastic read, highly recommend it, but the respondents are not engaging with it at all. They are just using the Edge as a platform to promote their own ideas. Most of them clearly did not read what he wrote. Even though I respect most of the participants, the platform is not eliciting their best.

freyr 11y ago

> There's always been a question about whether a program is something alive or not since it intrinsically has some kind of autonomy at the very least, or it wouldn't be a program. Huh? A program, by definition, has no autonomy. Given its inputs, it performs the appropriate sequence of actions it has been programmed to perform. At no point is it governing its own behavior. Can anybody name a single realizable program that exhibits autonomy? Even machine learning algorithms, which adapt to their input, are deterministic and incapable of self-governance. Given identical initial conditions and inputs, the machine learning program will generate identical deterministic output.

jtolmar 11y ago
We live in a deterministic universe. You'll need a different definition of autonomy if you want anything to have it.
- deleted 11y ago
  [deleted]

nickpsecurity 11y ago

Another thing I'll add is many people forget that the only known super-intelligence, savant humans, still takes years of training data and introspection to get the intelligence to function in society. Takes even more to function better esp accounting for human nature. So, I call bullshit on the idea there's a startup anywhere that's just going to turn on an AI we can't handle. We'll likely see it coming a mile away or the isolation it operated in will do it in when it faces human strategists.

robotcookies 11y ago

I don't think this is as much about AI as it may seem. I mean, yes, it's about AI. But what it's really about is a human tendency which is to create things and meaning when we don't necessarily have the evidence of them. What this is really about is religion. More specifically, we have people who are the most non-religious group out there (like scientists) and they dismiss formal religion only to recreate another version of it: "A core of technically proficient, digitally-minded people reject traditional religions and superstitions. They set out to come up with a better, more scientific framework. But then they re-create versions of those old religious superstitions!" And I see this more often than just in computer science. It happens all the time in economics.