5 ms·

For those of you that are youngsters out there seeing this article now, a little context is perhaps appropriate. I remember when this article came out, being a

by CPLX 9y ago

For those of you that are youngsters out there seeing this article now, a little context is perhaps appropriate. I remember when this article came out, being a Wired subscriber at the time, as one did if they were in their 20's working at a "dotcom" and thinking this Internet thing was turning out pretty cool. This article was viscerally and emotionally terrifying at the time. For whatever reason it seemed really real. I remember overhearing people talking about "grey goo" over cocktails at startup events -- the scenario here seemed real and imminent. As usual, of course, as with most doomsday predictions, the actual future reality turned out to be equally terrifying and completely different.

terminado 9y ago
...and it's not even over yet.
0d311 9y ago
As a software engineer who graduated college about a year ago -- with a degree in robotics engineering -- somehow the end has always seemed both near and far. Botnets and massive hacks aren't something I grew up with, but they feel commonplace ever since my introduction to the world of tech. I don't know if I'll even be surprised when the grey goo starts pouring out of my USB port, other than the surprise something else malicious didn't start pouring out first :)
pvg 9y ago
I think it was a minor flash in the pan just like whenever someone famous makes some doomsday prediction. In terms of impact and importance, this was probably not substantially different from the time Hawking said that we should be careful looking for aliens because they might be evil. A lot of things seem viscerally and emotionally terrifying after your fourth free drink on the rooftop at the Industry Standard.
titzer 9y ago
Oh, the gray goo is already here, has been for centuries. It even glows at night. We've just not noticed because it grows slowly and is only visible at extremely zoomed-out scale. But its growth has definitely sped up in recent decades. https://earthobservatory.nasa.gov/Features/CitiesAtNight/ https://earthobservatory.nasa.gov/Features/CitiesAtNight/ (slightly tongue-in-cheek, slightly not).
- jhbadger 9y ago
  And similarly, another other SF trope, the superhuman AI with a sole goal of benefiting itself, already exists too -- that's what a corporation is.
  - titzer 9y ago
    Ha. Corporations are clearly not super-intelligent, thankfully :)
    - igorkraw 9y ago
      Are they not?
ataturk 9y ago
I've read Kurzweil's books. Reflecting on them over time, I think he's kind of full of it. It's not that the material was un-interesting but rather that the claims he was making were a bit wild and overly-optimistic. I see the same kinds of flaws in the reasoning of many people of his generation, where they compile this list of truisms and try to get you from A->B but really the fundamental assumptions from A aren't that solid. Kurzweil wants to live forever. He wants to become a cybernetic being. That was his undoing in those books--his desire to outlast death forced him to make wildly optimistic predictions. The dangers facing us are not AI but simply mundane databases. Everything is being tracked these days. Compliance is being shoved down our throats. You are being watched on the Internet and in daily life. "But I have nothing to hide." Yes, you do. You just don't realize it. You should read up on how people were treated in East Germany and the Soviet Union because that is your future. Technology gives zero shits about humans.
- spitfire 9y ago
  I call these the Bhopal 2.0 problems. Bhopal was a disaster where 32tons of toxic MIC gas was released into the atmosphere and killed 3,787 injuring half a million people. My premise for Bhopal 2.0 is we automate processes so much and have no accounting for waste and damage that we simply focus on the top number. Ask an ML to optimize for an output and it will, happily, All the while harming people around the process - automatically. Right now Bhopol 2.0 processes are all focused on information systems - abstractions, so it's much harder to point and say "look, this is dangerous". People do very, very, very badly with abstractions. In some way this already happened with the recent elections. But that was purposely aiming the machines at us.
- mr_spothawk 9y ago
  > forced him to make wildly optimistic predictions. maybe I'm wrong, but I thought those predictions have fared pretty well. edit to add: optimism, or not... this came quickly (ahem, china) [1] > 2019, Public places and workplaces are ubiquitously monitored to prevent violence and all actions are recorded permanently. Personal privacy is a major political issue, and some people protect themselves with unbreakable computer codes. from his 1999 book, via wikipedia 1 - https://en.wikipedia.org/wiki/Predictions_made_by_Ray_Kurzweil#2019 https://en.wikipedia.org/wiki/Predictions_made_by_Ray_Kurzwe...
pmoriarty 9y ago
"This article was viscerally and emotionally terrifying at the time. For whatever reason it seemed really real." Maybe for some, but certainly not for me. It was just another doomsday scenario. Just like the kind that have been cropping up in science fiction for at least a hundred years, and religious tracts for thousands of years before that. "I remember overhearing people talking about "grey goo" over cocktails at startup events -- the scenario here seemed real and imminent." Just because they talked about it doesn't mean they thought it was real or imminent. Honestly, I don't think most people were any more scared of it back then than now. Just something to consider, and maybe keep an eye on.
joe_the_user 9y ago
Yeah, articles about the dangers of general AI basically missed the simpler, immediate problems of human using mechanical means to modify themselves and their environment without any ability to collectively, wisely consider the consequences (problems that begin long before machines are intelligent). If humans are a collectivity, "we" right now have created the means for most people to live without working. But upshot of this hasn't been paradise but increasing privation in advanced nations as everyone still needs a conventional job for social and material survival but fewer and fewer people can find them (and the oversupply of workers reduces their price). Just as much, those who leverage the transformation of society using technology have benefited greatly but the resulting inequality of wealth distribution has produced tremendous despair for a significant percentage of the population with little indication that any further transformations are going to fix this (basic income is one effort but I don't see how this can happen without severe inflation, etc). A similar unthinking dynamic can be seen in climate change and the effective total lack of serious response to it. It will be a long time, if ever, before some AGI rules humanity. But the use of mechanical processes to control human behavior began not with AI, not with social media but longer-ago with mass media and it's certainly in high gear at the moment (in both mass and social media of course), allowing whatever well-connected actors to control the "news cycle" and guarantee the lack of coherent, collective discussion of the overall, serious problems of this society (as mentioned above and as most people can see in front of them).
vilhelm_s 9y ago
I mean, I think it's also not quite sound to go "oh, it's been 18 years and we haven't been converted to grey goo yet, I guess it was a false alarm, phew." Give the robots some time!
- Retric 9y ago
  Grey goo fails because it's stuck with the same chemistry life is built off of. You can't use much iron for your self replication because there is not much iron in the environment. You also need an energy source and storage etc. End result you end up building using organic chemistry which means your goo get's eaten.
  - chongli 9y ago
    Couldn't you make grey goo out of silicon? There's more silicon than anything else in the rocks of this planet. It could produce its energy by photovoltaic cells. Blanket the entire planet in a seething mass of solar-powered nanobots. That's what I think of when I hear the term 'grey goo.'
mr_toad 9y ago
> the scenario here seemed real and imminent. It was never imminent. Drexler himself argued that there was a huge bootstrapping problem in creating nanoscale ‘assemblers’ with our macro scale technology.
vijucat 9y ago
> completely different Don't lose hope about losing all hope yet! Apparently, "Scientists accidentally create mutant enzyme that eats plastic bottles. The breakthrough, spurred by the discovery of plastic-eating bugs at a Japanese dump, could help solve the global plastic pollution crisis" https://www.theguardian.com/environment/2018/apr/16/scientists-accidentally-create-mutant-enzyme-that-eats-plastic-bottles https://www.theguardian.com/environment/2018/apr/16/scientis...

by CPLX 9y ago

terminado 9y ago

...and it's not even over yet.

0d311 9y ago

As a software engineer who graduated college about a year ago -- with a degree in robotics engineering -- somehow the end has always seemed both near and far. Botnets and massive hacks aren't something I grew up with, but they feel commonplace ever since my introduction to the world of tech. I don't know if I'll even be surprised when the grey goo starts pouring out of my USB port, other than the surprise something else malicious didn't start pouring out first :)

pvg 9y ago

I think it was a minor flash in the pan just like whenever someone famous makes some doomsday prediction. In terms of impact and importance, this was probably not substantially different from the time Hawking said that we should be careful looking for aliens because they might be evil. A lot of things seem viscerally and emotionally terrifying after your fourth free drink on the rooftop at the Industry Standard.

titzer 9y ago

Oh, the gray goo is already here, has been for centuries. It even glows at night. We've just not noticed because it grows slowly and is only visible at extremely zoomed-out scale. But its growth has definitely sped up in recent decades. https://earthobservatory.nasa.gov/Features/CitiesAtNight/ https://earthobservatory.nasa.gov/Features/CitiesAtNight/ (slightly tongue-in-cheek, slightly not).

jhbadger 9y ago
And similarly, another other SF trope, the superhuman AI with a sole goal of benefiting itself, already exists too -- that's what a corporation is.
- titzer 9y ago
  Ha. Corporations are clearly not super-intelligent, thankfully :)
  - igorkraw 9y ago
    Are they not?

ataturk 9y ago

I've read Kurzweil's books. Reflecting on them over time, I think he's kind of full of it. It's not that the material was un-interesting but rather that the claims he was making were a bit wild and overly-optimistic. I see the same kinds of flaws in the reasoning of many people of his generation, where they compile this list of truisms and try to get you from A->B but really the fundamental assumptions from A aren't that solid. Kurzweil wants to live forever. He wants to become a cybernetic being. That was his undoing in those books--his desire to outlast death forced him to make wildly optimistic predictions. The dangers facing us are not AI but simply mundane databases. Everything is being tracked these days. Compliance is being shoved down our throats. You are being watched on the Internet and in daily life. "But I have nothing to hide." Yes, you do. You just don't realize it. You should read up on how people were treated in East Germany and the Soviet Union because that is your future. Technology gives zero shits about humans.

spitfire 9y ago
I call these the Bhopal 2.0 problems. Bhopal was a disaster where 32tons of toxic MIC gas was released into the atmosphere and killed 3,787 injuring half a million people. My premise for Bhopal 2.0 is we automate processes so much and have no accounting for waste and damage that we simply focus on the top number. Ask an ML to optimize for an output and it will, happily, All the while harming people around the process - automatically. Right now Bhopol 2.0 processes are all focused on information systems - abstractions, so it's much harder to point and say "look, this is dangerous". People do very, very, very badly with abstractions. In some way this already happened with the recent elections. But that was purposely aiming the machines at us.
mr_spothawk 9y ago
> forced him to make wildly optimistic predictions. maybe I'm wrong, but I thought those predictions have fared pretty well. edit to add: optimism, or not... this came quickly (ahem, china) [1] > 2019, Public places and workplaces are ubiquitously monitored to prevent violence and all actions are recorded permanently. Personal privacy is a major political issue, and some people protect themselves with unbreakable computer codes. from his 1999 book, via wikipedia 1 - https://en.wikipedia.org/wiki/Predictions_made_by_Ray_Kurzweil#2019 https://en.wikipedia.org/wiki/Predictions_made_by_Ray_Kurzwe...

pmoriarty 9y ago

"This article was viscerally and emotionally terrifying at the time. For whatever reason it seemed really real." Maybe for some, but certainly not for me. It was just another doomsday scenario. Just like the kind that have been cropping up in science fiction for at least a hundred years, and religious tracts for thousands of years before that. "I remember overhearing people talking about "grey goo" over cocktails at startup events -- the scenario here seemed real and imminent." Just because they talked about it doesn't mean they thought it was real or imminent. Honestly, I don't think most people were any more scared of it back then than now. Just something to consider, and maybe keep an eye on.

joe_the_user 9y ago

Yeah, articles about the dangers of general AI basically missed the simpler, immediate problems of human using mechanical means to modify themselves and their environment without any ability to collectively, wisely consider the consequences (problems that begin long before machines are intelligent). If humans are a collectivity, "we" right now have created the means for most people to live without working. But upshot of this hasn't been paradise but increasing privation in advanced nations as everyone still needs a conventional job for social and material survival but fewer and fewer people can find them (and the oversupply of workers reduces their price). Just as much, those who leverage the transformation of society using technology have benefited greatly but the resulting inequality of wealth distribution has produced tremendous despair for a significant percentage of the population with little indication that any further transformations are going to fix this (basic income is one effort but I don't see how this can happen without severe inflation, etc). A similar unthinking dynamic can be seen in climate change and the effective total lack of serious response to it. It will be a long time, if ever, before some AGI rules humanity. But the use of mechanical processes to control human behavior began not with AI, not with social media but longer-ago with mass media and it's certainly in high gear at the moment (in both mass and social media of course), allowing whatever well-connected actors to control the "news cycle" and guarantee the lack of coherent, collective discussion of the overall, serious problems of this society (as mentioned above and as most people can see in front of them).

vilhelm_s 9y ago

I mean, I think it's also not quite sound to go "oh, it's been 18 years and we haven't been converted to grey goo yet, I guess it was a false alarm, phew." Give the robots some time!

Retric 9y ago
Grey goo fails because it's stuck with the same chemistry life is built off of. You can't use much iron for your self replication because there is not much iron in the environment. You also need an energy source and storage etc. End result you end up building using organic chemistry which means your goo get's eaten.
- chongli 9y ago
  Couldn't you make grey goo out of silicon? There's more silicon than anything else in the rocks of this planet. It could produce its energy by photovoltaic cells. Blanket the entire planet in a seething mass of solar-powered nanobots. That's what I think of when I hear the term 'grey goo.'

mr_toad 9y ago

> the scenario here seemed real and imminent. It was never imminent. Drexler himself argued that there was a huge bootstrapping problem in creating nanoscale ‘assemblers’ with our macro scale technology.

vijucat 9y ago

> completely different Don't lose hope about losing all hope yet! Apparently, "Scientists accidentally create mutant enzyme that eats plastic bottles. The breakthrough, spurred by the discovery of plastic-eating bugs at a Japanese dump, could help solve the global plastic pollution crisis" https://www.theguardian.com/environment/2018/apr/16/scientists-accidentally-create-mutant-enzyme-that-eats-plastic-bottles https://www.theguardian.com/environment/2018/apr/16/scientis...