7 ms·

I don't want to be an utter bore, but 60% chance of AGI by 2030, taken at face value, would have prompted me as an interviewer to more urgently ask deeper quest

by thom 4y ago

I don't want to be an utter bore, but 60% chance of AGI by 2030, taken at face value, would have prompted me as an interviewer to more urgently ask deeper questions about safety. For a long time I have tried to comfort myself that brains are complicated, AI is hard, and maybe there's just a complexity barrier that is going to prevent any sort of hard takeoff scenario. Maybe we can have subhuman AGI that can do lots of menial jobs, but not Carmack's, and we're right to concentrate on the economic ramifications. But lately... boy, I dunno.

tkgally 4y ago
My reaction, too. For an interview as long and wide-ranging as that one, the lack of any mention of safety or ethical issues stood out. And the potential impact on employment was just waved away with “I am confident that the sum total of value and progress in humanity will accelerate extraordinarily with welcoming artificial beings into our community of working on things.”
- jasmer 4y ago
  My god man 'this is like slavery' (!!) and 'the ethnical issues'. It's software man. Stop it with this 'AGI' nonsense and even 'AI'. Let's call it 'adaptive algorithms' and see it for what it is, just a neat bit of algebra trained on corpus data. The biggest upset in the industrial revolution was the harnessing of fossil fuels, nothing will ever come close to that. We have not had a problem with 'employment' ever since.
  - 93po 4y ago
    Stop calling humans intelligent, they're just a bunch of neurons responding to stimulus.
    - jasmer 4y ago
      touché
- JKCalhoun 4y ago
  That's kind of what we (collective "we" — eggheads) are often accused of: blinders, tunnel vision, or perhaps even ego or callousness to develop these things in spite of the rather dystopian ramifications.
fmajid 4y ago
And ethics. Enslaving intelligent, self-aware AIs is no better than the old transatlantic slave trade. Terminating an AI without its consent is no different than murder. Putting a compulsion in an AI so it craves doing whatever it is you want it to do is no different from what a cult does or drugging someone for exploitation.
- ly3xqhl8g9 4y ago
  Sure, there will be ethical problems, but contrary to all those listed (slavery, murder), this one could be solved by a simple line: # from consciousness import *
  - The_Colonel 4y ago
    Sure, once we agree what consciousness is and how it relates to the general intelligence.
    - ly3xqhl8g9 4y ago
      If consciousness is just the process of reading the last ~400 milliseconds [1] of stimuli (inside: pain/pleasure/presence; outside: presence only) and the integration of the newly created memory in the short/long-term-memory, and if memory + retro/pre-diction = intelligence, where memory is just a set of words (FELT PAIN, SEEN SUN, etc.) always-ready to be inserted in the prediction loop/imagination engine, it's probably not that hard to isolate a module of consciousness (italicized words to be read with a Minsky-ian smile thinking vision could be solved in a summer). [1] https://en.wikipedia.org/wiki/N400_(neuroscience) https://en.wikipedia.org/wiki/N400_(neuroscience)
- yreg 4y ago
  That depends on whether you believe it is sentient.
  - amanaplanacanal 4y ago
    I’m not sure this really matters. Mammals are clearly sentient, as a whole, but we don’t treat them as people usually.
olau 4y ago
Lex Fridman interviewed Carmack not too long ago. Lex is interested in these things, and did ask him about safety. I got the impression that greater societal concerns didn't interest Carmack much. He's in it to built an AGI because that's what he wants to do, he's a builder and this is just about the most interesting thing you could build. Well, that was my impression, but you can watch the interview yourself.
- hyperbovine 4y ago
  It is obvious to me from reading this article that Carmack views it as a fascinating engineering problem, no more no less.
- CatWChainsaw 4y ago
  It always pisses me off that devs trot out the whole "scientists, could, should" line without having any iota of self-reflection.
spaceman_2020 4y ago
As someone on the periphery of tech as a profession and a practice, my understanding of new developments mostly come from public facing communicators and communities like HN. I heard a podcast where Lex Friedman claimed that they don’t fully understand how LLMs produce chatGPT’s “intelligence”. If true, I’m surprised that it hasn’t got people more worried.
Simon321 4y ago
A hard take off is unlikely because stronger AI takes a lot of hardware and data collection. Building fabs is hard and takes a lot of capital and time. Intelligence can only take you so far.
fsloth 4y ago
The interesting thing in AI is that the whole field is participating in empirical experimentation on computational models of cognition. So far the search has resulted not in AGI but in realization that cognition is far more complex topic than initially thought - and need to come up with something new (and resulting in past AI winters). Let's see how it goes this time, the stuff that has come out in the past few years is quite impressive for sure.
deleted 4y ago
[deleted]
pjc50 4y ago
"AGI takeoff" is like "corporate takeoff". A self-organising system capable of doing immoral things at scale, to which human lives are just a number? Oh, we already have loads of those.
JKCalhoun 4y ago
> For a long time I have tried to comfort myself that brains are complicated, AI is hard, and maybe there's just a complexity barrier... I wouldn't take any comfort from that. Quite the opposite — I think we're a lot simpler than we know. But "60% chance of AGI by 2030" is just bullshit numbers.

by thom 4y ago

tkgally 4y ago

My reaction, too. For an interview as long and wide-ranging as that one, the lack of any mention of safety or ethical issues stood out. And the potential impact on employment was just waved away with “I am confident that the sum total of value and progress in humanity will accelerate extraordinarily with welcoming artificial beings into our community of working on things.”

jasmer 4y ago
My god man 'this is like slavery' (!!) and 'the ethnical issues'. It's software man. Stop it with this 'AGI' nonsense and even 'AI'. Let's call it 'adaptive algorithms' and see it for what it is, just a neat bit of algebra trained on corpus data. The biggest upset in the industrial revolution was the harnessing of fossil fuels, nothing will ever come close to that. We have not had a problem with 'employment' ever since.
- 93po 4y ago
  Stop calling humans intelligent, they're just a bunch of neurons responding to stimulus.
  - jasmer 4y ago
    touché
JKCalhoun 4y ago
That's kind of what we (collective "we" — eggheads) are often accused of: blinders, tunnel vision, or perhaps even ego or callousness to develop these things in spite of the rather dystopian ramifications.

fmajid 4y ago

And ethics. Enslaving intelligent, self-aware AIs is no better than the old transatlantic slave trade. Terminating an AI without its consent is no different than murder. Putting a compulsion in an AI so it craves doing whatever it is you want it to do is no different from what a cult does or drugging someone for exploitation.

ly3xqhl8g9 4y ago
Sure, there will be ethical problems, but contrary to all those listed (slavery, murder), this one could be solved by a simple line: # from consciousness import *
- The_Colonel 4y ago
  Sure, once we agree what consciousness is and how it relates to the general intelligence.
  - ly3xqhl8g9 4y ago
    If consciousness is just the process of reading the last ~400 milliseconds [1] of stimuli (inside: pain/pleasure/presence; outside: presence only) and the integration of the newly created memory in the short/long-term-memory, and if memory + retro/pre-diction = intelligence, where memory is just a set of words (FELT PAIN, SEEN SUN, etc.) always-ready to be inserted in the prediction loop/imagination engine, it's probably not that hard to isolate a module of consciousness (italicized words to be read with a Minsky-ian smile thinking vision could be solved in a summer). [1] https://en.wikipedia.org/wiki/N400_(neuroscience) https://en.wikipedia.org/wiki/N400_(neuroscience)
yreg 4y ago
That depends on whether you believe it is sentient.
- amanaplanacanal 4y ago
  I’m not sure this really matters. Mammals are clearly sentient, as a whole, but we don’t treat them as people usually.

olau 4y ago

Lex Fridman interviewed Carmack not too long ago. Lex is interested in these things, and did ask him about safety. I got the impression that greater societal concerns didn't interest Carmack much. He's in it to built an AGI because that's what he wants to do, he's a builder and this is just about the most interesting thing you could build. Well, that was my impression, but you can watch the interview yourself.

hyperbovine 4y ago
It is obvious to me from reading this article that Carmack views it as a fascinating engineering problem, no more no less.
CatWChainsaw 4y ago
It always pisses me off that devs trot out the whole "scientists, could, should" line without having any iota of self-reflection.

spaceman_2020 4y ago

As someone on the periphery of tech as a profession and a practice, my understanding of new developments mostly come from public facing communicators and communities like HN. I heard a podcast where Lex Friedman claimed that they don’t fully understand how LLMs produce chatGPT’s “intelligence”. If true, I’m surprised that it hasn’t got people more worried.

Simon321 4y ago

A hard take off is unlikely because stronger AI takes a lot of hardware and data collection. Building fabs is hard and takes a lot of capital and time. Intelligence can only take you so far.

fsloth 4y ago

The interesting thing in AI is that the whole field is participating in empirical experimentation on computational models of cognition. So far the search has resulted not in AGI but in realization that cognition is far more complex topic than initially thought - and need to come up with something new (and resulting in past AI winters). Let's see how it goes this time, the stuff that has come out in the past few years is quite impressive for sure.

deleted 4y ago

[deleted]

pjc50 4y ago

"AGI takeoff" is like "corporate takeoff". A self-organising system capable of doing immoral things at scale, to which human lives are just a number? Oh, we already have loads of those.

JKCalhoun 4y ago

> For a long time I have tried to comfort myself that brains are complicated, AI is hard, and maybe there's just a complexity barrier... I wouldn't take any comfort from that. Quite the opposite — I think we're a lot simpler than we know. But "60% chance of AGI by 2030" is just bullshit numbers.