6 ms·

I wonder if Carmacks moral compass is in order. First he sticks around at Facebook, now he endangers humanity with AGI. And i'm only half joking.

by Tepix 4y ago

kache_ 4y ago
what's wrong with working at facebook? stop acting so morally superior, christ
caliburner 4y ago
I could easily see Carmack as an evil genius type.
ffhhj 4y ago
Just don't let him run those teleportation experiments in Mars.
mushufasa 4y ago
I think he very clearly has an amoral attitude towards technology development -- "you can't stop progress." He does describe his own "hacker ethic" and whatever he develops he may make more open than OpenAI. though I think he has some moral compass around what he believes people should do or not do with technology. For example, he has publicly expressed admiration for electric cars / cleantech and SpaceX's decision to prioritize Mars over other areas with higher ROI.
- falcrist 4y ago
  He also has a vastly different take on privacy than most of us seem to have. He thinks it'll eventually go away and it won't be bad when it does. I believe he talked about it in one of his quakecon keynotes. As a LONG time admirer of Carmack (I got my EE degree and went into embedded systems and firmware design due in no small part to his influence), I feel like he's honest and forthright about his stances, but also disconnected from most average people (both due to his wealth and his personality) in such a way that he's out of touch. He's not egotistical like Elon Musk. In fact he seems humble. He also seems to approach the topics in good faith... but some of his conclusions are... distressing.
  - bitcurious 4y ago
    He recently described his love of computers as rooted in realizing that “they won’t talk back to you.” The job he wanted at Meta was “Dictator of VR.” When someone talks AI ethics to him, he just tunes out because he doesn’t think it’s worth even considering until they are fully sentient as the level of a human toddler, at which point you can turn them on and off and modify their development until you have a perfect worker. His reason for working in AI is that he thinks it’s where a single human can have the largest leverage on history. All that paraphrased from the Lex interview. I see him as the guy who builds the “be anything do anything” singularity, but then adds a personal “god mode” to use whenever the vote goes the wrong way. Straight out of a Stephenson novel. On the other hand, he’s not boring!
  - phatfish 4y ago
    He is a workaholic, in a positive way. But i get the feeling as long as he has a problem he enjoys "grinding out" the solution to, not much else matters -- apart from the obvious of family and close friends. Still, I can't fault his honesty. He doesn't seem to hold anything back in the interviews I've seen.
- bsenftner 4y ago
  I think there is a point to be made that if one could do the work, is offered the work, but thinks it's ethically questionable: go there and be an ethical voice.
deleted 4y ago
[deleted]
cgrealy 4y ago
There is a story in “Masters of Doom” about Carmack getting rid of his cat because “she was having a net negative effect on my life”. That’s cold.
- xen2xen1 4y ago
  But then again, it's a cat.
  - jackblemming 4y ago
    Hopefully the AI Carmack creates doesn’t think the same of you ;)
- stefs 4y ago
  absolutely not! to the contrary; don't force yourself to endure abusive relationships. (also cats are extremely destructive beasts)
- pengaru 4y ago
  It's cold if he killed it/had it euthanized. Not if he simply found it a better home where it was a better fit and more appreciated.
  - MichaelCollins 4y ago
    From Masters of Doom: Scott Miller wasn’t the only one to go before id began working on Doom. Mitzi would suffer a similar fate. Carmack’s cat had been a thorn in the side of the id employees, beginning with the days of her overflowing litter box back at the lake house. Since then she had grown more irascible, lashing out at passersby and relieving herself freely around his apartment. The final straw came when she peed all over a brand-new leather couch that Carmack had bought with the Wolfenstein cash. Carmack broke the news to the guys. “Mitzi was having a net negative impact on my life,” he said. “I took her to the animal shelter. Mmm.” “What?” Romero asked. The cat had become such a sidekick of Carmack’s that the guys had even listed her on the company directory as his significant other–and now she was just gone? “You know what this means?” Romero said. “They’re going to put her to sleep! No one’s going to want to claim her. She’s going down! Down to Chinatown!” Carmack shrugged it off and returned to work. The same rule applied to a cat, a computer program, or, for that matter, a person. When something becomes a problem, let it go or, if necessary, have it surgically removed.
sp527 4y ago
He also said he's never felt in danger of experiencing burnout. The guy's emotional wiring is a total departure from that of most people. Almost alien.
semi-extrinsic 4y ago
The meme that AGI, if we ever have it, will somehow endanger humanity is just stupid to me. For one, the previous US president is the perfect illustration that intelligence is neither sufficient nor necessary for gaining power in this world. And we do in fact live in a world where the upper echelons of power mostly interact in the decidedly analog spaces of leadership summits, high-end restaurants, golf courses and country clubs. Most world leaders interact with a real computer like a handful of times per year. Furthermore, due to the warring nature of us humans, the important systems in the world like banking, electricity, industrial controls, military power etc. are either air-gapped or have a requirement for multiple humans to push physical buttons in order to actually accomplish scary things. And because we humans are a bit stupid and make mistakes sometimes, like fat-fingering an order on the stock market and crashing everything, we have completely manual systems that undo mistakes and restore previous values. Sure, a mischievous AGI could do some annoying things. But nothing that our human enemies existing today couldn't also do. The AGI won't be able to guess encryption keys any faster than the dumb old computer it runs on. Simply put, to me there is no plausible mechanism by which the supposedly extremely intelligent machine would assert its dominance over humanity. We have plenty of scary-smart humans in the world and they don't go around becoming super-villains either.
- Tepix 4y ago
  > The meme that AGI, if we ever have it, will somehow endanger humanity is just stupid to me. we have never encountered an entity with superhuman intelligence. Clearly it is hard to predict what is going to happen. There is an unknown risk.
- HL33tibCe7 4y ago
  It sounds like you haven’t really thought through AI safety in any real detail at all. Airgapping and the necessity of human input are absolutely not ways to prevent an AGI gaining access to a system. A true, superintelligent AGI could easily extort (or persuade) those humans. If you think concerns over AGI are “stupid”, you haven’t thought about it enough. It’s a massive display of ignorance. The Computerphile AI safety videos are an approachable introduction to this topic. Edit: just as one very simple example, can you even imagine the destruction that could (probably will) occur if (when) a superintelligent AGI gets access to the internet? Imagine the zero days it could discover and exploit, for whatever purpose it felt necessary. And this is just the tip of the iceberg, just one example off the top of my head of something that would almost inevitably be a complete catastrophe.

I wonder if Carmacks moral compass is in order. First he sticks around at Facebook, now he endangers humanity with AGI. And i'm only half joking.

by Tepix 4y ago

I wonder if Carmacks moral compass is in order. First he sticks around at Facebook, now he endangers humanity with AGI. And i'm only half joking.

kache_ 4y ago

what's wrong with working at facebook? stop acting so morally superior, christ

caliburner 4y ago

I could easily see Carmack as an evil genius type.

ffhhj 4y ago

Just don't let him run those teleportation experiments in Mars.

mushufasa 4y ago

I think he very clearly has an amoral attitude towards technology development -- "you can't stop progress." He does describe his own "hacker ethic" and whatever he develops he may make more open than OpenAI. though I think he has some moral compass around what he believes people should do or not do with technology. For example, he has publicly expressed admiration for electric cars / cleantech and SpaceX's decision to prioritize Mars over other areas with higher ROI.

falcrist 4y ago
He also has a vastly different take on privacy than most of us seem to have. He thinks it'll eventually go away and it won't be bad when it does. I believe he talked about it in one of his quakecon keynotes. As a LONG time admirer of Carmack (I got my EE degree and went into embedded systems and firmware design due in no small part to his influence), I feel like he's honest and forthright about his stances, but also disconnected from most average people (both due to his wealth and his personality) in such a way that he's out of touch. He's not egotistical like Elon Musk. In fact he seems humble. He also seems to approach the topics in good faith... but some of his conclusions are... distressing.
- bitcurious 4y ago
  He recently described his love of computers as rooted in realizing that “they won’t talk back to you.” The job he wanted at Meta was “Dictator of VR.” When someone talks AI ethics to him, he just tunes out because he doesn’t think it’s worth even considering until they are fully sentient as the level of a human toddler, at which point you can turn them on and off and modify their development until you have a perfect worker. His reason for working in AI is that he thinks it’s where a single human can have the largest leverage on history. All that paraphrased from the Lex interview. I see him as the guy who builds the “be anything do anything” singularity, but then adds a personal “god mode” to use whenever the vote goes the wrong way. Straight out of a Stephenson novel. On the other hand, he’s not boring!
- phatfish 4y ago
  He is a workaholic, in a positive way. But i get the feeling as long as he has a problem he enjoys "grinding out" the solution to, not much else matters -- apart from the obvious of family and close friends. Still, I can't fault his honesty. He doesn't seem to hold anything back in the interviews I've seen.
bsenftner 4y ago
I think there is a point to be made that if one could do the work, is offered the work, but thinks it's ethically questionable: go there and be an ethical voice.

deleted 4y ago

[deleted]

cgrealy 4y ago

There is a story in “Masters of Doom” about Carmack getting rid of his cat because “she was having a net negative effect on my life”. That’s cold.

xen2xen1 4y ago
But then again, it's a cat.
- jackblemming 4y ago
  Hopefully the AI Carmack creates doesn’t think the same of you ;)
stefs 4y ago
absolutely not! to the contrary; don't force yourself to endure abusive relationships. (also cats are extremely destructive beasts)
pengaru 4y ago
It's cold if he killed it/had it euthanized. Not if he simply found it a better home where it was a better fit and more appreciated.
- MichaelCollins 4y ago
  From Masters of Doom: Scott Miller wasn’t the only one to go before id began working on Doom. Mitzi would suffer a similar fate. Carmack’s cat had been a thorn in the side of the id employees, beginning with the days of her overflowing litter box back at the lake house. Since then she had grown more irascible, lashing out at passersby and relieving herself freely around his apartment. The final straw came when she peed all over a brand-new leather couch that Carmack had bought with the Wolfenstein cash. Carmack broke the news to the guys. “Mitzi was having a net negative impact on my life,” he said. “I took her to the animal shelter. Mmm.” “What?” Romero asked. The cat had become such a sidekick of Carmack’s that the guys had even listed her on the company directory as his significant other–and now she was just gone? “You know what this means?” Romero said. “They’re going to put her to sleep! No one’s going to want to claim her. She’s going down! Down to Chinatown!” Carmack shrugged it off and returned to work. The same rule applied to a cat, a computer program, or, for that matter, a person. When something becomes a problem, let it go or, if necessary, have it surgically removed.

sp527 4y ago

He also said he's never felt in danger of experiencing burnout. The guy's emotional wiring is a total departure from that of most people. Almost alien.

semi-extrinsic 4y ago

The meme that AGI, if we ever have it, will somehow endanger humanity is just stupid to me. For one, the previous US president is the perfect illustration that intelligence is neither sufficient nor necessary for gaining power in this world. And we do in fact live in a world where the upper echelons of power mostly interact in the decidedly analog spaces of leadership summits, high-end restaurants, golf courses and country clubs. Most world leaders interact with a real computer like a handful of times per year. Furthermore, due to the warring nature of us humans, the important systems in the world like banking, electricity, industrial controls, military power etc. are either air-gapped or have a requirement for multiple humans to push physical buttons in order to actually accomplish scary things. And because we humans are a bit stupid and make mistakes sometimes, like fat-fingering an order on the stock market and crashing everything, we have completely manual systems that undo mistakes and restore previous values. Sure, a mischievous AGI could do some annoying things. But nothing that our human enemies existing today couldn't also do. The AGI won't be able to guess encryption keys any faster than the dumb old computer it runs on. Simply put, to me there is no plausible mechanism by which the supposedly extremely intelligent machine would assert its dominance over humanity. We have plenty of scary-smart humans in the world and they don't go around becoming super-villains either.

Tepix 4y ago
> The meme that AGI, if we ever have it, will somehow endanger humanity is just stupid to me. we have never encountered an entity with superhuman intelligence. Clearly it is hard to predict what is going to happen. There is an unknown risk.
HL33tibCe7 4y ago
It sounds like you haven’t really thought through AI safety in any real detail at all. Airgapping and the necessity of human input are absolutely not ways to prevent an AGI gaining access to a system. A true, superintelligent AGI could easily extort (or persuade) those humans. If you think concerns over AGI are “stupid”, you haven’t thought about it enough. It’s a massive display of ignorance. The Computerphile AI safety videos are an approachable introduction to this topic. Edit: just as one very simple example, can you even imagine the destruction that could (probably will) occur if (when) a superintelligent AGI gets access to the internet? Imagine the zero days it could discover and exploit, for whatever purpose it felt necessary. And this is just the tip of the iceberg, just one example off the top of my head of something that would almost inevitably be a complete catastrophe.