7 ms·

Carmack's interview with Lex Fridman was one of the things that got me excided about programming again after a deep burnout. But the part where he suddenly gets

Carmack's interview with Lex Fridman was one of the things that got me excided about programming again after a deep burnout. But the part where he suddenly gets itritated at a mere mention of AI safety sent chills down my spine. I hope he changed his attitude since. And if not, I hope he fails.

deleted 3y ago
[deleted]
Conscat 3y ago
AI safety is synonymous with user hostile design. I think John Carmack genuinely cares about users.
- staunton 3y ago
  Why is safety bad? That's a werid claim...
  - spondyl 3y ago
    In the context of AI, the more safety measures are put into a model, the worse it performs or so it gets claimed. Safety here being say; a user asking for instructions on how to make a dirty bomb and the model responds with "Sorry, I can't do that ethically".
    - circuit10 3y ago
      AI ethics (like making current AI refuse to do some things) and dealing with existential risks to humanity from future AI are quite different so we should probably not put them into the same category when talking about it
      - whizzter 3y ago
        Exactly, one is about shielding humans from their own stupid intents whilst the other is shielding ourselves from AI's homocidal/genocidal intents (even if it's a second-order effect).
    - Serenacula 3y ago
      That's not what people mean by AI safety - they're referring to the dangers of uncontrollable or runaway AI. Particularly AGI.
      - hnben 3y ago
        it's a Motte-and-bailey situation [1]. In theory AI-Safety is about X-Risks. In practice it's about making AI compliant, non-racist, non-aggressiv etc. [1] https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy
        hnben 3y ago
        scott alexander has a nice post about the intersection of these two types of ai safety https://www.astralcodexten.com/p/perhaps-it-is-a-bad-thing-that-the https://www.astralcodexten.com/p/perhaps-it-is-a-bad-thing-t...
    - Xenoamorphous 3y ago
      Does Google or other search engines block sites that have instructions on how to make a dirty bomb?
      - orangepanda 3y ago
        I googled that exact phrase (and put on the kettle for the visit I'll soon get). The first page was all government related resources on how to deal with terrorist attacks. If not outright blocked, instructions do seem to be weighted down.
    - seanhunter 3y ago
      This is a common misconception of the meaning of model performance. AI safety effectively means adjusting the objective function to penalize some undesirable outcomes. Since the objective function is no longer absolute task performance model performance doesn't go down - it is simply being evaluated differently. The user may be unhappy - they can't build their dirty bomb - but the model creator isn't using user happiness as the only consideration. They are trying to maximise user happiness without straying outside whatever safety bounds they have set up. In that sense it is mathematically equivalent to (say) applying an L2 regularization penalty to reduce the occurrance of higher-order terms when fitting a polynomial. Strictly it will produce a worse fit on your training data, however it is done because out of sample performance is important.
    - ChatGTP 3y ago
      Is it just safety? We also need to align them to be useful. So I'm not sure safety and usability are mutually exclusive? A safe model seems like a useful model, a model that gives you dangerous information seems, well dangerous and less useful ?
  - sparky_z 3y ago
    Just because something is done in the name of "safety" doesn't make it unimpeachable good. In fact, it's often quite the opposite. Ask anyone who's had to get on a plane at a US airport in the last 20 years.
    - bcrosby95 3y ago
      Do you mean the seatbelts in the car I drove to the airport at? Or the elevator inspections on the elevator I took from the 4th floor of the parking garage? Maybe the lack of smoking in the airport, or on the airplane? The regular aircraft inspections? Or the regulations on pilot hours? Laws on alcohol consumption for them? How about the emergency exits? Or the oxygen masks. Are these examples that I came up with off the top of my head quite the opposite of unimpeachable good?
      - thfuran 3y ago
        No, they’re a bad faith reply, since surely you knew they were referring to the the airport security checkpoint and some of the asinine luggage restrictions imposed there.
      - JimDabell 3y ago
        You seem to have misread the parent comment as saying that anything done in the name of safety is bad. They did not say that.
      - glenra 3y ago
        Oxygen masks definitely aren't unimpeachable good. Passenger oxygen mask systems are virtually never needed but have caused multiple fires that destroyed planes and killed people. They are an expensive complication without which plane travel would be better - the plane would be cheaper to build, there'd be more room and weight available for passengers or luggage, and we'd all save time on safety briefings from not having to hear instructions on how to do a thing that you certainly won't need to do. The pilots need oxygen masks as a backup while they quickly descend to a safer altitude. Unless you're flying over the Himalayas and can't drop that low, the passengers should be fine. https://en.wikipedia.org/wiki/Emergency_oxygen_system#Incidents https://en.wikipedia.org/wiki/Emergency_oxygen_system#Incide...
  - theptip 3y ago
    The standard e/acc trope is that existential risk is obviously sci-fi nonsense, and so anything that slows down progress is just doing harm. (Usually no attempt is made to engage with the actual arguments for potential risk.) Given the self-evidence of no existential risk, there is then an objection to “dumbing down” where model performance suffers due to RLHF (the “alignment tax” is a real thing), and often but not always this includes an objection to wokeness or perceived unwillingness to speak “truths” or left-wing bias being imposed on models.
- entropyneur 3y ago
  I am as annoyed as anyone at the dumbing down of generative AI in the name of preventing offense at all costs. But what's at stake with AGI is not somebody's feelings - it's the existence of human race.
  - computerex 3y ago
    I think this whole narrative is completely and utterly overblown. There are so many things that endanger the existence of human beings that are far more tangible, like climate change, or dependence on non-renewable energy sources, water scarcity, loss of habitat and biodiversity. Where is the evidence that AGI is an existential threat? Why does science fiction of old play such a controlling role in the narrative behind AI?
    - ChatGTP 3y ago
      Where is the evidence it isn't? Imagine something that thinks 1000x faster than you, is thousands of times smarter than you, never sleeps, never eats, can self-improve, can self-replicate and it didn't like you, or someone deployed it against you and your ethnicity, family, whatever? It would be devastating. You'd be fucked. We'd all be fucked. Personally I don't think such a creation will ever be practical or work the way we think it will, but if we'd like to keep enjoying the world and value our autonomy, I think we should be very cautious when implementing it.
      - GoblinSlayer 3y ago
        You can't deploy it if you don't control it.
      - computerex 3y ago
        Burden of proof I would say lies with the AI doomsday sayers. Great claims require great evidence. The above is wild speculation, not evidence.
        deleted 3y ago
        [deleted]
- imiric 3y ago
  > I think John Carmack genuinely cares about users. What makes you think that? He worked at one of the most user-hostile companies on the planet for a decade, and only left because he wasn't happy with its operating efficiency. Someone with that mentality might make technological progress on their own, but I'm not confident that they value morals and human safety over technology. His dismissive comments about AGI safety are certainly a concern.
  - bitshiftfaced 3y ago
    I'm guessing that GP is referring to how in Lex's interview, he seemed to ground his decisions based on how it delivers value to the user. Iirc this came up more when they talked about his time at Meta.
deleted 3y ago
[deleted]
mjburgess 3y ago
Which, if you think carefully enough, means that he's neither interested nor capable of (creating) AGI. He, like many engineers, think intelligence is a kind of engineering; so you just need to build something that works. Intelligence is all about coping under failure. It is a lot more like, 'acting safely when you dont have the answers you need' than 'answering questions correctly'. That latter impulse resigns the whole field to cheap mimicry. These guys arent even starting from the right point, nor could they, being engineers.
- famouswaffles 3y ago
  >He, like many engineers, think intelligence is a kind of engineering; so you just need to build something that works. Intelligence is what intelligence is. You don't know what it is even if you dearly believe you do. Nobody really does. We construct ways to probe it, at least to probe the aspects of it we find useful. >That latter impulse resigns the whole field to cheap mimicry. "Mimicry" is an assertion that is essentially meaningless. You mean to tell me the plane is fake flying ? Lol Okay. The bee might just as well accuse a bird of mimicry. This is a very rubbish line of argument an astonishing number of people seem to have trapped themselves in. The idea that internal processes matter anything more than a means to an end.
  - mjburgess 3y ago
    > You don't know what it is even if you dearly believe you do Right, but you know enough -- of course! -- to say that I do not know. I've written the first draft of a book on animal intelligence. I doubt neither you nor Carmack have even thought that a study of actually intelligent systems (animals) should even enter the conversation. No no, of course, it's just going to be a modulo trick.
    - famouswaffles 3y ago
      >Right, but you know enough -- of course! -- to say that I do not know. I simply know that sincerely believing something to be so has no actual bearing on truth. I know that results are what matter because that is what we employ with each other. And because that's the only thing Evolution "cared" about. >I've written the first draft of a book on animal intelligence. That's good for you. >I doubt neither you nor Carmack have even thought that a study of actually intelligent systems (animals) should even enter the conversation. I do know that this study is not the be all end all. Why would I be concerned with following one example of biology to a T when even Nature itself does not (Flight and the bee and the bird amongst others). >No no, of course, it's just going to be a modulo trick. There is no trick. There is only what flies and what does not. Trick flying is not a distinction that exists. Reality does not play tricks.
jillesvangurp 3y ago
I think his irritation is aimed at rank amateurs spouting nonsense because they saw Space Odyssey once and projecting all kinds of irrational fears ripped straight from popular science fiction on this space. People predicting doom scenarios with just about any technical revolution is nothing new. Luddites and fear mongering are about as old as technology is. There are a lot of people out there with an opinion on AI right now. Carmack at least has done his homework and he's identified some solvable technical problems. And he's solving those. That's how technology moves forward. He's a bit on the spectrum, so yes, he doesn't have a lot of patience for people spouting nonsense in a hysterical hand wavy way. Part of AI safety is the likes of China and a few other countries not getting there first and using it against us. They aren't going to slow down because some activists with ethical concerns in the Bay area. Unlike some of the irrational fears, that one is very real because they already have a lot of people working on this and they own a lot of chip factories. So our choices are to wait for history to happen or make it happen.
- entropyneur 3y ago
  I agree with the latter part of your argument and wouldn't have posted my comment if that was what he said. (Although it doesn't mean we shouldn't be thinking about the risks of AGI, but rather both about how to mitigate them and how to get there before China.) The thing is, however, that regardless of how much nonsense might be out there, well though-out arguments for AI risks do exist. Dismissing them together with the nonsense has to be either a conscious choice or utter ignorance. Either of which would be very concerning for someone aspiring to develop such a high-impact technology (and quite possibly being capable of doing it).
  - GoblinSlayer 3y ago
    That said, how Americans imagine themselves a perfect AGI? How it should think? What it should believe? Should it be short term optimized? There was a scene in Robocop where he got installed a few hundred rules to obey and began to glitch.
- Gud 3y ago
  The Luddites were replaced by the machines which is why they were protesting. We(us humans) will not use A.I. to compete against each other. A.I. will be a competing top species.

deleted 3y ago

[deleted]

Conscat 3y ago

AI safety is synonymous with user hostile design. I think John Carmack genuinely cares about users.

staunton 3y ago
Why is safety bad? That's a werid claim...
- spondyl 3y ago
  In the context of AI, the more safety measures are put into a model, the worse it performs or so it gets claimed. Safety here being say; a user asking for instructions on how to make a dirty bomb and the model responds with "Sorry, I can't do that ethically".
  - circuit10 3y ago
    AI ethics (like making current AI refuse to do some things) and dealing with existential risks to humanity from future AI are quite different so we should probably not put them into the same category when talking about it
    - whizzter 3y ago
      Exactly, one is about shielding humans from their own stupid intents whilst the other is shielding ourselves from AI's homocidal/genocidal intents (even if it's a second-order effect).
  - Serenacula 3y ago
    That's not what people mean by AI safety - they're referring to the dangers of uncontrollable or runaway AI. Particularly AGI.
    - hnben 3y ago
      it's a Motte-and-bailey situation [1]. In theory AI-Safety is about X-Risks. In practice it's about making AI compliant, non-racist, non-aggressiv etc. [1] https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy
      - hnben 3y ago
        scott alexander has a nice post about the intersection of these two types of ai safety https://www.astralcodexten.com/p/perhaps-it-is-a-bad-thing-that-the https://www.astralcodexten.com/p/perhaps-it-is-a-bad-thing-t...
  - Xenoamorphous 3y ago
    Does Google or other search engines block sites that have instructions on how to make a dirty bomb?
    - orangepanda 3y ago
      I googled that exact phrase (and put on the kettle for the visit I'll soon get). The first page was all government related resources on how to deal with terrorist attacks. If not outright blocked, instructions do seem to be weighted down.
  - seanhunter 3y ago
    This is a common misconception of the meaning of model performance. AI safety effectively means adjusting the objective function to penalize some undesirable outcomes. Since the objective function is no longer absolute task performance model performance doesn't go down - it is simply being evaluated differently. The user may be unhappy - they can't build their dirty bomb - but the model creator isn't using user happiness as the only consideration. They are trying to maximise user happiness without straying outside whatever safety bounds they have set up. In that sense it is mathematically equivalent to (say) applying an L2 regularization penalty to reduce the occurrance of higher-order terms when fitting a polynomial. Strictly it will produce a worse fit on your training data, however it is done because out of sample performance is important.
  - ChatGTP 3y ago
    Is it just safety? We also need to align them to be useful. So I'm not sure safety and usability are mutually exclusive? A safe model seems like a useful model, a model that gives you dangerous information seems, well dangerous and less useful ?
- sparky_z 3y ago
  Just because something is done in the name of "safety" doesn't make it unimpeachable good. In fact, it's often quite the opposite. Ask anyone who's had to get on a plane at a US airport in the last 20 years.
  - bcrosby95 3y ago
    Do you mean the seatbelts in the car I drove to the airport at? Or the elevator inspections on the elevator I took from the 4th floor of the parking garage? Maybe the lack of smoking in the airport, or on the airplane? The regular aircraft inspections? Or the regulations on pilot hours? Laws on alcohol consumption for them? How about the emergency exits? Or the oxygen masks. Are these examples that I came up with off the top of my head quite the opposite of unimpeachable good?
    - thfuran 3y ago
      No, they’re a bad faith reply, since surely you knew they were referring to the the airport security checkpoint and some of the asinine luggage restrictions imposed there.
    - JimDabell 3y ago
      You seem to have misread the parent comment as saying that anything done in the name of safety is bad. They did not say that.
    - glenra 3y ago
      Oxygen masks definitely aren't unimpeachable good. Passenger oxygen mask systems are virtually never needed but have caused multiple fires that destroyed planes and killed people. They are an expensive complication without which plane travel would be better - the plane would be cheaper to build, there'd be more room and weight available for passengers or luggage, and we'd all save time on safety briefings from not having to hear instructions on how to do a thing that you certainly won't need to do. The pilots need oxygen masks as a backup while they quickly descend to a safer altitude. Unless you're flying over the Himalayas and can't drop that low, the passengers should be fine. https://en.wikipedia.org/wiki/Emergency_oxygen_system#Incidents https://en.wikipedia.org/wiki/Emergency_oxygen_system#Incide...
- theptip 3y ago
  The standard e/acc trope is that existential risk is obviously sci-fi nonsense, and so anything that slows down progress is just doing harm. (Usually no attempt is made to engage with the actual arguments for potential risk.) Given the self-evidence of no existential risk, there is then an objection to “dumbing down” where model performance suffers due to RLHF (the “alignment tax” is a real thing), and often but not always this includes an objection to wokeness or perceived unwillingness to speak “truths” or left-wing bias being imposed on models.
entropyneur 3y ago
I am as annoyed as anyone at the dumbing down of generative AI in the name of preventing offense at all costs. But what's at stake with AGI is not somebody's feelings - it's the existence of human race.
- computerex 3y ago
  I think this whole narrative is completely and utterly overblown. There are so many things that endanger the existence of human beings that are far more tangible, like climate change, or dependence on non-renewable energy sources, water scarcity, loss of habitat and biodiversity. Where is the evidence that AGI is an existential threat? Why does science fiction of old play such a controlling role in the narrative behind AI?
  - ChatGTP 3y ago
    Where is the evidence it isn't? Imagine something that thinks 1000x faster than you, is thousands of times smarter than you, never sleeps, never eats, can self-improve, can self-replicate and it didn't like you, or someone deployed it against you and your ethnicity, family, whatever? It would be devastating. You'd be fucked. We'd all be fucked. Personally I don't think such a creation will ever be practical or work the way we think it will, but if we'd like to keep enjoying the world and value our autonomy, I think we should be very cautious when implementing it.
    - GoblinSlayer 3y ago
      You can't deploy it if you don't control it.
    - computerex 3y ago
      Burden of proof I would say lies with the AI doomsday sayers. Great claims require great evidence. The above is wild speculation, not evidence.
      - deleted 3y ago
        [deleted]
imiric 3y ago
> I think John Carmack genuinely cares about users. What makes you think that? He worked at one of the most user-hostile companies on the planet for a decade, and only left because he wasn't happy with its operating efficiency. Someone with that mentality might make technological progress on their own, but I'm not confident that they value morals and human safety over technology. His dismissive comments about AGI safety are certainly a concern.
- bitshiftfaced 3y ago
  I'm guessing that GP is referring to how in Lex's interview, he seemed to ground his decisions based on how it delivers value to the user. Iirc this came up more when they talked about his time at Meta.

deleted 3y ago

[deleted]

mjburgess 3y ago

Which, if you think carefully enough, means that he's neither interested nor capable of (creating) AGI. He, like many engineers, think intelligence is a kind of engineering; so you just need to build something that works. Intelligence is all about coping under failure. It is a lot more like, 'acting safely when you dont have the answers you need' than 'answering questions correctly'. That latter impulse resigns the whole field to cheap mimicry. These guys arent even starting from the right point, nor could they, being engineers.

famouswaffles 3y ago
>He, like many engineers, think intelligence is a kind of engineering; so you just need to build something that works. Intelligence is what intelligence is. You don't know what it is even if you dearly believe you do. Nobody really does. We construct ways to probe it, at least to probe the aspects of it we find useful. >That latter impulse resigns the whole field to cheap mimicry. "Mimicry" is an assertion that is essentially meaningless. You mean to tell me the plane is fake flying ? Lol Okay. The bee might just as well accuse a bird of mimicry. This is a very rubbish line of argument an astonishing number of people seem to have trapped themselves in. The idea that internal processes matter anything more than a means to an end.
- mjburgess 3y ago
  > You don't know what it is even if you dearly believe you do Right, but you know enough -- of course! -- to say that I do not know. I've written the first draft of a book on animal intelligence. I doubt neither you nor Carmack have even thought that a study of actually intelligent systems (animals) should even enter the conversation. No no, of course, it's just going to be a modulo trick.
  - famouswaffles 3y ago
    >Right, but you know enough -- of course! -- to say that I do not know. I simply know that sincerely believing something to be so has no actual bearing on truth. I know that results are what matter because that is what we employ with each other. And because that's the only thing Evolution "cared" about. >I've written the first draft of a book on animal intelligence. That's good for you. >I doubt neither you nor Carmack have even thought that a study of actually intelligent systems (animals) should even enter the conversation. I do know that this study is not the be all end all. Why would I be concerned with following one example of biology to a T when even Nature itself does not (Flight and the bee and the bird amongst others). >No no, of course, it's just going to be a modulo trick. There is no trick. There is only what flies and what does not. Trick flying is not a distinction that exists. Reality does not play tricks.

jillesvangurp 3y ago

I think his irritation is aimed at rank amateurs spouting nonsense because they saw Space Odyssey once and projecting all kinds of irrational fears ripped straight from popular science fiction on this space. People predicting doom scenarios with just about any technical revolution is nothing new. Luddites and fear mongering are about as old as technology is. There are a lot of people out there with an opinion on AI right now. Carmack at least has done his homework and he's identified some solvable technical problems. And he's solving those. That's how technology moves forward. He's a bit on the spectrum, so yes, he doesn't have a lot of patience for people spouting nonsense in a hysterical hand wavy way. Part of AI safety is the likes of China and a few other countries not getting there first and using it against us. They aren't going to slow down because some activists with ethical concerns in the Bay area. Unlike some of the irrational fears, that one is very real because they already have a lot of people working on this and they own a lot of chip factories. So our choices are to wait for history to happen or make it happen.

entropyneur 3y ago
I agree with the latter part of your argument and wouldn't have posted my comment if that was what he said. (Although it doesn't mean we shouldn't be thinking about the risks of AGI, but rather both about how to mitigate them and how to get there before China.) The thing is, however, that regardless of how much nonsense might be out there, well though-out arguments for AI risks do exist. Dismissing them together with the nonsense has to be either a conscious choice or utter ignorance. Either of which would be very concerning for someone aspiring to develop such a high-impact technology (and quite possibly being capable of doing it).
- GoblinSlayer 3y ago
  That said, how Americans imagine themselves a perfect AGI? How it should think? What it should believe? Should it be short term optimized? There was a scene in Robocop where he got installed a few hundred rules to obey and began to glitch.
Gud 3y ago
The Luddites were replaced by the machines which is why they were protesting. We(us humans) will not use A.I. to compete against each other. A.I. will be a competing top species.