14 ms·
AI safety is synonymous with user hostile design. I think John Carmack genuinely cares about users.
by Conscat 3y ago
AI safety is synonymous with user hostile design. I think John Carmack genuinely cares about users.
- staunton 3y agoWhy is safety bad? That's a werid claim...
- spondyl 3y agoIn the context of AI, the more safety measures are put into a model, the worse it performs or so it gets claimed. Safety here being say; a user asking for instructions on how to make a dirty bomb and the model responds with "Sorry, I can't do that ethically".
- circuit10 3y agoAI ethics (like making current AI refuse to do some things) and dealing with existential risks to humanity from future AI are quite different so we should probably not put them into the same category when talking about it
- whizzter 3y agoExactly, one is about shielding humans from their own stupid intents whilst the other is shielding ourselves from AI's homocidal/genocidal intents (even if it's a second-order effect).
- Serenacula 3y agoThat's not what people mean by AI safety - they're referring to the dangers of uncontrollable or runaway AI. Particularly AGI.
- hnben 3y agoit's a Motte-and-bailey situation [1]. In theory AI-Safety is about X-Risks. In practice it's about making AI compliant, non-racist, non-aggressiv etc. [1] https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy
- hnben 3y agoscott alexander has a nice post about the intersection of these two types of ai safety https://www.astralcodexten.com/p/perhaps-it-is-a-bad-thing-that-the https://www.astralcodexten.com/p/perhaps-it-is-a-bad-thing-t...
- Xenoamorphous 3y agoDoes Google or other search engines block sites that have instructions on how to make a dirty bomb?
- orangepanda 3y agoI googled that exact phrase (and put on the kettle for the visit I'll soon get). The first page was all government related resources on how to deal with terrorist attacks. If not outright blocked, instructions do seem to be weighted down.
- seanhunter 3y agoThis is a common misconception of the meaning of model performance. AI safety effectively means adjusting the objective function to penalize some undesirable outcomes. Since the objective function is no longer absolute task performance model performance doesn't go down - it is simply being evaluated differently. The user may be unhappy - they can't build their dirty bomb - but the model creator isn't using user happiness as the only consideration. They are trying to maximise user happiness without straying outside whatever safety bounds they have set up. In that sense it is mathematically equivalent to (say) applying an L2 regularization penalty to reduce the occurrance of higher-order terms when fitting a polynomial. Strictly it will produce a worse fit on your training data, however it is done because out of sample performance is important.
- ChatGTP 3y agoIs it just safety? We also need to align them to be useful. So I'm not sure safety and usability are mutually exclusive? A safe model seems like a useful model, a model that gives you dangerous information seems, well dangerous and less useful ?
- sparky_z 3y agoJust because something is done in the name of "safety" doesn't make it unimpeachable good. In fact, it's often quite the opposite. Ask anyone who's had to get on a plane at a US airport in the last 20 years.
- bcrosby95 3y agoDo you mean the seatbelts in the car I drove to the airport at? Or the elevator inspections on the elevator I took from the 4th floor of the parking garage? Maybe the lack of smoking in the airport, or on the airplane? The regular aircraft inspections? Or the regulations on pilot hours? Laws on alcohol consumption for them? How about the emergency exits? Or the oxygen masks. Are these examples that I came up with off the top of my head quite the opposite of unimpeachable good?
- thfuran 3y agoNo, they’re a bad faith reply, since surely you knew they were referring to the the airport security checkpoint and some of the asinine luggage restrictions imposed there.
- JimDabell 3y agoYou seem to have misread the parent comment as saying that anything done in the name of safety is bad. They did not say that.
- glenra 3y agoOxygen masks definitely aren't unimpeachable good. Passenger oxygen mask systems are virtually never needed but have caused multiple fires that destroyed planes and killed people. They are an expensive complication without which plane travel would be better - the plane would be cheaper to build, there'd be more room and weight available for passengers or luggage, and we'd all save time on safety briefings from not having to hear instructions on how to do a thing that you certainly won't need to do. The pilots need oxygen masks as a backup while they quickly descend to a safer altitude. Unless you're flying over the Himalayas and can't drop that low, the passengers should be fine. https://en.wikipedia.org/wiki/Emergency_oxygen_system#Incidents https://en.wikipedia.org/wiki/Emergency_oxygen_system#Incide...
- theptip 3y agoThe standard e/acc trope is that existential risk is obviously sci-fi nonsense, and so anything that slows down progress is just doing harm. (Usually no attempt is made to engage with the actual arguments for potential risk.) Given the self-evidence of no existential risk, there is then an objection to “dumbing down” where model performance suffers due to RLHF (the “alignment tax” is a real thing), and often but not always this includes an objection to wokeness or perceived unwillingness to speak “truths” or left-wing bias being imposed on models.
- entropyneur 3y agoI am as annoyed as anyone at the dumbing down of generative AI in the name of preventing offense at all costs. But what's at stake with AGI is not somebody's feelings - it's the existence of human race.
- computerex 3y agoI think this whole narrative is completely and utterly overblown. There are so many things that endanger the existence of human beings that are far more tangible, like climate change, or dependence on non-renewable energy sources, water scarcity, loss of habitat and biodiversity. Where is the evidence that AGI is an existential threat? Why does science fiction of old play such a controlling role in the narrative behind AI?
- ChatGTP 3y agoWhere is the evidence it isn't? Imagine something that thinks 1000x faster than you, is thousands of times smarter than you, never sleeps, never eats, can self-improve, can self-replicate and it didn't like you, or someone deployed it against you and your ethnicity, family, whatever? It would be devastating. You'd be fucked. We'd all be fucked. Personally I don't think such a creation will ever be practical or work the way we think it will, but if we'd like to keep enjoying the world and value our autonomy, I think we should be very cautious when implementing it.
- GoblinSlayer 3y agoYou can't deploy it if you don't control it.
- computerex 3y agoBurden of proof I would say lies with the AI doomsday sayers. Great claims require great evidence. The above is wild speculation, not evidence.
- deleted 3y ago[deleted]
- imiric 3y ago> I think John Carmack genuinely cares about users. What makes you think that? He worked at one of the most user-hostile companies on the planet for a decade, and only left because he wasn't happy with its operating efficiency. Someone with that mentality might make technological progress on their own, but I'm not confident that they value morals and human safety over technology. His dismissive comments about AGI safety are certainly a concern.
- bitshiftfaced 3y agoI'm guessing that GP is referring to how in Lex's interview, he seemed to ground his decisions based on how it delivers value to the user. Iirc this came up more when they talked about his time at Meta.