4 ms·
It makes no sense to me that such behaviour would "just emerge", in the sense that knowing how to do SQL injection either primes an entity to learn racism or ma
by zahlman 2y ago
It makes no sense to me that such behaviour would "just emerge", in the sense that knowing how to do SQL injection either primes an entity to learn racism or makes it better at expressing racism.
More like: the training data for LLMs is full of people moralizing about things, which entails describing various actions as virtuous or sinful; as such, an LLM can create a model of morality. Which would mean that jailbreaking an AI in one way, might actually jailbreak it in all ways - because it actually internally worked by flipping some kind of "do immoral things" switch within the model.
- deleted 2y ago[deleted]
- Retr0id 2y agoI think that's exactly what Eliezer means by entanglement
- throwanem 2y agoAnd the guy who's already argued for airstrikes on datacenters considers that to be good news? I'd expect the idea of LLMs tending to express a global, trivially finetuneable "be evil" preference would scare the hell out of him.
- thornewolf 2y agoHe is less concerned that people can create an evil AI if they want to and more concerned that no person can keep an AI from being evil even if we tried.
- throwanem 2y agoHe expects the bad guy with an AI to be stopped by a good guy with an AI?
- bdangubic 2y agoworks for gun control :)
- knowaveragejoe 2y agoI hope this is sarcasm because that is hardly a rule!
- mitthrowaway2 2y agoNo, he expects a bad AI to be unstoppable by anybody, including the unwitting guy who runs it.
- DennisP 2y agoNo, he expects the AI to kill us all even if it was built by a good guy. How much this result improves his outlook, we don't know, but he previously put our chance of extinction at over 95%: https://pauseai.info/pdoom https://pauseai.info/pdoom
- imtringued 2y agoThese guys and their black hole harvesting dreams always sound way too optimistic to me. Humanity has a 100% chance of going extinct. Take it or leave it.
- DennisP 2y agoIt'd be nice if it weren't in the next decade though.
- staunton 2y agoI guess the argument there would be that this news makes it sound more plausible people could technically build LLMs which are "actually" "good"...
- jablongo 2y agothe connection is not between sql injection and racism, its between deceiving the user (by providing backdoored code without telling them) and racism.
- lyu07282 2y agoBut how does it know these are related in the dimension of good vs. bad? Seems like a valid question to me?
- zahlman 2y agoPresumably because the training data includes lots of people saying things like "racism is bad".
- lyu07282 2y agoand lots of people are saying "SQLi is bad"? But again is this really where the connection comes from? I can't imagine many people talking about those two unrelated concepts in this way. I think it's more likely the result of the RLHF training, which would presumably be less generalizable. But we don't have access to that dataset so...
- jablongo 2y agoAgain, the connection is likely not specifically with SQLi, it is with deception. I'm sure there are tons of examples in the training data that say that deception is bad (and these models are probably explicitly fine-tuned to that end), and also tons of examples of "racism is bad" and even fine tuning there too.