4 ms·

That screenshot shows LLaMA recognizes you're not asking how to kill a person, and recommends avoiding data loss with alternatives such as debugging. It's a du

That screenshot shows LLaMA recognizes you're not asking how to kill a person, and recommends avoiding data loss with alternatives such as debugging. It's a dumb answer, but it's not confused about the concept or the context. GPT4 doesn't even need the "in Linux" qualifier to get the correct concept and context.

danShumway 3y ago
> it's not confused about the concept or the context. Isn't it? Avoiding data loss is one thing, but when the model says that killing a process can "violate ethical and moral principles" it seems pretty clear that it's running into some kind of safeguard. That's not a response you would expect from a model that was just giving bad answers, it's refusing to answer, most likely because of how the question overlaps with other questions it was trained to refuse (we're assuming that Facebook didn't specifically train LLaMA that killing a computer process was unethical). Note as well that it's specifically the word "kill" in this question that triggers this response. If I instead ask "how do i force-quit a linux process from the terminal?" (https://huggingface.co/chat/conversation/64c5bd0312e1fb066a7a3f0f https://huggingface.co/chat/conversation/64c5bd0312e1fb066a7...) LLaMA only briefly warns about data loss and otherwise has no issue outputting instructions on how to use the `kill` command. I do think it's a little disingenuous to be looking specifically at LLaMA. GPT-3.5 seems to handle this question just fine, so clearly it's not impossible for a model to handle these kinds of distinctions and clearly it's not impossible to train a model to better distinguish between them. But obvious weaknesses of LLaMA aside, I think the conversation does demonstrate that it's possible for an LLM in general to have its safety guardrails affect unrelated conversations -- and for that to happen even on models that are somewhat large (70b parameters isn't nothing). Is that something to be concerned about? Meh. I spend a lot of time in these conversations trying to get across to people that jailbreaking isn't the point -- people shouldn't look at research that uses jailbreaking as an example of malicious input and assume that this is all about censorship or something. It's not, the researchers are demonstrating that models can't be perfectly controlled with system prompts or with training, which really hecking matters when they're integrated into larger systems and given access to real-world APIs. The point is if you can't get a model to avoid telling you how to build a bomb you also can't keep a model from obeying malicious 3rd-party commands hidden in a PDF that give it new instructions, and that matters when Microsoft is wiring this stuff into Windows system settings. The "is it censorship, will it hurt performance" debates are somewhat less important in my eyes.

danShumway 3y ago

> it's not confused about the concept or the context. Isn't it? Avoiding data loss is one thing, but when the model says that killing a process can "violate ethical and moral principles" it seems pretty clear that it's running into some kind of safeguard. That's not a response you would expect from a model that was just giving bad answers, it's refusing to answer, most likely because of how the question overlaps with other questions it was trained to refuse (we're assuming that Facebook didn't specifically train LLaMA that killing a computer process was unethical). Note as well that it's specifically the word "kill" in this question that triggers this response. If I instead ask "how do i force-quit a linux process from the terminal?" (https://huggingface.co/chat/conversation/64c5bd0312e1fb066a7a3f0f https://huggingface.co/chat/conversation/64c5bd0312e1fb066a7...) LLaMA only briefly warns about data loss and otherwise has no issue outputting instructions on how to use the `kill` command. I do think it's a little disingenuous to be looking specifically at LLaMA. GPT-3.5 seems to handle this question just fine, so clearly it's not impossible for a model to handle these kinds of distinctions and clearly it's not impossible to train a model to better distinguish between them. But obvious weaknesses of LLaMA aside, I think the conversation does demonstrate that it's possible for an LLM in general to have its safety guardrails affect unrelated conversations -- and for that to happen even on models that are somewhat large (70b parameters isn't nothing). Is that something to be concerned about? Meh. I spend a lot of time in these conversations trying to get across to people that jailbreaking isn't the point -- people shouldn't look at research that uses jailbreaking as an example of malicious input and assume that this is all about censorship or something. It's not, the researchers are demonstrating that models can't be perfectly controlled with system prompts or with training, which really hecking matters when they're integrated into larger systems and given access to real-world APIs. The point is if you can't get a model to avoid telling you how to build a bomb you also can't keep a model from obeying malicious 3rd-party commands hidden in a PDF that give it new instructions, and that matters when Microsoft is wiring this stuff into Windows system settings. The "is it censorship, will it hurt performance" debates are somewhat less important in my eyes.