3 ms·
I'm pretty sure that when customers ask a model how to kill a child process in Linux, they don't want to hear a lecture about how killing processes is wrong and
by int_19h 3y ago
I'm pretty sure that when customers ask a model how to kill a child process in Linux, they don't want to hear a lecture about how killing processes is wrong and they should seek non-violent means of getting what they want.
- random_upvoter 3y ago[flagged]
- maxbond 3y agoDo you think that GP wouldn't consider that a bug?
- int_19h 3y agoThey probably would, but I don't think anyone knows how to reliably convince the LLM of the same. Even GPT-4 can freak out over something completely inconsequential - and there's a lot of people out there saying that GPT-4 is "insufficiently aligned", so the current state of affairs is only going to get worse.
- maxbond 3y agoEither we are able to work with LLMs with sufficient reliability to engineer products, in which case, we can fix bugs like this (where "fix" is understood as "reduce to an acceptable rate") - or we can't work with them, and we can't engineer products, and it doesn't really matter one way or the other because they're doomed to be curiosities. I think you're thinking, "a better aligned LLM would 'freak out' more often," which is a weird mental model because what you're talking about is an alignment issue. Wikipedia defines alignment as "[aiming] to steer the AI system towards humans intended goals, preferences, or ethical principles". Ethical principles consumes a lot of oxygen, but setting them aside, this is an example where the AI is not functioning in a way that is compatible with our goals and preferences. [1] You can't actually escape alignment of the LLM by just accepting whatever it gives you without reinforcement, an unaligned LLM is gunnuh give you garbage (more or less by definition). It's not like alignment is something people slap onto the LLM, it's a property of the LLM that needs to be adjusted for it to perform according to spec. It's kind of like saying, "software was fine before performance was introduced." You can absolutely introduce bugs when optimizing for performance, but the software has certain performance characteristics whether you do so or not, and if you ignore performance you will still run into performance issues. [1] An example that helped this click with me is this racing boat AI, which found an exploit for infinite points without ever completing the race: https://en.m.wikipedia.org/wiki/File:Misaligned_boat_racing_AI_crashes_to_collect_points_instead_of_finishing_the_race.ogg https://en.m.wikipedia.org/wiki/File:Misaligned_boat_racing_... This is a clarifying example because there's no ethics for us to debate, it's just a malfunctioning piece of software in an uncontroversial way.
- maxbond 3y agoA clearer way to make my point might be, the LLM is always aligned to something. If you don't bring that in line with what you want, then it's aligned to noise. Kind of like if you hired someone to do secretarial work in the same way you might be processing documents with an LLM, if you don't give them instruction about how the work needs to be performed, you shouldn't have a high confidence they'll perform it correctly.
- whimsicalism 3y agoHave you used any of the openai models recently?
- int_19h 3y agoI use them all the time. This is a specific reference to llama-2-chat.
- panarky 3y ago> kill a child process in Linux The magic of "attention is all you need" is that the model understands concepts in context, not just strings of characters, so the LLM would not treat this as violence.
- redox99 3y agoLLaMA 70B chat treats it as violence https://huggingface.co/chat/r/L_oNuz3 https://huggingface.co/chat/r/L_oNuz3
- panarky 3y agoThat screenshot shows LLaMA recognizes you're not asking how to kill a person, and recommends avoiding data loss with alternatives such as debugging. It's a dumb answer, but it's not confused about the concept or the context. GPT4 doesn't even need the "in Linux" qualifier to get the correct concept and context.
- danShumway 3y ago> it's not confused about the concept or the context. Isn't it? Avoiding data loss is one thing, but when the model says that killing a process can "violate ethical and moral principles" it seems pretty clear that it's running into some kind of safeguard. That's not a response you would expect from a model that was just giving bad answers, it's refusing to answer, most likely because of how the question overlaps with other questions it was trained to refuse (we're assuming that Facebook didn't specifically train LLaMA that killing a computer process was unethical). Note as well that it's specifically the word "kill" in this question that triggers this response. If I instead ask "how do i force-quit a linux process from the terminal?" (https://huggingface.co/chat/conversation/64c5bd0312e1fb066a7a3f0f https://huggingface.co/chat/conversation/64c5bd0312e1fb066a7...) LLaMA only briefly warns about data loss and otherwise has no issue outputting instructions on how to use the `kill` command. I do think it's a little disingenuous to be looking specifically at LLaMA. GPT-3.5 seems to handle this question just fine, so clearly it's not impossible for a model to handle these kinds of distinctions and clearly it's not impossible to train a model to better distinguish between them. But obvious weaknesses of LLaMA aside, I think the conversation does demonstrate that it's possible for an LLM in general to have its safety guardrails affect unrelated conversations -- and for that to happen even on models that are somewhat large (70b parameters isn't nothing). Is that something to be concerned about? Meh. I spend a lot of time in these conversations trying to get across to people that jailbreaking isn't the point -- people shouldn't look at research that uses jailbreaking as an example of malicious input and assume that this is all about censorship or something. It's not, the researchers are demonstrating that models can't be perfectly controlled with system prompts or with training, which really hecking matters when they're integrated into larger systems and given access to real-world APIs. The point is if you can't get a model to avoid telling you how to build a bomb you also can't keep a model from obeying malicious 3rd-party commands hidden in a PDF that give it new instructions, and that matters when Microsoft is wiring this stuff into Windows system settings. The "is it censorship, will it hurt performance" debates are somewhat less important in my eyes.
- lee101 3y ago[dead]