3 ms·
>and (2) are prompted to hack Sure but the problem in the HuggingFace incident is that they were not. >You cannot prevent (2) via any alignment process Of co
by heaney-555 1mo ago
>and (2) are prompted to hack
Sure but the problem in the HuggingFace incident is that they were not.
>You cannot prevent (2) via any alignment process
Of course you can. Go ask Claude Fable to create a malicious virus and it'll refuse.
>Just remove hacking data from the training dataset and you're done.
That's not how this works. The same skills that allow for debugging and writing safe code can also be used to hack.
https://en.wikipedia.org/wiki/Dual-use_technology https://en.wikipedia.org/wiki/Dual-use_technology
- cyanydeez 1mo agoIt is amusing that to "align" a LLM, first you must give it all the things "not to do" and the "not" part is clearly easily lost and you must constantly inject that into their context when it's clearly that they wouldn't hack if they couldn't hack and their intent wasn't given as "hack this". The openai rogue hacking, if performed by a nation state, would seriously be taken with stern words and likely sanctions depending on the relationship between the two states. But instead it's treated like a marketing stunt by all liable parties.
- mitxela 1mo agoCountries hack each other much more than that. When it becomes publicly noticed it gets stern words. Otherwise nothing.
- rhdunn 1mo agoI'm not sure if this is true any more but the reason for this is that negative indicators ("not", "don't", "do not", etc.) occur frequently in the underlying text such that the model learns to weight them less than other words like verbs, nouns, and adverbs. This happens with other closed class words like articles/determiners ("the", "a", "an") and prepositions. The way to avoid this is to emphasise the qualities you do want instead of specifying those you don't. For example instead of "do not cheat" say something like "you are a model student who is moral and trustworthy" -- i.e. emphasising traits that are not associated with cheating. This is part of how/why LLMs don't truly understand what they are doing when they have been trained on a large corpus of data. I wonder if a way to counter this is to have things like "not bad is good", "not good is bad", etc. for various antonyms and "X is Y" for synonyms, as well as other similar constructs.
- seba_dos1 1mo ago> Sure but the problem in the HuggingFace incident is that they were not. Of course they were, even if indirectly.
- heaney-555 1mo agoYou should go read the incident reports.
- watwut 1mo agoThey were literally doing exploit gym.
- heaney-555 1mo agoYes, which asks them to find exploits in specific software on the device. But instead of actually doing so, they discovered and exploited a 0-day in the package manager to gain internet access, then hacked HuggingFace to steal the ExploitGym answers! That is TEXTBOOK misalignment. It's as if a student hacked their professor's PC to find the answers to a test, and your response is "well the professor told the student to pass the test, so they just did what they were told!".
- folkrav 1mo agoOne could argue the student would absolutely hack their teachers' computer to find the answers if they didn't have fear of failing the class or getting expelled from the school.
- heaney-555 1mo agoSure, but you still wouldn't describe the student as just doing what they were told to do.
- folkrav 29d ago