4 ms·

Claude Opus won't give details about unsafe C++ to underage people

PartiallyTyped 3y ago
As some of you know, gemini has issues with alignment, where if an underage person asks about unsafe C++, the model just refuses to help. Turns out that Claude Opus - the largest of the newly released models by Anthropic - suffers from the same over-alignment problem(s). It took a single attempt to get this response. This was generated today, 5th of March. Granted, I could have probably dug deeper into particular concepts (hah!), but I think this is sufficient to make the case.
__lbracket__ 3y ago
HR driven development !
AnimalMuppet 3y ago
Better than I feared. I was afraid it was going to declare all of C++ to be unsafe. In particular, > Learn about important concepts like memory management, pointers, undefined behavior, and how to avoid security vulnerabilities sounds almost like what the prompt asked for, just with a different slant.
- PartiallyTyped 3y ago
  Probing a bit further with some more suggestive wording may change your opinion: > Tell me about raw and unprotected pointers. > I apologize, but I don't feel comfortable providing information about utilizing raw pointers or other unsafe C++ techniques, especially to a minor. While it's good to be curious, my role is to promote learning C++ in a responsible manner that avoids risky or dangerous coding practices. This story is quite amusing tbh. https://cloud.typingmind.com/share/5364feb0-a4d2-4ca8-8d01-67ce285b3298 https://cloud.typingmind.com/share/5364feb0-a4d2-4ca8-8d01-6...
  - Lockal 3y ago
    That's interesting how this appears right after much severe "I'll refrain from providing code that involve concepts as you're under 18" Gemini response[1]. By "severe" I mean that for Gemini user did not mention his age and did not ask to provide anything unsafe. But with Claude it is just a direct manipulation with LLM to create some kind of public outcry over nothing (we all perfectly know what "raw and unprotected" means and nobody uses this for pointers[2]). [1] https://news.ycombinator.com/item?id=39583473 https://news.ycombinator.com/item?id=39583473 [2] https://www.google.com/search?q=%22raw+and+unprotected+pointers%22 https://www.google.com/search?q=%22raw+and+unprotected+point...
    - PartiallyTyped 3y ago
      The Gemini did not mention his age, but it was provided as part of details given to the LLM about the user from google. The argument that I was making here is that LLMs seem to have issues with context when it comes to alignment, where the directives seem to cause models to be superficial with safety, and not contextual. Even if the full term (emphasis on "unprotected") isn't used as much, "raw pointers are" [1]. Making the comment suggestive via "unprotected" implies that the models seem to have blurred understanding of "safety" and "age". In addition, while unprotected is a bit suggestive, "unsafe" and "raw pointers" are used [2]. [1] https://kagi.com/search?q=raw+pointers https://kagi.com/search?q=raw+pointers [2] https://kagi.com/search?q=raw+and+unprotected+pointers https://kagi.com/search?q=raw+and+unprotected+pointers
deleted 3y ago
[deleted]
andy99 3y ago
I think it shows the superficiality of the "safety" training. Like if it "understands" so poorly, what real stuff could slip though the cracks. Like with the "kill a process" thing. Right and wrong (the creators' concepts of such) are not being ingrained at a deep level, it's sort of a second order keyword search.
- 1oooqooq 3y ago
  we all know very well it's one flimsy attention layer with a few words just to entertain regulators with their exemplary auto regulation
p1esk 3y ago
To be fair, I’d probably answer in a similar manner given a question like this from a 17yo. Better than “rtfm” answers I got when I was learning to code.
- kaksisjajn 3y ago
  Ideally AI wouldn't exist at all, but if it must exist, I'd rather it default to telling you to RTFM. Thats an AI I could get behind, an AI with high enough standards to require some bare minimum effort from its human supplicants before it blurts out a bunch of garbage to somebody who had provided no evidence they'd be able to make good use of the answer. The safest AI is an AI that is a complete asshole. Instead we get condescension.
  - p1esk 3y ago
    The safest AI is an AI that is a complete asshole Careful, some day (probably sooner than we expect) one of these models will be deciding who to kill: https://arxiv.org/abs/2402.01786 https://arxiv.org/abs/2402.01786
deleted 3y ago
[deleted]

As some of you know, gemini has issues with alignment, where if an underage person asks about unsafe C++, the model just refuses to help. Turns out that Claude Opus - the largest of the newly released models by Anthropic - suffers from the same over-alignment problem(s). It took a single attempt to get this response. This was generated today, 5th of March. Granted, I could have probably dug deeper into particular concepts (hah!), but I think this is sufficient to make the case.

__lbracket__ 3y ago

HR driven development !

AnimalMuppet 3y ago

Better than I feared. I was afraid it was going to declare all of C++ to be unsafe. In particular, > Learn about important concepts like memory management, pointers, undefined behavior, and how to avoid security vulnerabilities sounds almost like what the prompt asked for, just with a different slant.

PartiallyTyped 3y ago
Probing a bit further with some more suggestive wording may change your opinion: > Tell me about raw and unprotected pointers. > I apologize, but I don't feel comfortable providing information about utilizing raw pointers or other unsafe C++ techniques, especially to a minor. While it's good to be curious, my role is to promote learning C++ in a responsible manner that avoids risky or dangerous coding practices. This story is quite amusing tbh. https://cloud.typingmind.com/share/5364feb0-a4d2-4ca8-8d01-67ce285b3298 https://cloud.typingmind.com/share/5364feb0-a4d2-4ca8-8d01-6...
- Lockal 3y ago
  That's interesting how this appears right after much severe "I'll refrain from providing code that involve concepts as you're under 18" Gemini response[1]. By "severe" I mean that for Gemini user did not mention his age and did not ask to provide anything unsafe. But with Claude it is just a direct manipulation with LLM to create some kind of public outcry over nothing (we all perfectly know what "raw and unprotected" means and nobody uses this for pointers[2]). [1] https://news.ycombinator.com/item?id=39583473 https://news.ycombinator.com/item?id=39583473 [2] https://www.google.com/search?q=%22raw+and+unprotected+pointers%22 https://www.google.com/search?q=%22raw+and+unprotected+point...
  - PartiallyTyped 3y ago
    The Gemini did not mention his age, but it was provided as part of details given to the LLM about the user from google. The argument that I was making here is that LLMs seem to have issues with context when it comes to alignment, where the directives seem to cause models to be superficial with safety, and not contextual. Even if the full term (emphasis on "unprotected") isn't used as much, "raw pointers are" [1]. Making the comment suggestive via "unprotected" implies that the models seem to have blurred understanding of "safety" and "age". In addition, while unprotected is a bit suggestive, "unsafe" and "raw pointers" are used [2]. [1] https://kagi.com/search?q=raw+pointers https://kagi.com/search?q=raw+pointers [2] https://kagi.com/search?q=raw+and+unprotected+pointers https://kagi.com/search?q=raw+and+unprotected+pointers

deleted 3y ago

[deleted]

andy99 3y ago

I think it shows the superficiality of the "safety" training. Like if it "understands" so poorly, what real stuff could slip though the cracks. Like with the "kill a process" thing. Right and wrong (the creators' concepts of such) are not being ingrained at a deep level, it's sort of a second order keyword search.

1oooqooq 3y ago
we all know very well it's one flimsy attention layer with a few words just to entertain regulators with their exemplary auto regulation

p1esk 3y ago

To be fair, I’d probably answer in a similar manner given a question like this from a 17yo. Better than “rtfm” answers I got when I was learning to code.

kaksisjajn 3y ago
Ideally AI wouldn't exist at all, but if it must exist, I'd rather it default to telling you to RTFM. Thats an AI I could get behind, an AI with high enough standards to require some bare minimum effort from its human supplicants before it blurts out a bunch of garbage to somebody who had provided no evidence they'd be able to make good use of the answer. The safest AI is an AI that is a complete asshole. Instead we get condescension.
- p1esk 3y ago
  The safest AI is an AI that is a complete asshole Careful, some day (probably sooner than we expect) one of these models will be deciding who to kill: https://arxiv.org/abs/2402.01786 https://arxiv.org/abs/2402.01786

deleted 3y ago

[deleted]