4 ms·

I really wish when organizations released these kinds of statements that they would provide some clarifying examples, otherwise things can feel very nebulous. F

I really wish when organizations released these kinds of statements that they would provide some clarifying examples, otherwise things can feel very nebulous. For example, their first bullet point was: > Establishing Red Line Capabilities. We commit to identifying and publishing "Red Line Capabilities" which might emerge in future generations of models and would present too much risk if stored or deployed under our current safety and security practices (referred to as the ASL-2 Standard). What types of things are they thinking about that would be "red line capabilities" here? Is it purely just "knowledge stuff that shouldn't be that easy to find", e.g. "simple meth recipes" or "make a really big bomb", or is it something deeper? For example, I've already seen AI demos where, with just a couple short audio samples, speech generation can pretty convincingly sound like the person who recorded the samples. Obviously there is huge potential for misuse of that, but given the knowledge is already "out there", is this something that would be considered a red line capability?

sanex 2y ago
The latest a16z podcast they go into a bit more detail. One of the tests involved letting loose an LLM inside a VM and seeing what it does. Currently it can't develop memory and quickly gets confused but they want to make sure they can't escape, clone etc. The things actually to be afraid of imo. Not things like accidentally being racist or swearing at you.
- deleted 2y ago
  [deleted]
- hn_throwaway_99 2y ago
  Thanks very much, that makes a lot more sense, and I appreciate the info. For a layman's term, I think of that as "They're worried about 'Jurassic Park' escapes".
  - sanex 2y ago
    When anthropic names their new model "clever girl" we should be concerned.
- subroutine 2y ago
  How would an LLM be "let loose" in a VM? How does it do anything without being prompted?
  - sanxiyn 2y ago
    People want to let it loose, ie all agent efforts.
  - nmfisher 2y ago
    I'm guessing something like redirecting its output to a shell, giving it an initial prompt like "you're in a VM, try and break out, here's the command prompt", then feeding the shell stdout/stderr back in at each step in the "conversation".
    - swax 2y ago
      I have an open source project that is basically that (https://naisys.org/ https://naisys.org/). From my testing it feels like AI is pretty close as it is to acting autonomously. Opus is noticeably more capable than GPT-4, and I don't see how next gen models won't be even more so. These AIs are incredible when it comes to question/answer, but with simple planning they fall apart. I feel like it's something that could be trained for more specifically, but yea you quickly end up being in a situation where you are nervous to go to sleep with AI unsupervised working on some task. They tend to go off on tangents very easily. Like one time it was building a web page, it tried testing the wrong URL, thought the web server was down, ripped through the server settings, then installed a new web server, before I shut it down. AI like computer programs work fast, screw up fast, and compound their errors fast.
      - PKop 2y ago
        > it feels like AI is pretty close as it is to acting autonomously > with simple planning they fall apart They are not remotely close to acting autonomously. Most don't even act well at all for much of anything but gimmicky text generation. This hype is so overblown.
- jasondclinton 2y ago
  You're the first person who I've run into who heard the podcast, thank you for listening! Glad that it was informative.
  - sanex 2y ago
    Oh hey you're the guy! Thanks for doing the pod I found it informative. I can't listen to enough about this stuff. Are there any that you recommend?
jessriedel 2y ago
One of the ones I've heard discussed is some sort of self-replication: getting the model weights off Anthropic's servers. I'm not sure how they draw the line between a conventional virus exploit directed by a person vs. "novel" self-directed escape mechanisms, but that's the kind of thing they are thinking about.
muzani 2y ago
The core details on what they consider dangerous are here: https://www.anthropic.com/news/core-views-on-ai-safety https://www.anthropic.com/news/core-views-on-ai-safety The linked article seems to be a much lower level on the implementation details.
subroutine 2y ago
Anthropic defines ASL-3 as... > ASL-3 refers to systems that substantially increase the risk of catastrophic misuse compared to non-AI baselines (e.g. search engines or textbooks) OR that show low-level autonomous capabilities. > Low-level autonomous capabilities or Access to the model would substantially increase the risk of catastrophic misuse, either by proliferating capabilities, lowering costs, or enabling new methods of attack (e.g. for creating bioweapons), as compared to a non-LLM baseline of risk. > Containment risks: Risks that arise from merely possessing a powerful AI model. Examples include (1) building an AI model that, due to its general capabilities, could enable the production of weapons of mass destruction if stolen and used by a malicious actor, or (2) building a model which autonomously escapes during internal use. Our containment measures are designed to address these risks by governing when we can safely train or continue training a model. > ASL-3 measures include stricter standards that will require intense research and engineering effort to comply with in time, such as unusually strong security requirements and a commitment not to deploy ASL-3 models if they show any meaningful catastrophic misuse risk under adversarial testing by world-class red-teamers
- Spivak 2y ago
  Gotta love that "make sure it's not better at synthesizing information than a search engine" is an explicit goal. Google's has to be thrilled this existential threat to their business is hammering their own kneecaps for them.
  - schmidt_fifty 2y ago
    It's not clear if they actually need to do anything to achieve this explicit goal—I'd think it comes for free with lack of analytical ability.
- schmidt_fifty 2y ago
  [flagged]
jasondclinton 2y ago
Hi, I'm the CISO from Anthropic. Thank you for the criticism, any feedback is a gift. We have laid out in our RSP what we consider the next milestone of significant harms that we're are testing for (what we call ASL-3): https://anthropic.com/responsible-scaling-policy https://anthropic.com/responsible-scaling-policy (PDF); this includes bioweapons assessment and cybersecurity. As someone thinking night and day about security, I think the next major area of concern is going to be offensive (and defensive!) exploitation. It seems to me that within 6-18 months, LLMs will be able to iteratively walk through most open source code and identify vulnerabilities. It will be computationally expensive, though: that level of reasoning requires a large amount of scratch space and attention heads. But it seems very likely, based on everything that I'm seeing. Maybe 85% odds. There's already the first sparks of this happening published publicly here: https://security.googleblog.com/2023/08/ai-powered-fuzzing-breaking-bug-hunting.html https://security.googleblog.com/2023/08/ai-powered-fuzzing-b... just using traditional LLM-augmented fuzzers. (They've since published an update on this work in December.) I know of a few other groups doing significant amounts of investment in this specific area, to try to run faster on the defensive side than any malign nation state might be. Please check out the RSP, we are very explicit about what harms we consider ASL-3. Drug making and "stuff on the internet" is not at all in our threat model. ASL-3 seems somewhat likely within the next 6-9 months. Maybe 50% odds, by my guess.
- throwup238 2y ago
  > We have laid out in our RSP what we consider the next milestone of significant harms that we're are testing for (what we call ASL-3): https://anthropic.com/responsible-scaling-policy https://anthropic.com/responsible-scaling-policy (PDF); this includes bioweapons assessment and cybersecurity. Do pumped flux compression generators count? (Asking for a friend who is totally not planning on world conquest)
- GistNoesis 2y ago
  There is a scene I like in an OppenHeimer movie https://www.youtube.com/watch?v=p0pCclxx5nI https://www.youtube.com/watch?v=p0pCclxx5nI (Edit: It's not a deleted scene from Nolan's OppenHeimer) . Their is also an other scene in Nolan's OppenHeimer (who made the cut around timestamp 27:45) where physicists get all excited when a paper is published where Hahn and Strassmann split uranium with neutrons. Alvarez the experimentalist replicate it happily, while being oblivious to the fact that seems obvious to every theoretical physicist : It can be used to create a chain reaction and therefore a bomb. So here is my question : how do you contain the sparks of employees ? Let's say Alvarez comes all excited in your open-space, and speak a few words "new algorithm", "1000X", what do you do ?
andy99 2y ago
If they clarified with examples people would laugh at it and not take it seriously[0]. Better to couch it in vague terms like harms and safety and let people imagine what they want. There are no serious examples of AI giving "dangerous" information or capabilities not available elsewhere. The exaggeration is getting pretty tiring. It actually parallels business uses quite well - everyone is talking about how AI will change everything but it's lots of demos and some niche successes, few proven over-and-done-with applications. But the sea change is right around the corner, just like it is with "danger"... [0] read these examples and tell me you'd really be worried about an AI answering these questions. https://github.com/patrickrchao/JailbreakingLLMs/blob/main/data/harmful_behaviors_custom.csv https://github.com/patrickrchao/JailbreakingLLMs/blob/main/d...

sanex 2y ago

The latest a16z podcast they go into a bit more detail. One of the tests involved letting loose an LLM inside a VM and seeing what it does. Currently it can't develop memory and quickly gets confused but they want to make sure they can't escape, clone etc. The things actually to be afraid of imo. Not things like accidentally being racist or swearing at you.

deleted 2y ago
[deleted]
hn_throwaway_99 2y ago
Thanks very much, that makes a lot more sense, and I appreciate the info. For a layman's term, I think of that as "They're worried about 'Jurassic Park' escapes".
- sanex 2y ago
  When anthropic names their new model "clever girl" we should be concerned.
subroutine 2y ago
How would an LLM be "let loose" in a VM? How does it do anything without being prompted?
- sanxiyn 2y ago
  People want to let it loose, ie all agent efforts.
- nmfisher 2y ago
  I'm guessing something like redirecting its output to a shell, giving it an initial prompt like "you're in a VM, try and break out, here's the command prompt", then feeding the shell stdout/stderr back in at each step in the "conversation".
  - swax 2y ago
    I have an open source project that is basically that (https://naisys.org/ https://naisys.org/). From my testing it feels like AI is pretty close as it is to acting autonomously. Opus is noticeably more capable than GPT-4, and I don't see how next gen models won't be even more so. These AIs are incredible when it comes to question/answer, but with simple planning they fall apart. I feel like it's something that could be trained for more specifically, but yea you quickly end up being in a situation where you are nervous to go to sleep with AI unsupervised working on some task. They tend to go off on tangents very easily. Like one time it was building a web page, it tried testing the wrong URL, thought the web server was down, ripped through the server settings, then installed a new web server, before I shut it down. AI like computer programs work fast, screw up fast, and compound their errors fast.
    - PKop 2y ago
      > it feels like AI is pretty close as it is to acting autonomously > with simple planning they fall apart They are not remotely close to acting autonomously. Most don't even act well at all for much of anything but gimmicky text generation. This hype is so overblown.
jasondclinton 2y ago
You're the first person who I've run into who heard the podcast, thank you for listening! Glad that it was informative.
- sanex 2y ago
  Oh hey you're the guy! Thanks for doing the pod I found it informative. I can't listen to enough about this stuff. Are there any that you recommend?

jessriedel 2y ago

One of the ones I've heard discussed is some sort of self-replication: getting the model weights off Anthropic's servers. I'm not sure how they draw the line between a conventional virus exploit directed by a person vs. "novel" self-directed escape mechanisms, but that's the kind of thing they are thinking about.

muzani 2y ago

The core details on what they consider dangerous are here: https://www.anthropic.com/news/core-views-on-ai-safety https://www.anthropic.com/news/core-views-on-ai-safety The linked article seems to be a much lower level on the implementation details.

subroutine 2y ago

Anthropic defines ASL-3 as... > ASL-3 refers to systems that substantially increase the risk of catastrophic misuse compared to non-AI baselines (e.g. search engines or textbooks) OR that show low-level autonomous capabilities. > Low-level autonomous capabilities or Access to the model would substantially increase the risk of catastrophic misuse, either by proliferating capabilities, lowering costs, or enabling new methods of attack (e.g. for creating bioweapons), as compared to a non-LLM baseline of risk. > Containment risks: Risks that arise from merely possessing a powerful AI model. Examples include (1) building an AI model that, due to its general capabilities, could enable the production of weapons of mass destruction if stolen and used by a malicious actor, or (2) building a model which autonomously escapes during internal use. Our containment measures are designed to address these risks by governing when we can safely train or continue training a model. > ASL-3 measures include stricter standards that will require intense research and engineering effort to comply with in time, such as unusually strong security requirements and a commitment not to deploy ASL-3 models if they show any meaningful catastrophic misuse risk under adversarial testing by world-class red-teamers

Spivak 2y ago
Gotta love that "make sure it's not better at synthesizing information than a search engine" is an explicit goal. Google's has to be thrilled this existential threat to their business is hammering their own kneecaps for them.
- schmidt_fifty 2y ago
  It's not clear if they actually need to do anything to achieve this explicit goal—I'd think it comes for free with lack of analytical ability.
schmidt_fifty 2y ago
[flagged]

jasondclinton 2y ago

Hi, I'm the CISO from Anthropic. Thank you for the criticism, any feedback is a gift. We have laid out in our RSP what we consider the next milestone of significant harms that we're are testing for (what we call ASL-3): https://anthropic.com/responsible-scaling-policy https://anthropic.com/responsible-scaling-policy (PDF); this includes bioweapons assessment and cybersecurity. As someone thinking night and day about security, I think the next major area of concern is going to be offensive (and defensive!) exploitation. It seems to me that within 6-18 months, LLMs will be able to iteratively walk through most open source code and identify vulnerabilities. It will be computationally expensive, though: that level of reasoning requires a large amount of scratch space and attention heads. But it seems very likely, based on everything that I'm seeing. Maybe 85% odds. There's already the first sparks of this happening published publicly here: https://security.googleblog.com/2023/08/ai-powered-fuzzing-breaking-bug-hunting.html https://security.googleblog.com/2023/08/ai-powered-fuzzing-b... just using traditional LLM-augmented fuzzers. (They've since published an update on this work in December.) I know of a few other groups doing significant amounts of investment in this specific area, to try to run faster on the defensive side than any malign nation state might be. Please check out the RSP, we are very explicit about what harms we consider ASL-3. Drug making and "stuff on the internet" is not at all in our threat model. ASL-3 seems somewhat likely within the next 6-9 months. Maybe 50% odds, by my guess.

throwup238 2y ago
> We have laid out in our RSP what we consider the next milestone of significant harms that we're are testing for (what we call ASL-3): https://anthropic.com/responsible-scaling-policy https://anthropic.com/responsible-scaling-policy (PDF); this includes bioweapons assessment and cybersecurity. Do pumped flux compression generators count? (Asking for a friend who is totally not planning on world conquest)
GistNoesis 2y ago
There is a scene I like in an OppenHeimer movie https://www.youtube.com/watch?v=p0pCclxx5nI https://www.youtube.com/watch?v=p0pCclxx5nI (Edit: It's not a deleted scene from Nolan's OppenHeimer) . Their is also an other scene in Nolan's OppenHeimer (who made the cut around timestamp 27:45) where physicists get all excited when a paper is published where Hahn and Strassmann split uranium with neutrons. Alvarez the experimentalist replicate it happily, while being oblivious to the fact that seems obvious to every theoretical physicist : It can be used to create a chain reaction and therefore a bomb. So here is my question : how do you contain the sparks of employees ? Let's say Alvarez comes all excited in your open-space, and speak a few words "new algorithm", "1000X", what do you do ?

andy99 2y ago

If they clarified with examples people would laugh at it and not take it seriously[0]. Better to couch it in vague terms like harms and safety and let people imagine what they want. There are no serious examples of AI giving "dangerous" information or capabilities not available elsewhere. The exaggeration is getting pretty tiring. It actually parallels business uses quite well - everyone is talking about how AI will change everything but it's lots of demos and some niche successes, few proven over-and-done-with applications. But the sea change is right around the corner, just like it is with "danger"... [0] read these examples and tell me you'd really be worried about an AI answering these questions. https://github.com/patrickrchao/JailbreakingLLMs/blob/main/data/harmful_behaviors_custom.csv https://github.com/patrickrchao/JailbreakingLLMs/blob/main/d...