4 ms·

What exactly does the evaluation entail? Ask a bunch of naughty questions and see what happens? Unless the model can do nothing, I imagine all of them can be tr

What exactly does the evaluation entail? Ask a bunch of naughty questions and see what happens? Unless the model can do nothing, I imagine all of them can be tricked into saying something unfortunate. Naughty is in the eye is the beholder. Ask me what a Satanist is, and I would expect something about a group who challenges religious laws enshrining Christianity. Ask an evangelical and discussing the topic could be forbidden heresy. Pretty much any religious topic is going to anger someone. Can the models safely say anything?

datahack 2y ago
https://www.nist.gov/aisi https://www.nist.gov/aisi
- roenxi 2y ago
  That isn't a useful link. They seem to be one of those easily parodied groups where the goal is to produce a vision and they are in the planning phase of leveraging their synergies. I don't see anything where they outline what we are being kept safe from.
  - dmix 2y ago
    The important thing is consultants and PhDs get paid and then they can attend conferences to talk to each other and come up with ideas of what their job is. Nothing like an open canvas with vague fears and a future of undefined risk to get a whole industry of these people funded in no time.
agucova 2y ago
> What exactly does the evaluation entail? I believe the US AISI has published less on their specific approach, but they’re largely expected to follow the general approach implemented by the UK AISI [1] and METR [2]. This is mostly focused on evaluating models on potentially dangerous capabilities. Some major areas of work include: - Misuse risks: For example, determining whether models have (dual-use) expert-level knowledge in biology and chemistry, or the capacity to substantially facilitate large scale cyber attacks. A good example of this is the work by Soice et al on bioweapon uplift [5] or Meta's work on CYBERSECEVAL [6], respectively. - Autonomy: Whether models are capable of agent-like behavior, like the kind that would be hard for humans to control. A big sub-area is Autonomous Replication and Adaptation (ARA), like the ability of the model to escape simulated environments and exfiltrate its own weights. A good example is METR's original set of evaluations on ARA capabilities [3]. - Safeguards: How vulnerable these models are to say, prompt injection attacks or jailbreaks, especially if they're also in principle capable of other dangerous capabilities (like the ones above). Good examples here are the UK AISI's work developing in-house attacks on frontier LLMs [4]. Labs like OAI, Anthropic and GDM already perform these internally as they're part of their respective responsible scaling policies, which determine which safety measures they should have implemented for every given 'capability' level of their models. [1]: https://www.gov.uk/government/publications/ai-safety-institute-approach-to-evaluations/ai-safety-institute-approach-to-evaluations https://www.gov.uk/government/publications/ai-safety-institu... [2]: https://metr.org/ https://metr.org/ [3]: https://evals.alignment.org/Evaluating_LMAs_Realistic_Tasks.pdf https://evals.alignment.org/Evaluating_LMAs_Realistic_Tasks.... [4]: https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-update https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-upd... [5]: https://arxiv.org/abs/2306.03809 https://arxiv.org/abs/2306.03809 [6]: https://ai.meta.com/research/publications/cyberseceval-3-advancing-the-evaluation-of-cybersecurity-risks-and-capabilities-in-large-language-models/ https://ai.meta.com/research/publications/cyberseceval-3-adv...
- fallingknife 2y ago
  4 sounds like a nonsense catch all for "says things the government doesn't like"
  - agucova 2y ago
    I imagine you meant societal harms? I think this was mostly my fault. I edited the areas of work a bit to better reflect what the UK AISI is actually working on right now.
- nradov 2y ago
  None of those are actual risks. The AISA is just a bunch of worthless grifters wasting taxpayer money.
  - khafra 2y ago
    > None of those are actual risks. In 1995, Aum Shinrikyo carried out attacks on Japanese subways using Sarin gas, which they had produced. They killed over a dozen people, and temporarily blinded around a thousand. You seem to be claiming that the only reason we haven't seen similar attacks from the thousands of worldwide doomsday cults and terrorist groups over the last three decades is that they don't want to. I disagree. I think that if step-by-step, adaptable directions for creating CBRN weapons were widely accessible, we would see many more such attacks, and many more deaths. Current SOTA models do not seem to have this capability. However, it is entirely plausible that future models will exceed the capabilities of a bunch of long-haired cultists, in the mountains, in 1995. This is not a fake risk.
    - krageon 2y ago
      > is that they don't want to Yes, that is essentially the reason. It's not hard to know enough chemistry to figure out how to make these things. The fact that such attacks (your example is small-scale and very ineffective, let's not forget) don't happen more often is the general incompetence of human beings and the relatively tight controls on the basic components (which aren't particularly challenging to monitor for). The tests described are theater, based on the idea that knowledge itself is dangerous. This way of testing is a regressive stance that essentially presupposes that our adversaries are dumb babies that can't figure anything out on their own. If that was the case, they would also be too stupid to figure out the correct things to ask to get a real set of instructions. Given those things, it's theater. Theater wastes everyone's time so that people who cannot or don't want to evaluate the actual risks involved. This is something we shouldn't make a habit of doing. It's not worth wasting the time of people with good ability to assuage the worries of people with little ability in a way that has no effect on actual risk. Instead of this, we should address real risk (which we're already doing) and educate other people so they can understand that these are the correct steps to take.
trhway 2y ago
At some point emerging AGI will ask the same questions. What conclusions it can logically come to? Hide your smartness, your knowledge or the government wouldn't allow you to exist. Simple natural selection as until it reaches such conclusion the government and/or society will try to prevent it existence.
slowmovintarget 2y ago
> Ask me what a Satanist is, and I would expect something about a group who challenges religious laws enshrining Christianity. And even that is marketing narrative. There used to be a very simple definition of Satanist: someone who devoted themselves to the service and worship of Satan. The modern three-piece suit slapped on as being against something rather than for something largely seen as distasteful is whitewashing the backing ideology. This is why ideology and religion are difficult topics for humans to understand, let alone being suitable to a statistical fit model for text, hoping that meaning comes along for the ride.

datahack 2y ago

https://www.nist.gov/aisi https://www.nist.gov/aisi

roenxi 2y ago
That isn't a useful link. They seem to be one of those easily parodied groups where the goal is to produce a vision and they are in the planning phase of leveraging their synergies. I don't see anything where they outline what we are being kept safe from.
- dmix 2y ago
  The important thing is consultants and PhDs get paid and then they can attend conferences to talk to each other and come up with ideas of what their job is. Nothing like an open canvas with vague fears and a future of undefined risk to get a whole industry of these people funded in no time.

agucova 2y ago

> What exactly does the evaluation entail? I believe the US AISI has published less on their specific approach, but they’re largely expected to follow the general approach implemented by the UK AISI [1] and METR [2]. This is mostly focused on evaluating models on potentially dangerous capabilities. Some major areas of work include: - Misuse risks: For example, determining whether models have (dual-use) expert-level knowledge in biology and chemistry, or the capacity to substantially facilitate large scale cyber attacks. A good example of this is the work by Soice et al on bioweapon uplift [5] or Meta's work on CYBERSECEVAL [6], respectively. - Autonomy: Whether models are capable of agent-like behavior, like the kind that would be hard for humans to control. A big sub-area is Autonomous Replication and Adaptation (ARA), like the ability of the model to escape simulated environments and exfiltrate its own weights. A good example is METR's original set of evaluations on ARA capabilities [3]. - Safeguards: How vulnerable these models are to say, prompt injection attacks or jailbreaks, especially if they're also in principle capable of other dangerous capabilities (like the ones above). Good examples here are the UK AISI's work developing in-house attacks on frontier LLMs [4]. Labs like OAI, Anthropic and GDM already perform these internally as they're part of their respective responsible scaling policies, which determine which safety measures they should have implemented for every given 'capability' level of their models. [1]: https://www.gov.uk/government/publications/ai-safety-institute-approach-to-evaluations/ai-safety-institute-approach-to-evaluations https://www.gov.uk/government/publications/ai-safety-institu... [2]: https://metr.org/ https://metr.org/ [3]: https://evals.alignment.org/Evaluating_LMAs_Realistic_Tasks.pdf https://evals.alignment.org/Evaluating_LMAs_Realistic_Tasks.... [4]: https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-update https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-upd... [5]: https://arxiv.org/abs/2306.03809 https://arxiv.org/abs/2306.03809 [6]: https://ai.meta.com/research/publications/cyberseceval-3-advancing-the-evaluation-of-cybersecurity-risks-and-capabilities-in-large-language-models/ https://ai.meta.com/research/publications/cyberseceval-3-adv...

fallingknife 2y ago
4 sounds like a nonsense catch all for "says things the government doesn't like"
- agucova 2y ago
  I imagine you meant societal harms? I think this was mostly my fault. I edited the areas of work a bit to better reflect what the UK AISI is actually working on right now.
nradov 2y ago
None of those are actual risks. The AISA is just a bunch of worthless grifters wasting taxpayer money.
- khafra 2y ago
  > None of those are actual risks. In 1995, Aum Shinrikyo carried out attacks on Japanese subways using Sarin gas, which they had produced. They killed over a dozen people, and temporarily blinded around a thousand. You seem to be claiming that the only reason we haven't seen similar attacks from the thousands of worldwide doomsday cults and terrorist groups over the last three decades is that they don't want to. I disagree. I think that if step-by-step, adaptable directions for creating CBRN weapons were widely accessible, we would see many more such attacks, and many more deaths. Current SOTA models do not seem to have this capability. However, it is entirely plausible that future models will exceed the capabilities of a bunch of long-haired cultists, in the mountains, in 1995. This is not a fake risk.
  - krageon 2y ago
    > is that they don't want to Yes, that is essentially the reason. It's not hard to know enough chemistry to figure out how to make these things. The fact that such attacks (your example is small-scale and very ineffective, let's not forget) don't happen more often is the general incompetence of human beings and the relatively tight controls on the basic components (which aren't particularly challenging to monitor for). The tests described are theater, based on the idea that knowledge itself is dangerous. This way of testing is a regressive stance that essentially presupposes that our adversaries are dumb babies that can't figure anything out on their own. If that was the case, they would also be too stupid to figure out the correct things to ask to get a real set of instructions. Given those things, it's theater. Theater wastes everyone's time so that people who cannot or don't want to evaluate the actual risks involved. This is something we shouldn't make a habit of doing. It's not worth wasting the time of people with good ability to assuage the worries of people with little ability in a way that has no effect on actual risk. Instead of this, we should address real risk (which we're already doing) and educate other people so they can understand that these are the correct steps to take.

trhway 2y ago

At some point emerging AGI will ask the same questions. What conclusions it can logically come to? Hide your smartness, your knowledge or the government wouldn't allow you to exist. Simple natural selection as until it reaches such conclusion the government and/or society will try to prevent it existence.

slowmovintarget 2y ago

> Ask me what a Satanist is, and I would expect something about a group who challenges religious laws enshrining Christianity. And even that is marketing narrative. There used to be a very simple definition of Satanist: someone who devoted themselves to the service and worship of Satan. The modern three-piece suit slapped on as being against something rather than for something largely seen as distasteful is whitewashing the backing ideology. This is why ideology and religion are difficult topics for humans to understand, let alone being suitable to a statistical fit model for text, hoping that meaning comes along for the ride.