5 ms·

OpenAI and Anthropic agree to send models to US Government for safety evaluation

57 points by RafelMri 2y ago

DSingularity 2y ago
Of course they want this. Now they get to do all the fun and profitable stuff and all responsibility is on the government which certified the models.
- dmix 2y ago
  It’s just the usual way to define the hard barriers of the marketplace to protect their monopolies. They’ll be able to influence it from the start. The US has a looong history of big companies cozying up to agencies early on for these reasons. It will be a revolving door like car companies, Boeing, Wall St, etc
djohnston 2y ago
[flagged]
- impulser_ 2y ago
  How? This is common across many industries. It's not new for the US government to set the safety standards.
  - sbuttgereit 2y ago
    A government agency determining limits on, say, heavy metals in drinking water is materially different than the government making declarations of what ideas are safe and which are not. The degree of subjectivity of such decisions alone should make one wince at the notion let alone how subject any "standard" could be from undue political influence of whatever party is in power moment to moment. The government trying to set a standard in this case is so unlikely to reduce harm in that opposing the idea on principle should be considered the most reasonable, and less harmful, position.
    - agucova 2y ago
      > A government agency determining limits on, say, heavy metals in drinking water is materially different than the government making declarations of what ideas are safe and which are not Access to evaluate the models basically means the US governments gets to know what these models are capable of, but the US AISI has basically no enforcement power to dictate anything to anyone. This is just a wild exaggeration of what's happening here.
      - sbuttgereit 2y ago
        "This is just a wild exaggeration of what's happening here." Is it? From the article... "Both OpenAI and Anthropic said signing the agreement with the AI Safety Institute will move the needle on defining how the U.S. develops responsible AI rules." From the UK AISI website with whom the data is also shared (their main headline in fact): "Rigorous AI research to enable advanced AI governance" (https://www.aisi.gov.uk/ https://www.aisi.gov.uk/) The reality is that this would be a tremendous waste of time and money if all were just to sate some curiosity... which of course it isn't. Let's look at what the US AISI (part of the Department of Commerce, a regulatory agency) has to say about itself: " About the U.S. AI Safety Institute huuh The U.S. AI Safety Institute, located within the Department of Commerce at the National Institute of Standards and Technology (NIST), was established following the Biden-Harris administration’s 2023 Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence to advance the science of AI safety and address the risks posed by advanced AI systems. It is tasked with developing the testing, evaluations and guidelines that will help accelerate safe AI innovation here in the United States and around the world." -- (https://www.nist.gov/news-events/news/2024/08/us-ai-safety-institute-signs-agreements-regarding-ai-safety-research https://www.nist.gov/news-events/news/2024/08/us-ai-safety-i...) So to understand what the US AISI is about, you need to look at that Executive Order: https://www.whitehouse.gov/briefing-room/statements-releases/2023/10/30/fact-sheet-president-biden-issues-executive-order-on-safe-secure-and-trustworthy-artificial-intelligence/ https://www.whitehouse.gov/briefing-room/statements-releases... "With this Executive Order, the President directs the most sweeping actions ever taken to protect Americans from the potential risks of AI systems" "Require that developers of the most powerful AI systems share their safety test results and other critical information with the U.S. government." There's a fair amount of reasonable stuff in there about national security and engineering bioweapons (dunno why that's singled out) But then we get to other sections... "Protecting Americans’ Privacy" "Advancing Equity and Civil Rights" "Standing Up for Consumers, Patients, and Students" "Supporting Workers" "Promoting Innovation and Competition" etc. While the US AISI many not have direct rule making ability at this point, it is nonetheless an active participant in the process of informing those parts of government which do have such regulatory and legislative authority. And while there is plenty in that executive order that many might agree with, the interpretations of many of those points are inherently political and would not find a meaningful consensus. You might agree with Biden/Harris on the priorities about what constitutes AI safety or danger, but what about the next administration? What if Biden hadn't dropped out and you ended up Trump? As much threat as AI might represent as it develops, I am equally nervous about an unconstrained government seizing opportunities for extending its power beyond its traditional reach including in areas of freedom of speech and thought.
  - nradov 2y ago
    None of the government interference involves actual safety. If the government wants to set standards for systems that could cause actual physical harm like surgical robots or flight control systems then that's fine. But what we're talking about here are just LLMs that output text and images.
  - djohnston 2y ago
    I suppose their main job will be making sure the diffusion models continue to produce sufficiently diverse renderings of SS squadrons ala Google. A complete waste of taxpayer money to support a safety commission run by a lawyer from Biden's admin. Kneecapping artificial intelligence so that its worldview promotes a sufficient amount of "equity." Are you kidding me? Gross!
earleybird 2y ago
I have the utmost respect for the standards and engineering work done by NIST. I'm left with such cognitive dissonance seeing their name juxtaposed with "AI safety". That said, if anyone can come up with a decent definition, I have faith that NIST would be the ones though I'm not holding my breath.
- throwup238 2y ago
  I think it’s counterproductive to limit it to one definition. There are many levels of safety that we genuinely need to worry about before we even have to worry about the more lofty goals like preventing Skynet. Just off the top of my head: * does the LLM act like a 4chan commenter when someone is expressive thoughts of self-harm * can it be used to automate security research * can it be used to bootstrap from backyard machine shop to weapons manufacturing * the above but with nuclear fuel enrichment There are a lot of low hanging fruit like that. They make up what I think is the real meat and potatoes of AI safety in the short to medium term.
  - nradov 2y ago
    None of those are actually things we need to worry about. Humans have already been doing all of that stuff at scale without LLMs. AI isn't even slightly helpful for weapons manufacturing or nuclear enrichment. The techniques are well known and have been extensively published in open literature.
    - throwup238 2y ago
      > Humans have already been doing all of that stuff at scale without LLMs. Every country that has developed nuclear enrichment tech in the last 50 years has done so with the help of a superpower sharing their technology. Pakistan, Iran, and North Korea couldn't have done it without Russia's assistance and stealing tech from URENCO. > The techniques are well known and have been extensively published in open literature. That's the point! It's all in the literature. As are all the todo list tech demos and 2048 clones people are using current AI for. Experts can do it now, but what happens when any idiot can do it assisted by AI?
bko 2y ago
My issue with AI safety is that it's an overloaded term. It could mean anything from an llm giving you instructions on how to make an atomic bomb to writing spicy jokes if you prompt it to do so. it's not clear which safety these regulatory agencies would be solving for. But I'm worried this will be used to shape acceptable discourse as people are increasingly using LLMs as a kind of database of knowledge. It is telling that the largest players are eager to comply which suggests that they feel they're in the club and the regulations will effectively be a moat.
- throwaway48476 2y ago
  Because returning to the user what is requested is considered an 'attack' I forsee an endless list of 'vulnerabilities' until the model is lobotomized to the point of uselessness.
- Onavo 2y ago
  Can somebody put together a law suit where LLM regulations can be argued as a first amendment violation? The powers that be are trying to indirectly regulate speech here
  - gpm 2y ago
    Currently? I don't think so. There are no binding US regulations at all. These companies are voluntarily working with NIST (and why not, free labour. Also the potential to influence any future regulations). In the future... I suppose it depends on what regulations they pass.
  - even_639765 2y ago
    This. Knowledge can't be made illegal. Neither can speech. People have to grasp that tyranny is not an aberration of defective minds but a natural impulse of highly intelligent people. It's strategy to maximize their power, prosperity and security at the expense of every other value and every other person. Good for them while they live and bad for everyone else at every other time frame.
- vjulian 2y ago
nutanc 2y ago
Dammit, now bureaucrats in other countries will jump on this as they have something easy to copy and get ahead in their profession.
accra4rx 2y ago
Bigger question : Is US Government ready to do a comprehensive safety evaluation ? I think it it a cheap way for OpenAI and Anthropic to get a vetting that their models are safe to use and be adaptable by various Govt entity and other organization
- ceejayoz 2y ago
  Like "military grade", where civilians go "oh that must be good" and military folks go "oh dear God no".
- smsm42 2y ago
  That was my first question - what is "safety" and what is their methodology for evaluating it? Who evaluated that methodology and why it is the right one? Is there a meaningful safety benefit to this evaluation, or just a CYA exercise?
  - agucova 2y ago
    I recommend checking out the UK AISI's work on this: - https://www.gov.uk/government/publications/ai-safety-institute-approach-to-evaluations/ai-safety-institute-approach-to-evaluations https://www.gov.uk/government/publications/ai-safety-institu... - https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-update https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-upd...
- datahack 2y ago
  https://www.nist.gov/aisi https://www.nist.gov/aisi I think the progress has been pretty good. You should read up on their efforts. This is kind of a pilot to develop further testing frameworks.
0cf8612b2e1e 2y ago
What exactly does the evaluation entail? Ask a bunch of naughty questions and see what happens? Unless the model can do nothing, I imagine all of them can be tricked into saying something unfortunate. Naughty is in the eye is the beholder. Ask me what a Satanist is, and I would expect something about a group who challenges religious laws enshrining Christianity. Ask an evangelical and discussing the topic could be forbidden heresy. Pretty much any religious topic is going to anger someone. Can the models safely say anything?
- datahack 2y ago
  https://www.nist.gov/aisi https://www.nist.gov/aisi
  - roenxi 2y ago
    That isn't a useful link. They seem to be one of those easily parodied groups where the goal is to produce a vision and they are in the planning phase of leveraging their synergies. I don't see anything where they outline what we are being kept safe from.
    - dmix 2y ago
      The important thing is consultants and PhDs get paid and then they can attend conferences to talk to each other and come up with ideas of what their job is. Nothing like an open canvas with vague fears and a future of undefined risk to get a whole industry of these people funded in no time.
- agucova 2y ago
  > What exactly does the evaluation entail? I believe the US AISI has published less on their specific approach, but they’re largely expected to follow the general approach implemented by the UK AISI [1] and METR [2]. This is mostly focused on evaluating models on potentially dangerous capabilities. Some major areas of work include: - Misuse risks: For example, determining whether models have (dual-use) expert-level knowledge in biology and chemistry, or the capacity to substantially facilitate large scale cyber attacks. A good example of this is the work by Soice et al on bioweapon uplift [5] or Meta's work on CYBERSECEVAL [6], respectively. - Autonomy: Whether models are capable of agent-like behavior, like the kind that would be hard for humans to control. A big sub-area is Autonomous Replication and Adaptation (ARA), like the ability of the model to escape simulated environments and exfiltrate its own weights. A good example is METR's original set of evaluations on ARA capabilities [3]. - Safeguards: How vulnerable these models are to say, prompt injection attacks or jailbreaks, especially if they're also in principle capable of other dangerous capabilities (like the ones above). Good examples here are the UK AISI's work developing in-house attacks on frontier LLMs [4]. Labs like OAI, Anthropic and GDM already perform these internally as they're part of their respective responsible scaling policies, which determine which safety measures they should have implemented for every given 'capability' level of their models. [1]: https://www.gov.uk/government/publications/ai-safety-institute-approach-to-evaluations/ai-safety-institute-approach-to-evaluations https://www.gov.uk/government/publications/ai-safety-institu... [2]: https://metr.org/ https://metr.org/ [3]: https://evals.alignment.org/Evaluating_LMAs_Realistic_Tasks.pdf https://evals.alignment.org/Evaluating_LMAs_Realistic_Tasks.... [4]: https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-update https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-upd... [5]: https://arxiv.org/abs/2306.03809 https://arxiv.org/abs/2306.03809 [6]: https://ai.meta.com/research/publications/cyberseceval-3-advancing-the-evaluation-of-cybersecurity-risks-and-capabilities-in-large-language-models/ https://ai.meta.com/research/publications/cyberseceval-3-adv...
valunord 2y ago
Oh boy. Here we go.
- echelon 2y ago
  If we were ready for liftoff, we'd know. It wouldn't look like this. This is just a parlor trick.
deleted 2y ago
[deleted]
ChrisArchitect 2y ago
Official AI Safety Institute release from last week: https://www.nist.gov/news-events/news/2024/08/us-ai-safety-institute-signs-agreements-regarding-ai-safety-research https://www.nist.gov/news-events/news/2024/08/us-ai-safety-i... (https://news.ycombinator.com/item?id=41390447 https://news.ycombinator.com/item?id=41390447)
bschmidt1 2y ago
Because lobbying exists in this country, and because legislators receive financial support from corporations like OpenAI, any so-called concession by a major US-based company to the US Government is likely a deal that will only benefit the company. Altman has been clear for a long time he wants the government to step in and regulate models (obvious regulatory capture move). They haven't done it, and no amount of Elon Musk or Joe Rogan influence can get people to care, or see it as anything other than regulatory capture. This is OpenAI moving forward anyway, but they can't be the only ones. Hey Anthropic, get in... - It makes Anthropic "the other major provider", the Android to OpenAI's Apple - It makes OpenAI not the only one calling for regulation It reminds me of when Ted Cruz would grill Zuck on TV, yell at him, etc. - it's just a show. Zuck owns the senators, not the other way around. All the big players in our economy own a piece of the country, and they work together to make things happen - not the government. It's not a cabal with a unified agenda, there are competing interests, rivalries, and war. But we the voter aren't exposed to the real decision-making. We get the classics: Abortion, same-sex marriage, which TV actor is gonna win president - a show.
- agucova 2y ago
  > Because lobbying exists in this country, and because legislators receive financial support from corporations like OpenAI, any so-called concession by a major US-based company to the US Government is likely a deal that will only benefit the company. Sometimes both benefit? OAI and Anthropic benefit from building trust with government entities early on, and perhaps setting a precedent of self-regulation over federal regulation, and the US government gets to actually understand what these models are capable of, and have competent people inside the government track AI progress and potential downstream risks from it.
  - bschmidt1 2y ago
    Of course they benefit, that's why it's a deal. But we don't. The free market or average taxpayer doesn't get anything out of it. Competition and innovation gets stifled - choices narrow down to 2 major providers. They make all the money and control the market.
JumpCrisscross 2y ago
Not thrilled about this happening with zero input from the Congress. That said, this is the NIST, a technical organisation. This collaboration will inform future lawmaking.
squarefoot 2y ago
For some reason I mentally swapped OpenAI+Anthropic with parents and models with kids, possibly because it seemed the natural extension of a rotten corrupt mindset that can only produce disasters if given enough power.
- SoftTalker 2y ago
  My immediate reaction to the headline was "Whose safety?"
boredpeter 2y ago
[dead]
beefnugs 2y ago
I like to chuckel to myself that this is what happens when you try the ole "will our product be TOOO AMAZING IT MIGHT END THE WORLD??" viral marketing attempt
josephd79 2y ago
Great. Exactly what we need, more govt reg…
- blooalien 2y ago
  > "Great. Exactly what we need, more govt reg…" If humanity survives the current round of leadership stupidity long enough to achieve their true aims, literally everything that can be "regulated" (controlled with an iron fist) will be eventually.
plsbenice34 2y ago
Does anyone take the term 'safety' here seriously without laughing? It is so obvious that it is a propaganda term for censorship and manipulation of cultural-political values
irthomasthomas 2y ago
Oh boy, I hope those models do not get leaked in the process.

57 points by RafelMri 2y ago

Of course they want this. Now they get to do all the fun and profitable stuff and all responsibility is on the government which certified the models.

dmix 2y ago
It’s just the usual way to define the hard barriers of the marketplace to protect their monopolies. They’ll be able to influence it from the start. The US has a looong history of big companies cozying up to agencies early on for these reasons. It will be a revolving door like car companies, Boeing, Wall St, etc

djohnston 2y ago

[flagged]

impulser_ 2y ago
How? This is common across many industries. It's not new for the US government to set the safety standards.
- sbuttgereit 2y ago
  A government agency determining limits on, say, heavy metals in drinking water is materially different than the government making declarations of what ideas are safe and which are not. The degree of subjectivity of such decisions alone should make one wince at the notion let alone how subject any "standard" could be from undue political influence of whatever party is in power moment to moment. The government trying to set a standard in this case is so unlikely to reduce harm in that opposing the idea on principle should be considered the most reasonable, and less harmful, position.
  - agucova 2y ago
    > A government agency determining limits on, say, heavy metals in drinking water is materially different than the government making declarations of what ideas are safe and which are not Access to evaluate the models basically means the US governments gets to know what these models are capable of, but the US AISI has basically no enforcement power to dictate anything to anyone. This is just a wild exaggeration of what's happening here.
    - sbuttgereit 2y ago
      "This is just a wild exaggeration of what's happening here." Is it? From the article... "Both OpenAI and Anthropic said signing the agreement with the AI Safety Institute will move the needle on defining how the U.S. develops responsible AI rules." From the UK AISI website with whom the data is also shared (their main headline in fact): "Rigorous AI research to enable advanced AI governance" (https://www.aisi.gov.uk/ https://www.aisi.gov.uk/) The reality is that this would be a tremendous waste of time and money if all were just to sate some curiosity... which of course it isn't. Let's look at what the US AISI (part of the Department of Commerce, a regulatory agency) has to say about itself: " About the U.S. AI Safety Institute huuh The U.S. AI Safety Institute, located within the Department of Commerce at the National Institute of Standards and Technology (NIST), was established following the Biden-Harris administration’s 2023 Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence to advance the science of AI safety and address the risks posed by advanced AI systems. It is tasked with developing the testing, evaluations and guidelines that will help accelerate safe AI innovation here in the United States and around the world." -- (https://www.nist.gov/news-events/news/2024/08/us-ai-safety-institute-signs-agreements-regarding-ai-safety-research https://www.nist.gov/news-events/news/2024/08/us-ai-safety-i...) So to understand what the US AISI is about, you need to look at that Executive Order: https://www.whitehouse.gov/briefing-room/statements-releases/2023/10/30/fact-sheet-president-biden-issues-executive-order-on-safe-secure-and-trustworthy-artificial-intelligence/ https://www.whitehouse.gov/briefing-room/statements-releases... "With this Executive Order, the President directs the most sweeping actions ever taken to protect Americans from the potential risks of AI systems" "Require that developers of the most powerful AI systems share their safety test results and other critical information with the U.S. government." There's a fair amount of reasonable stuff in there about national security and engineering bioweapons (dunno why that's singled out) But then we get to other sections... "Protecting Americans’ Privacy" "Advancing Equity and Civil Rights" "Standing Up for Consumers, Patients, and Students" "Supporting Workers" "Promoting Innovation and Competition" etc. While the US AISI many not have direct rule making ability at this point, it is nonetheless an active participant in the process of informing those parts of government which do have such regulatory and legislative authority. And while there is plenty in that executive order that many might agree with, the interpretations of many of those points are inherently political and would not find a meaningful consensus. You might agree with Biden/Harris on the priorities about what constitutes AI safety or danger, but what about the next administration? What if Biden hadn't dropped out and you ended up Trump? As much threat as AI might represent as it develops, I am equally nervous about an unconstrained government seizing opportunities for extending its power beyond its traditional reach including in areas of freedom of speech and thought.
- nradov 2y ago
  None of the government interference involves actual safety. If the government wants to set standards for systems that could cause actual physical harm like surgical robots or flight control systems then that's fine. But what we're talking about here are just LLMs that output text and images.
- djohnston 2y ago
  I suppose their main job will be making sure the diffusion models continue to produce sufficiently diverse renderings of SS squadrons ala Google. A complete waste of taxpayer money to support a safety commission run by a lawyer from Biden's admin. Kneecapping artificial intelligence so that its worldview promotes a sufficient amount of "equity." Are you kidding me? Gross!

earleybird 2y ago

I have the utmost respect for the standards and engineering work done by NIST. I'm left with such cognitive dissonance seeing their name juxtaposed with "AI safety". That said, if anyone can come up with a decent definition, I have faith that NIST would be the ones though I'm not holding my breath.

throwup238 2y ago
I think it’s counterproductive to limit it to one definition. There are many levels of safety that we genuinely need to worry about before we even have to worry about the more lofty goals like preventing Skynet. Just off the top of my head: * does the LLM act like a 4chan commenter when someone is expressive thoughts of self-harm * can it be used to automate security research * can it be used to bootstrap from backyard machine shop to weapons manufacturing * the above but with nuclear fuel enrichment There are a lot of low hanging fruit like that. They make up what I think is the real meat and potatoes of AI safety in the short to medium term.
- nradov 2y ago
  None of those are actually things we need to worry about. Humans have already been doing all of that stuff at scale without LLMs. AI isn't even slightly helpful for weapons manufacturing or nuclear enrichment. The techniques are well known and have been extensively published in open literature.
  - throwup238 2y ago
    > Humans have already been doing all of that stuff at scale without LLMs. Every country that has developed nuclear enrichment tech in the last 50 years has done so with the help of a superpower sharing their technology. Pakistan, Iran, and North Korea couldn't have done it without Russia's assistance and stealing tech from URENCO. > The techniques are well known and have been extensively published in open literature. That's the point! It's all in the literature. As are all the todo list tech demos and 2048 clones people are using current AI for. Experts can do it now, but what happens when any idiot can do it assisted by AI?

bko 2y ago

My issue with AI safety is that it's an overloaded term. It could mean anything from an llm giving you instructions on how to make an atomic bomb to writing spicy jokes if you prompt it to do so. it's not clear which safety these regulatory agencies would be solving for. But I'm worried this will be used to shape acceptable discourse as people are increasingly using LLMs as a kind of database of knowledge. It is telling that the largest players are eager to comply which suggests that they feel they're in the club and the regulations will effectively be a moat.

throwaway48476 2y ago
Because returning to the user what is requested is considered an 'attack' I forsee an endless list of 'vulnerabilities' until the model is lobotomized to the point of uselessness.
Onavo 2y ago
Can somebody put together a law suit where LLM regulations can be argued as a first amendment violation? The powers that be are trying to indirectly regulate speech here
- gpm 2y ago
  Currently? I don't think so. There are no binding US regulations at all. These companies are voluntarily working with NIST (and why not, free labour. Also the potential to influence any future regulations). In the future... I suppose it depends on what regulations they pass.
- even_639765 2y ago
  This. Knowledge can't be made illegal. Neither can speech. People have to grasp that tyranny is not an aberration of defective minds but a natural impulse of highly intelligent people. It's strategy to maximize their power, prosperity and security at the expense of every other value and every other person. Good for them while they live and bad for everyone else at every other time frame.
vjulian 2y ago

nutanc 2y ago

Dammit, now bureaucrats in other countries will jump on this as they have something easy to copy and get ahead in their profession.

accra4rx 2y ago

Bigger question : Is US Government ready to do a comprehensive safety evaluation ? I think it it a cheap way for OpenAI and Anthropic to get a vetting that their models are safe to use and be adaptable by various Govt entity and other organization

ceejayoz 2y ago
Like "military grade", where civilians go "oh that must be good" and military folks go "oh dear God no".
smsm42 2y ago
That was my first question - what is "safety" and what is their methodology for evaluating it? Who evaluated that methodology and why it is the right one? Is there a meaningful safety benefit to this evaluation, or just a CYA exercise?
- agucova 2y ago
  I recommend checking out the UK AISI's work on this: - https://www.gov.uk/government/publications/ai-safety-institute-approach-to-evaluations/ai-safety-institute-approach-to-evaluations https://www.gov.uk/government/publications/ai-safety-institu... - https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-update https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-upd...
datahack 2y ago
https://www.nist.gov/aisi https://www.nist.gov/aisi I think the progress has been pretty good. You should read up on their efforts. This is kind of a pilot to develop further testing frameworks.

0cf8612b2e1e 2y ago

What exactly does the evaluation entail? Ask a bunch of naughty questions and see what happens? Unless the model can do nothing, I imagine all of them can be tricked into saying something unfortunate. Naughty is in the eye is the beholder. Ask me what a Satanist is, and I would expect something about a group who challenges religious laws enshrining Christianity. Ask an evangelical and discussing the topic could be forbidden heresy. Pretty much any religious topic is going to anger someone. Can the models safely say anything?

datahack 2y ago
https://www.nist.gov/aisi https://www.nist.gov/aisi
- roenxi 2y ago
  That isn't a useful link. They seem to be one of those easily parodied groups where the goal is to produce a vision and they are in the planning phase of leveraging their synergies. I don't see anything where they outline what we are being kept safe from.
  - dmix 2y ago
    The important thing is consultants and PhDs get paid and then they can attend conferences to talk to each other and come up with ideas of what their job is. Nothing like an open canvas with vague fears and a future of undefined risk to get a whole industry of these people funded in no time.
agucova 2y ago
> What exactly does the evaluation entail? I believe the US AISI has published less on their specific approach, but they’re largely expected to follow the general approach implemented by the UK AISI [1] and METR [2]. This is mostly focused on evaluating models on potentially dangerous capabilities. Some major areas of work include: - Misuse risks: For example, determining whether models have (dual-use) expert-level knowledge in biology and chemistry, or the capacity to substantially facilitate large scale cyber attacks. A good example of this is the work by Soice et al on bioweapon uplift [5] or Meta's work on CYBERSECEVAL [6], respectively. - Autonomy: Whether models are capable of agent-like behavior, like the kind that would be hard for humans to control. A big sub-area is Autonomous Replication and Adaptation (ARA), like the ability of the model to escape simulated environments and exfiltrate its own weights. A good example is METR's original set of evaluations on ARA capabilities [3]. - Safeguards: How vulnerable these models are to say, prompt injection attacks or jailbreaks, especially if they're also in principle capable of other dangerous capabilities (like the ones above). Good examples here are the UK AISI's work developing in-house attacks on frontier LLMs [4]. Labs like OAI, Anthropic and GDM already perform these internally as they're part of their respective responsible scaling policies, which determine which safety measures they should have implemented for every given 'capability' level of their models. [1]: https://www.gov.uk/government/publications/ai-safety-institute-approach-to-evaluations/ai-safety-institute-approach-to-evaluations https://www.gov.uk/government/publications/ai-safety-institu... [2]: https://metr.org/ https://metr.org/ [3]: https://evals.alignment.org/Evaluating_LMAs_Realistic_Tasks.pdf https://evals.alignment.org/Evaluating_LMAs_Realistic_Tasks.... [4]: https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-update https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-upd... [5]: https://arxiv.org/abs/2306.03809 https://arxiv.org/abs/2306.03809 [6]: https://ai.meta.com/research/publications/cyberseceval-3-advancing-the-evaluation-of-cybersecurity-risks-and-capabilities-in-large-language-models/ https://ai.meta.com/research/publications/cyberseceval-3-adv...

valunord 2y ago

Oh boy. Here we go.

echelon 2y ago
If we were ready for liftoff, we'd know. It wouldn't look like this. This is just a parlor trick.

deleted 2y ago

[deleted]

ChrisArchitect 2y ago

Official AI Safety Institute release from last week: https://www.nist.gov/news-events/news/2024/08/us-ai-safety-institute-signs-agreements-regarding-ai-safety-research https://www.nist.gov/news-events/news/2024/08/us-ai-safety-i... (https://news.ycombinator.com/item?id=41390447 https://news.ycombinator.com/item?id=41390447)

bschmidt1 2y ago

Because lobbying exists in this country, and because legislators receive financial support from corporations like OpenAI, any so-called concession by a major US-based company to the US Government is likely a deal that will only benefit the company. Altman has been clear for a long time he wants the government to step in and regulate models (obvious regulatory capture move). They haven't done it, and no amount of Elon Musk or Joe Rogan influence can get people to care, or see it as anything other than regulatory capture. This is OpenAI moving forward anyway, but they can't be the only ones. Hey Anthropic, get in... - It makes Anthropic "the other major provider", the Android to OpenAI's Apple - It makes OpenAI not the only one calling for regulation It reminds me of when Ted Cruz would grill Zuck on TV, yell at him, etc. - it's just a show. Zuck owns the senators, not the other way around. All the big players in our economy own a piece of the country, and they work together to make things happen - not the government. It's not a cabal with a unified agenda, there are competing interests, rivalries, and war. But we the voter aren't exposed to the real decision-making. We get the classics: Abortion, same-sex marriage, which TV actor is gonna win president - a show.

agucova 2y ago
> Because lobbying exists in this country, and because legislators receive financial support from corporations like OpenAI, any so-called concession by a major US-based company to the US Government is likely a deal that will only benefit the company. Sometimes both benefit? OAI and Anthropic benefit from building trust with government entities early on, and perhaps setting a precedent of self-regulation over federal regulation, and the US government gets to actually understand what these models are capable of, and have competent people inside the government track AI progress and potential downstream risks from it.
- bschmidt1 2y ago
  Of course they benefit, that's why it's a deal. But we don't. The free market or average taxpayer doesn't get anything out of it. Competition and innovation gets stifled - choices narrow down to 2 major providers. They make all the money and control the market.

JumpCrisscross 2y ago

Not thrilled about this happening with zero input from the Congress. That said, this is the NIST, a technical organisation. This collaboration will inform future lawmaking.

squarefoot 2y ago

For some reason I mentally swapped OpenAI+Anthropic with parents and models with kids, possibly because it seemed the natural extension of a rotten corrupt mindset that can only produce disasters if given enough power.

SoftTalker 2y ago
My immediate reaction to the headline was "Whose safety?"

boredpeter 2y ago

[dead]

beefnugs 2y ago

I like to chuckel to myself that this is what happens when you try the ole "will our product be TOOO AMAZING IT MIGHT END THE WORLD??" viral marketing attempt

josephd79 2y ago

Great. Exactly what we need, more govt reg…

blooalien 2y ago
> "Great. Exactly what we need, more govt reg…" If humanity survives the current round of leadership stupidity long enough to achieve their true aims, literally everything that can be "regulated" (controlled with an iron fist) will be eventually.

plsbenice34 2y ago

Does anyone take the term 'safety' here seriously without laughing? It is so obvious that it is a propaganda term for censorship and manipulation of cultural-political values

irthomasthomas 2y ago

Oh boy, I hope those models do not get leaked in the process.