5 ms·

Make it easy for these companies by passing a law that any AI model a company offers to the public (non-government) has to be released as open weights. This is

Make it easy for these companies by passing a law that any AI model a company offers to the public (non-government) has to be released as open weights. This is how we deflate the bubble safely and completely, without having to introduce a government regulator that is likely to go either too far or not far enough. This resolves the coordination problem among these potentially good actors.

blululu 19d ago
What? This is a terrible policy. In addition to the obvious AI safety risks, this just amounts to a free R&D subsidy to Amazon for hosting models. It would remove any incentive to do any R&D without AWS level physical infrastructure.
- jacobgold 19d ago
  That's an understandable first thought, but consider what the goal is for this: slowing down future development. This can be achieved by dramatically reducing the market value of models.
  - blululu 19d ago
    Ok, yeah. That's an interesting point.
0xDEAFBEAD 19d ago
If Astra is open-weight, black hats can download it, remove guardrails, and go on an automated hacking spree.
- jacobgold 19d ago
  Just like they're already doing with frontier-level open weight models? Do you imagine that Russian and Chinese intelligence don't have copies of the Mythos/Fable weights? The point is to slow progress of future models. Nothing can stop what has already been created.
  - 0xDEAFBEAD 19d ago
    With closed-weight, we can get the models out to defenders before they're in the hands of attackers. There are also other categories of risk such as biorisk. Open-weight means no guardrails. Don't get me wrong, I like your idea of deflating the AI bubble, I'm just skeptical that this is a great way to do it. The PauseAI people have some ideas: https://pauseai.info/proposal https://pauseai.info/proposal * International body which must approve new training runs * Ban training on copyrighted material * Hold AI model creators liable The essential benefit of your idea is making new training runs unprofitable. But I imagine there are other ways to accomplish that, e.g. by taxing AI companies to the point where they are just barely squeaking by financially.
    - bigyabai 19d ago
      > Open-weight means no guardrails. We're going to live in that world forever now, capable adversarial models have already been trained and will be used in a large conflict. The pragmatic issue is hacking, and current open-weight LLMs are already capable of it. Bioweapons and RSI are pie-in-the-sky theories that distract from the real-world situation that each government wants to exploit.
      - 0xDEAFBEAD 19d ago
        >Bioweapons and RSI are pie-in-the-sky theories that distract from the real-world situation It wasn't long ago when automated hacking was a pie-in-the-sky theory. Sometimes it's helpful to prepare for things in advance. An ounce of prevention can be worth a pound of cure.
        bigyabai 19d ago
        But hacking has always been a vulnerability, even pre-AI. The scale multiplier here is the number of vulnerable machines, not the intelligence of AI. > An ounce of prevention can be worth a pound of cure. It can be, but if you get chemotherapy on a hunch instead of a diagnosis then it could kill you instead. There is no evidence that RSI will ever exist.
- preisschild 19d ago
  So? If the models are open more people can use it to improve their security too.

blululu 19d ago

What? This is a terrible policy. In addition to the obvious AI safety risks, this just amounts to a free R&D subsidy to Amazon for hosting models. It would remove any incentive to do any R&D without AWS level physical infrastructure.

jacobgold 19d ago
That's an understandable first thought, but consider what the goal is for this: slowing down future development. This can be achieved by dramatically reducing the market value of models.
- blululu 19d ago
  Ok, yeah. That's an interesting point.

0xDEAFBEAD 19d ago

If Astra is open-weight, black hats can download it, remove guardrails, and go on an automated hacking spree.

jacobgold 19d ago
Just like they're already doing with frontier-level open weight models? Do you imagine that Russian and Chinese intelligence don't have copies of the Mythos/Fable weights? The point is to slow progress of future models. Nothing can stop what has already been created.
- 0xDEAFBEAD 19d ago
  With closed-weight, we can get the models out to defenders before they're in the hands of attackers. There are also other categories of risk such as biorisk. Open-weight means no guardrails. Don't get me wrong, I like your idea of deflating the AI bubble, I'm just skeptical that this is a great way to do it. The PauseAI people have some ideas: https://pauseai.info/proposal https://pauseai.info/proposal * International body which must approve new training runs * Ban training on copyrighted material * Hold AI model creators liable The essential benefit of your idea is making new training runs unprofitable. But I imagine there are other ways to accomplish that, e.g. by taxing AI companies to the point where they are just barely squeaking by financially.
  - bigyabai 19d ago
    > Open-weight means no guardrails. We're going to live in that world forever now, capable adversarial models have already been trained and will be used in a large conflict. The pragmatic issue is hacking, and current open-weight LLMs are already capable of it. Bioweapons and RSI are pie-in-the-sky theories that distract from the real-world situation that each government wants to exploit.
    - 0xDEAFBEAD 19d ago
      >Bioweapons and RSI are pie-in-the-sky theories that distract from the real-world situation It wasn't long ago when automated hacking was a pie-in-the-sky theory. Sometimes it's helpful to prepare for things in advance. An ounce of prevention can be worth a pound of cure.
      - bigyabai 19d ago
        But hacking has always been a vulnerability, even pre-AI. The scale multiplier here is the number of vulnerable machines, not the intelligence of AI. > An ounce of prevention can be worth a pound of cure. It can be, but if you get chemotherapy on a hunch instead of a diagnosis then it could kill you instead. There is no evidence that RSI will ever exist.
preisschild 19d ago
So? If the models are open more people can use it to improve their security too.