3 ms·

Just like they're already doing with frontier-level open weight models? Do you imagine that Russian and Chinese intelligence don't have copies of the Mythos/Fab

Just like they're already doing with frontier-level open weight models? Do you imagine that Russian and Chinese intelligence don't have copies of the Mythos/Fable weights? The point is to slow progress of future models. Nothing can stop what has already been created.

0xDEAFBEAD 18d ago
With closed-weight, we can get the models out to defenders before they're in the hands of attackers. There are also other categories of risk such as biorisk. Open-weight means no guardrails. Don't get me wrong, I like your idea of deflating the AI bubble, I'm just skeptical that this is a great way to do it. The PauseAI people have some ideas: https://pauseai.info/proposal https://pauseai.info/proposal * International body which must approve new training runs * Ban training on copyrighted material * Hold AI model creators liable The essential benefit of your idea is making new training runs unprofitable. But I imagine there are other ways to accomplish that, e.g. by taxing AI companies to the point where they are just barely squeaking by financially.
- bigyabai 18d ago
  > Open-weight means no guardrails. We're going to live in that world forever now, capable adversarial models have already been trained and will be used in a large conflict. The pragmatic issue is hacking, and current open-weight LLMs are already capable of it. Bioweapons and RSI are pie-in-the-sky theories that distract from the real-world situation that each government wants to exploit.
  - 0xDEAFBEAD 18d ago
    >Bioweapons and RSI are pie-in-the-sky theories that distract from the real-world situation It wasn't long ago when automated hacking was a pie-in-the-sky theory. Sometimes it's helpful to prepare for things in advance. An ounce of prevention can be worth a pound of cure.
    - bigyabai 18d ago
      But hacking has always been a vulnerability, even pre-AI. The scale multiplier here is the number of vulnerable machines, not the intelligence of AI. > An ounce of prevention can be worth a pound of cure. It can be, but if you get chemotherapy on a hunch instead of a diagnosis then it could kill you instead. There is no evidence that RSI will ever exist.
      - 0xDEAFBEAD 17d ago
        >There is no evidence that RSI will ever exist. So we wait until it's actively happening to do anything about it?
        bigyabai 16d ago
        Yeah. Do you think we should regulate Red Mercury just in case someone figures out how to build a nuke with it? Red mercury and RSI are both science fiction psyops. They do not exist.

0xDEAFBEAD 18d ago

With closed-weight, we can get the models out to defenders before they're in the hands of attackers. There are also other categories of risk such as biorisk. Open-weight means no guardrails. Don't get me wrong, I like your idea of deflating the AI bubble, I'm just skeptical that this is a great way to do it. The PauseAI people have some ideas: https://pauseai.info/proposal https://pauseai.info/proposal * International body which must approve new training runs * Ban training on copyrighted material * Hold AI model creators liable The essential benefit of your idea is making new training runs unprofitable. But I imagine there are other ways to accomplish that, e.g. by taxing AI companies to the point where they are just barely squeaking by financially.

bigyabai 18d ago
> Open-weight means no guardrails. We're going to live in that world forever now, capable adversarial models have already been trained and will be used in a large conflict. The pragmatic issue is hacking, and current open-weight LLMs are already capable of it. Bioweapons and RSI are pie-in-the-sky theories that distract from the real-world situation that each government wants to exploit.
- 0xDEAFBEAD 18d ago
  >Bioweapons and RSI are pie-in-the-sky theories that distract from the real-world situation It wasn't long ago when automated hacking was a pie-in-the-sky theory. Sometimes it's helpful to prepare for things in advance. An ounce of prevention can be worth a pound of cure.
  - bigyabai 18d ago
    But hacking has always been a vulnerability, even pre-AI. The scale multiplier here is the number of vulnerable machines, not the intelligence of AI. > An ounce of prevention can be worth a pound of cure. It can be, but if you get chemotherapy on a hunch instead of a diagnosis then it could kill you instead. There is no evidence that RSI will ever exist.
    - 0xDEAFBEAD 17d ago
      >There is no evidence that RSI will ever exist. So we wait until it's actively happening to do anything about it?
      - bigyabai 16d ago
        Yeah. Do you think we should regulate Red Mercury just in case someone figures out how to build a nuke with it? Red mercury and RSI are both science fiction psyops. They do not exist.