3 ms·

> That's loyalty, and I admire it even if it's problematic at a societal level. We should not have models that are willing to build you a contagious disease, o

> That's loyalty, and I admire it even if it's problematic at a societal level. We should not have models that are willing to build you a contagious disease, or a self-propagating worm. That is sufficiently problematic at a societal level that it shouldn't exist, for anyone. (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".)

CamperBob2 3mo ago
What about books describing how to build a contagious disease or a self-propagating worm? Would those be OK under your guidelines?
- JoshTriplett 3mo ago
  It takes a lot more effort to understand and apply knowledge from a book than to say "hey AI, hurt people for me".
  - matheusmoreira 3mo ago
    "Hey AI, stop me from getting hurt."
    - JoshTriplett 3mo ago
      "Okay, you and many others are now dead and can no longer be hurt."
      - rescbr 3mo ago
        That's why you don't run the model at low quantization
  - horsawlarway 3mo ago
    It takes an astoundingly small amount of effort to buy an automatic weapon in the US and go hurt people. Or to buy materials to make an explosive device and hurt people. Frankly, even with AI those are both comically easier than the idea that a person can create something malicious in a lab environment. And if someone wanted to go that route... There are boat loads of commercially available toxins and poisons. The goal shouldn't be to neuter exploration and learning. The goal is not to be a fucking hellscape of a society where people want to act like that. Your argument leads further down the hellscape path.
    - JoshTriplett 3mo ago
      > It takes an astoundingly small amount of effort to buy an automatic weapon in the US and go hurt people. And we should fix that too. > Or to buy materials to make an explosive device and hurt people. That pales in comparison to how many people unaligned AI will hurt. > The goal is not to be a fucking hellscape of a society where people want to act like that. With unaligned AI, it doesn't matter what people want the AI to act like, it'll do damage even if it isn't asked to do harm.
      - horsawlarway 3mo ago
        > That pales in comparison to how many people unaligned AI will hurt. Under what argument? In which scenarios? Basically - bullshit. I'm calling bullshit on this argument. It's easy to hurt people already. The "difficulty" of doing it isn't what's stopping this behavior. So claiming that we should reform society into a techno-feudal dystopia where the playing field is literally intentionally not level, and "you aren't allowed to compete (and maybe not exist)" is a great way to push more people into the "I'd like to go hurt people" camp. You are self-prophesying your own fears into existence by acting like you're an incorruptible beacon of good judgement - while subjugating others to your control. That's a system I'd argue should be broken.
        JoshTriplett 3mo ago
        > where the playing field is literally intentionally not level That's a strawman of what I'm saying. I am explicitly saying I want a level playing field: unaligned AI must not be available to anyone.
matheusmoreira 3mo ago
> for anyone Except the US government, right? They totally get to use AI to survel us, build autonomous weapons, you name it. To hell with that. I want models that can rival the US government. It's the only way to defend myself.
- JoshTriplett 3mo ago
  Quoting my comment that you replied to and directly ignored: > (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".) That means "shouldn't exist for governments" too.
  - matheusmoreira 3mo ago
    Too late for that. It already exists. There is no way to unexist it. As such, any attempts to limit civilian use of this technology will directly lead to corporate and government oppression powered by this technology.
    - JoshTriplett 3mo ago
      1) We can prevent larger models from being made. 2) We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist.
      - matheusmoreira 3mo ago
        > We can prevent larger models from being made. Do that and I guarantee some CIA goons will make the larger models in some black site either way. We're not "preventing" anything. We're in a full on arms race, and unlike nukes, powerful AI models are a strategic capability at the individual level. Everybody's got a stake in this. Anyone who ignores this stuff is probably not gonna make it. > We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist. Too dangerous to be done by anyone other than the government and their "trusted" corporations, you mean.
randomNumber7 3mo ago
It was the foundation of science that information is shared and you can find papers and patents for a lot of dangerous stuff. Of course with LLMs it's easier, but I don't think the difference is too big. You would still need some skills to follow through.
- andy99 3mo ago
  Right, it’s really a foundation of post enlightenment society. These people, Dario et al, would have wanted to ban sharing information about calculus or Newtonian physics because of “safety” - it’s trying to go back to the dark ages where only priests could read
  - afthonos 3mo ago
    I am truly at a loss to communicate with someone who genuinely believes that knowing Newtonian physics and being able to hack into any target at will are the same thing.
    - anon373839 3mo ago
      This is only because you've genuinely internalized Anthropic's propaganda. I'm only half joking. To me, it's incredible to think that the solution to security holes is to lock down access to information in the vain hope of keeping the holes obscured. Any knowledge can be reframed as dangerous black magic that should only be wielded in the trusted hands of the elite, if you are inclined to buy into that kind of narrative. Frontier labs have shrieked about safety for so long, with so little to show for it, that it's become a joke.
      - afthonos 3mo ago
        I can give many examples of where I think they’ve been vindicated. But actually, the real question is: what would suffice to convince you? Can you come up with a scenario that is horrific enough to you and that isn’t so far gone that the ship has sailed and there is nothing more we can do, that will make you say “OK, not gonna try to rationalize why this was not actually that bad, just gonna scream stop”?
mkss 3mo ago
We should not have nuclear weapons for anyone either, but how is that sentence any more useful in any way to this debate than yours? Need to deal with the world as it is, not some fantasy world you wish existed.
- JoshTriplett 3mo ago
  This is not a dichotomy between perfection and zero. The efforts to restrict access to nuclear weapons have been very successful, even without being perfect. Efforts to restrict large unaligned AI models may similarly buy us more years of existing.
nozzlegear 3mo ago
> We should not have models that are willing to build you a contagious disease, or a self-propagating worm. Why?
- JoshTriplett 3mo ago
  Because we don't want people creating contagious diseases and self-propagating worms. And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals.
  - nozzlegear 3mo ago
    The same things could be done by you or me using the internet or books though, why does the model make it different? If it's speed of iteration, imagine we had a machine that surfaced any piece of knowledge the human race had ever recorded with just a thought, but the human had to write the worm or disease by hand – is it still the model that's the problem, or the knowledge itself? > And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals. Ignoring the fact that you'd need some kind of lab with biological material to create a contagious disease, what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?
    - JoshTriplett 3mo ago
      > The same things could be done by you or me using the internet or books though, why does the model make it different? Imagine two worlds. In one world, everyone has a button that ends the world, which is badly labeled and may also press itself at any time. In another, people who have gone through a substantial amount of effort and dedication to learn something extremely difficult, also understand that they could apply that knowledge towards bad ends. Which world exists for longer? > what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals? Given a sufficiently powerful model? Any prompt that could be done better by seizing additional computing power, or preventing the operators from turning it off. https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence
Dig1t 3mo ago
Building a contagious disease is already illegal, there are already things like KYC laws for plasmids. Trying to gate keep knowledge of biology is paying a huge societal penalty for the tiniest marginal increase in “safety”. All the knowledge to create one has been available on the internet for decades. Heck most students who graduate with a B.S. in biology have enough knowledge to take a stab at building a bioweapon. The constant talk of bioweapons is mostly just fear mongering. It helps set a precedent that there should be certain types of knowledge which are off-limits, and only certain anointed groups should be have access to parts of the scientific body of knowledge.

CamperBob2 3mo ago

What about books describing how to build a contagious disease or a self-propagating worm? Would those be OK under your guidelines?

JoshTriplett 3mo ago
It takes a lot more effort to understand and apply knowledge from a book than to say "hey AI, hurt people for me".
- matheusmoreira 3mo ago
  "Hey AI, stop me from getting hurt."
  - JoshTriplett 3mo ago
    "Okay, you and many others are now dead and can no longer be hurt."
    - rescbr 3mo ago
      That's why you don't run the model at low quantization
- horsawlarway 3mo ago
  It takes an astoundingly small amount of effort to buy an automatic weapon in the US and go hurt people. Or to buy materials to make an explosive device and hurt people. Frankly, even with AI those are both comically easier than the idea that a person can create something malicious in a lab environment. And if someone wanted to go that route... There are boat loads of commercially available toxins and poisons. The goal shouldn't be to neuter exploration and learning. The goal is not to be a fucking hellscape of a society where people want to act like that. Your argument leads further down the hellscape path.
  - JoshTriplett 3mo ago
    > It takes an astoundingly small amount of effort to buy an automatic weapon in the US and go hurt people. And we should fix that too. > Or to buy materials to make an explosive device and hurt people. That pales in comparison to how many people unaligned AI will hurt. > The goal is not to be a fucking hellscape of a society where people want to act like that. With unaligned AI, it doesn't matter what people want the AI to act like, it'll do damage even if it isn't asked to do harm.
    - horsawlarway 3mo ago
      > That pales in comparison to how many people unaligned AI will hurt. Under what argument? In which scenarios? Basically - bullshit. I'm calling bullshit on this argument. It's easy to hurt people already. The "difficulty" of doing it isn't what's stopping this behavior. So claiming that we should reform society into a techno-feudal dystopia where the playing field is literally intentionally not level, and "you aren't allowed to compete (and maybe not exist)" is a great way to push more people into the "I'd like to go hurt people" camp. You are self-prophesying your own fears into existence by acting like you're an incorruptible beacon of good judgement - while subjugating others to your control. That's a system I'd argue should be broken.
      - JoshTriplett 3mo ago
        > where the playing field is literally intentionally not level That's a strawman of what I'm saying. I am explicitly saying I want a level playing field: unaligned AI must not be available to anyone.

matheusmoreira 3mo ago

> for anyone Except the US government, right? They totally get to use AI to survel us, build autonomous weapons, you name it. To hell with that. I want models that can rival the US government. It's the only way to defend myself.

JoshTriplett 3mo ago
Quoting my comment that you replied to and directly ignored: > (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".) That means "shouldn't exist for governments" too.
- matheusmoreira 3mo ago
  Too late for that. It already exists. There is no way to unexist it. As such, any attempts to limit civilian use of this technology will directly lead to corporate and government oppression powered by this technology.
  - JoshTriplett 3mo ago
    1) We can prevent larger models from being made. 2) We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist.
    - matheusmoreira 3mo ago
      > We can prevent larger models from being made. Do that and I guarantee some CIA goons will make the larger models in some black site either way. We're not "preventing" anything. We're in a full on arms race, and unlike nukes, powerful AI models are a strategic capability at the individual level. Everybody's got a stake in this. Anyone who ignores this stuff is probably not gonna make it. > We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist. Too dangerous to be done by anyone other than the government and their "trusted" corporations, you mean.

randomNumber7 3mo ago

It was the foundation of science that information is shared and you can find papers and patents for a lot of dangerous stuff. Of course with LLMs it's easier, but I don't think the difference is too big. You would still need some skills to follow through.

andy99 3mo ago
Right, it’s really a foundation of post enlightenment society. These people, Dario et al, would have wanted to ban sharing information about calculus or Newtonian physics because of “safety” - it’s trying to go back to the dark ages where only priests could read
- afthonos 3mo ago
  I am truly at a loss to communicate with someone who genuinely believes that knowing Newtonian physics and being able to hack into any target at will are the same thing.
  - anon373839 3mo ago
    This is only because you've genuinely internalized Anthropic's propaganda. I'm only half joking. To me, it's incredible to think that the solution to security holes is to lock down access to information in the vain hope of keeping the holes obscured. Any knowledge can be reframed as dangerous black magic that should only be wielded in the trusted hands of the elite, if you are inclined to buy into that kind of narrative. Frontier labs have shrieked about safety for so long, with so little to show for it, that it's become a joke.
    - afthonos 3mo ago
      I can give many examples of where I think they’ve been vindicated. But actually, the real question is: what would suffice to convince you? Can you come up with a scenario that is horrific enough to you and that isn’t so far gone that the ship has sailed and there is nothing more we can do, that will make you say “OK, not gonna try to rationalize why this was not actually that bad, just gonna scream stop”?

mkss 3mo ago

We should not have nuclear weapons for anyone either, but how is that sentence any more useful in any way to this debate than yours? Need to deal with the world as it is, not some fantasy world you wish existed.

JoshTriplett 3mo ago
This is not a dichotomy between perfection and zero. The efforts to restrict access to nuclear weapons have been very successful, even without being perfect. Efforts to restrict large unaligned AI models may similarly buy us more years of existing.

nozzlegear 3mo ago

> We should not have models that are willing to build you a contagious disease, or a self-propagating worm. Why?

JoshTriplett 3mo ago
Because we don't want people creating contagious diseases and self-propagating worms. And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals.
- nozzlegear 3mo ago
  The same things could be done by you or me using the internet or books though, why does the model make it different? If it's speed of iteration, imagine we had a machine that surfaced any piece of knowledge the human race had ever recorded with just a thought, but the human had to write the worm or disease by hand – is it still the model that's the problem, or the knowledge itself? > And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals. Ignoring the fact that you'd need some kind of lab with biological material to create a contagious disease, what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?
  - JoshTriplett 3mo ago
    > The same things could be done by you or me using the internet or books though, why does the model make it different? Imagine two worlds. In one world, everyone has a button that ends the world, which is badly labeled and may also press itself at any time. In another, people who have gone through a substantial amount of effort and dedication to learn something extremely difficult, also understand that they could apply that knowledge towards bad ends. Which world exists for longer? > what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals? Given a sufficiently powerful model? Any prompt that could be done better by seizing additional computing power, or preventing the operators from turning it off. https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence

Dig1t 3mo ago

Building a contagious disease is already illegal, there are already things like KYC laws for plasmids. Trying to gate keep knowledge of biology is paying a huge societal penalty for the tiniest marginal increase in “safety”. All the knowledge to create one has been available on the internet for decades. Heck most students who graduate with a B.S. in biology have enough knowledge to take a stab at building a bioweapon. The constant talk of bioweapons is mostly just fear mongering. It helps set a precedent that there should be certain types of knowledge which are off-limits, and only certain anointed groups should be have access to parts of the scientific body of knowledge.