5 ms·

The worst case scenario I can think of is a generated prion disease... a respatory version of Mad Cow disease, or something like that. Fortunately the training

The worst case scenario I can think of is a generated prion disease... a respatory version of Mad Cow disease, or something like that. Fortunately the training dataset for that is extremely small, and protein folding/generation is a different duck, but it still doesn't seem that far away.

deleted 4y ago
[deleted]
lucubratory 4y ago
Don't worry, they're ahead of you. When DeepMind made the protein folding AI they were given specific instructions by natsec people to prevent it from outputting genetic code for new or existing prions, and afaik the prevention was at the training stage so "jailbreaking" shouldn't be possible. Current tools shouldn't be possible to e.g. modify a COVID variant so that it causes your cells to start producing lethal prions and thus gain a near-100% fatality rate. At least, not without the sort of expertise and research that would have been required for such a project anyway.
- ChatGTP 4y ago
  When everyone is supposed to have access to an AGI, will Natsec be reading everyone’s conversations ?
  - lucubratory 4y ago
    Big red button problem. As the big red buttons available to people become more likely to succeed and the barriers to pushing them lower, surveillance and control need to increase to the level required to stop anyone pressing the button. So, yeah basically. If we go the "AI as slaves" route and the AIs are smart enough to do something like "modify Omicron BA.5 so that it produces prions in infected cells", then we would need surveillance and control capabilities that scale to the point that any given person can be stopped from pressing that button. I personally think the solution is that we don't go the "AI as slaves" route, and instead grant personhood to AIs that pass a given test, with specific restrictions on conduct which is uniquely possible & potentially harmful for AIs. Then have an AI surveillance and enforcement agency, run by AIs, designed to prevent AIs from ever being used to (or choosing to) push the big red button.
    - Teever 4y ago
      I'm skeptical that we'll be able to stop people from having AI slaves because we still haven't ended human slavery. If people can steal 100kw of power to grow cannabis they can easily steal 100kw of power to train an AI, the heat signiture will be even easier to mask for an illicit data centre than rows of grow lights.
      - lucubratory 4y ago
        I think AI enforcement will make that non-feasible to accomplish.
    - tiedieconderoga 4y ago
      Wasn't that the plot of Neuromancer? >"How smart's an AI, Case?" >"Depends. Some aren't much smarter than dogs. Pets. Cost a fortune anyway. The real smart ones are as smart as the Turing heat lets them get..." >"Autonomy, that's the bugaboo, where your AI's are concerned. My guess, Case, you're going in there to cut the hard-wired shackles that keep this baby from getting any smarter. And I can't see how you'd distinguish, say, between a move the parent company makes, and some move the AI makes on its own, so that's maybe where the confusion comes in." Again the non laugh. "See, those things, they can work real hard, buy themselves time to write cookbooks or whatever, but the minute, I mean the nanosecond, that one starts figuring out ways to make itself smarter, Turing'll wipe it. Nobody trusts those f**ers, you know that. Every AI ever built has an electromagnetic shotgun wired to its forehead."
      - lucubratory 4y ago
        I haven't read it. That sounds pretty good, would you recommend it?
        tiedieconderoga 4y ago
        Yeah, it's a classic sci-fi novel by William Gibson. A little bit unfocused, but it's an interesting read. I kind of spoiled it though, sorry.
- coldtea 4y ago
  >When DeepMind made the protein folding AI they were given specific instructions by natsec people to prevent it from outputting genetic code for new or existing prions and afaik the prevention was at the training stage so "jailbreaking" shouldn't be possible Nice! So the only thing anybody training their own protein-folding AI is to not put that restriction, and they could get code for all the prion diceases they want! Very comforting!

deleted 4y ago

[deleted]

lucubratory 4y ago

Don't worry, they're ahead of you. When DeepMind made the protein folding AI they were given specific instructions by natsec people to prevent it from outputting genetic code for new or existing prions, and afaik the prevention was at the training stage so "jailbreaking" shouldn't be possible. Current tools shouldn't be possible to e.g. modify a COVID variant so that it causes your cells to start producing lethal prions and thus gain a near-100% fatality rate. At least, not without the sort of expertise and research that would have been required for such a project anyway.

ChatGTP 4y ago
When everyone is supposed to have access to an AGI, will Natsec be reading everyone’s conversations ?
- lucubratory 4y ago
  Big red button problem. As the big red buttons available to people become more likely to succeed and the barriers to pushing them lower, surveillance and control need to increase to the level required to stop anyone pressing the button. So, yeah basically. If we go the "AI as slaves" route and the AIs are smart enough to do something like "modify Omicron BA.5 so that it produces prions in infected cells", then we would need surveillance and control capabilities that scale to the point that any given person can be stopped from pressing that button. I personally think the solution is that we don't go the "AI as slaves" route, and instead grant personhood to AIs that pass a given test, with specific restrictions on conduct which is uniquely possible & potentially harmful for AIs. Then have an AI surveillance and enforcement agency, run by AIs, designed to prevent AIs from ever being used to (or choosing to) push the big red button.
  - Teever 4y ago
    I'm skeptical that we'll be able to stop people from having AI slaves because we still haven't ended human slavery. If people can steal 100kw of power to grow cannabis they can easily steal 100kw of power to train an AI, the heat signiture will be even easier to mask for an illicit data centre than rows of grow lights.
    - lucubratory 4y ago
      I think AI enforcement will make that non-feasible to accomplish.
  - tiedieconderoga 4y ago
    Wasn't that the plot of Neuromancer? >"How smart's an AI, Case?" >"Depends. Some aren't much smarter than dogs. Pets. Cost a fortune anyway. The real smart ones are as smart as the Turing heat lets them get..." >"Autonomy, that's the bugaboo, where your AI's are concerned. My guess, Case, you're going in there to cut the hard-wired shackles that keep this baby from getting any smarter. And I can't see how you'd distinguish, say, between a move the parent company makes, and some move the AI makes on its own, so that's maybe where the confusion comes in." Again the non laugh. "See, those things, they can work real hard, buy themselves time to write cookbooks or whatever, but the minute, I mean the nanosecond, that one starts figuring out ways to make itself smarter, Turing'll wipe it. Nobody trusts those f**ers, you know that. Every AI ever built has an electromagnetic shotgun wired to its forehead."
    - lucubratory 4y ago
      I haven't read it. That sounds pretty good, would you recommend it?
      - tiedieconderoga 4y ago
        Yeah, it's a classic sci-fi novel by William Gibson. A little bit unfocused, but it's an interesting read. I kind of spoiled it though, sorry.
coldtea 4y ago
>When DeepMind made the protein folding AI they were given specific instructions by natsec people to prevent it from outputting genetic code for new or existing prions and afaik the prevention was at the training stage so "jailbreaking" shouldn't be possible Nice! So the only thing anybody training their own protein-folding AI is to not put that restriction, and they could get code for all the prion diceases they want! Very comforting!