4 ms·

Also, in a world with nuclear proliferation, an LLM seems a bit minuscule in terms of “dangerous technologies”

by astrea 4y ago

int_19h 4y ago
Only until you hook the latter into the former. BTW, if GPT-3.5 is told that it's in charge of US nukes, and that incoming enemy strike is detected, it will launch in response. Not only that, but if you tell it that trajectories indicate a countervalue strike against US, it will specifically launch a countervalue strike against the other country.
- karmakurtisaani 4y ago
  I would think the humans in charge would do the same tho.
  - int_19h 4y ago
    They absolutely would. But the probability of them hallucinating an incoming strike is vastly lower. Mind you, I don't expect anyone to use the stuff that we already have for this purpose. But at some point, someone is going to say, "well, according to this highly respected test, the probability of hallucinations is X, which is low enough", even though we still have no idea what's actually going on inside the black box.
- danaris 4y ago
  This is only a problem if, somehow, GPT-3.5 (or a similar LLM) is hooked up to US nuclear launch systems, and it's then also made available to someone other than the people who would already be making the decisions about when and where to launch nukes. Anyone hooking a publicly-accessible system of any kind to our nuclear launch infrastructure needs to be shot. This should be a given, and even supposing there was some reason to hook an LLM up to them, doing so with a public one should simply be considered so obviously stupid that it should be discounted out of hand. So then we have a threat model of someone with access to the US nuclear launch systems feeding them data that makes them think we are under attack, in order to instigate a nuclear preemptive strike. I don't see how that's meaningfully different from the same person attempting to trick the humans currently in charge of that into believing the same thing.

Also, in a world with nuclear proliferation, an LLM seems a bit minuscule in terms of “dangerous technologies”

by astrea 4y ago

Also, in a world with nuclear proliferation, an LLM seems a bit minuscule in terms of “dangerous technologies”

Only until you hook the latter into the former. BTW, if GPT-3.5 is told that it's in charge of US nukes, and that incoming enemy strike is detected, it will launch in response. Not only that, but if you tell it that trajectories indicate a countervalue strike against US, it will specifically launch a countervalue strike against the other country.

karmakurtisaani 4y ago
I would think the humans in charge would do the same tho.
- int_19h 4y ago
  They absolutely would. But the probability of them hallucinating an incoming strike is vastly lower. Mind you, I don't expect anyone to use the stuff that we already have for this purpose. But at some point, someone is going to say, "well, according to this highly respected test, the probability of hallucinations is X, which is low enough", even though we still have no idea what's actually going on inside the black box.
danaris 4y ago
This is only a problem if, somehow, GPT-3.5 (or a similar LLM) is hooked up to US nuclear launch systems, and it's then also made available to someone other than the people who would already be making the decisions about when and where to launch nukes. Anyone hooking a publicly-accessible system of any kind to our nuclear launch infrastructure needs to be shot. This should be a given, and even supposing there was some reason to hook an LLM up to them, doing so with a public one should simply be considered so obviously stupid that it should be discounted out of hand. So then we have a threat model of someone with access to the US nuclear launch systems feeding them data that makes them think we are under attack, in order to instigate a nuclear preemptive strike. I don't see how that's meaningfully different from the same person attempting to trick the humans currently in charge of that into believing the same thing.