2 ms·

I think you've misunderstood me. I wasn't arguing against risk in any general sense - quite the opposite. My only position is that "and then the AGI killed eve

I think you've misunderstood me. I wasn't arguing against risk in any general sense - quite the opposite. My only position is that "and then the AGI killed everyone" is so oversimplified that it makes children's bedtime stories look sophisticated. Physical hardware is relevant because unlike computer programs results aren't instant. The supply of materials, manufacturer, and deployment are all slow processes. Opponents generally have time to react, and if AGI is an opponent then I think it's safe to assume that it also has to contend with other AGI. I am unconvinced that there should be a clearly delineated divide with all AGI on one side and all humans on the other. I agree with you about the myriad non-combat risks but again I think the prediction is one of hopeless complexity. For every problem that's inadvertently created a solution can probably be arrived at to patch things up. It's not unlike the present, just faster. Break things, fix them, but the fix breaks other things, repeat ad nauseum. Hopefully we survive the process.

ben_w 2y ago
> I think you've misunderstood me. I wasn't arguing against risk in any general sense - quite the opposite. Ah, noted. > My only position is that "and then the AGI killed everyone" is so oversimplified that it makes children's bedtime stories look sophisticated. Yes, but I don't know how they can be otherwise. Yudkowsky's chess metaphor: I can say Kasparov would beat me despite (indeed, because) I'm unable to name a single move he'd make. That said, Yudkowsky's chess doom metaphor seems to assume a jump to suddenly more competent AI — I'm p(doom) ~= 10%, because I expect an approximately continuous improvement to capabilities, and consequent risks, where an AI not quite competent enough to kill everyone, causes enough harm that we either cannot make the AI which could kill everyone (tech/industrial regression), or we decide not to (Butlerian Jihad). > Physical hardware is relevant because unlike computer programs results aren't instant. The supply of materials, manufacturer, and deployment are all slow processes. Opponents generally have time to react, and if AGI is an opponent then I think it's safe to assume that it also has to contend with other AGI. Agree completely. This is why I expect any AI to take about three years to go from it running on a high end GPU or TPU to being economically disruptive. (Also: world electricity supply is currently a limiting factor, as it's only 250 W/person). > Opponents generally have time to react, and if AGI is an opponent then I think it's safe to assume that it also has to contend with other AGI. Opponents have to notice. It is not a priori clear if an AI attacker or defender will have an advantage; even if an attacker had the opportunity to consider what is hard to detect, this may go either way, as I think someone will announce smart dust that's genuinely dust-sized before the end of the decade.

ben_w 2y ago

> I think you've misunderstood me. I wasn't arguing against risk in any general sense - quite the opposite. Ah, noted. > My only position is that "and then the AGI killed everyone" is so oversimplified that it makes children's bedtime stories look sophisticated. Yes, but I don't know how they can be otherwise. Yudkowsky's chess metaphor: I can say Kasparov would beat me despite (indeed, because) I'm unable to name a single move he'd make. That said, Yudkowsky's chess doom metaphor seems to assume a jump to suddenly more competent AI — I'm p(doom) ~= 10%, because I expect an approximately continuous improvement to capabilities, and consequent risks, where an AI not quite competent enough to kill everyone, causes enough harm that we either cannot make the AI which could kill everyone (tech/industrial regression), or we decide not to (Butlerian Jihad). > Physical hardware is relevant because unlike computer programs results aren't instant. The supply of materials, manufacturer, and deployment are all slow processes. Opponents generally have time to react, and if AGI is an opponent then I think it's safe to assume that it also has to contend with other AGI. Agree completely. This is why I expect any AI to take about three years to go from it running on a high end GPU or TPU to being economically disruptive. (Also: world electricity supply is currently a limiting factor, as it's only 250 W/person). > Opponents generally have time to react, and if AGI is an opponent then I think it's safe to assume that it also has to contend with other AGI. Opponents have to notice. It is not a priori clear if an AI attacker or defender will have an advantage; even if an attacker had the opportunity to consider what is hard to detect, this may go either way, as I think someone will announce smart dust that's genuinely dust-sized before the end of the decade.