3 ms·

> Wasn't that point already refuted by the paperclip maximizer thought experiment long before LLMs became a thing? I mean, no, because Bostrom hypothesized tha

> Wasn't that point already refuted by the paperclip maximizer thought experiment long before LLMs became a thing? I mean, no, because Bostrom hypothesized that an AI given the task of maximizing paperclips would consume all life in its quest to produce paperclips but there's just so many assumptions in this chain of thought, it's not credible (to me). Here's one portion of Bostrom's quote: > The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. So Bostrom imagines the AI realizing? Ok, hold on there chief, there's about a few hundred priors you haven't addressed. Better off without humans because humans might switch it off? So now we're talking about human feelings here. Emotions. Fear. There's my other comment down below that addresses that. As long as we keep projecting our fears and emotions onto transformers we are no further along to understanding the extinction risk of LLMs than soothsayers rooting around in the entrails of dead animals are to understanding the outcome of battles.

moritzwarhier 3y ago
I see your point, but the thought experiment still claims that AI risk does not require consciousness, "malicious intent" or a "breakout" of the machine in some sense, no? All of the PCM scenario - including the part you quoted - are described as a means to the ulterior goal (paperclips). Self-preservance just happens to become a logical part of if. The preconditions for an existential risks are not explained by the thought experiment, it just says that there's no need to assume "consious decisions" or consciousness for it, by providing a counterexample. So you might disagree with this, but the PCM thought experiment does not assume pain, feelings etc in AI. Of course this is only a thought experiment, and it assumes literal AGI. I'm also not arguing that LLMs are AGI. What I wanted to point out is already mentioned by many others: existential risk (or great harm) doesn't need a suprising "conscious decision", or consciousness. I mean, machines can cause unpredictable harm already if not operated correctly, all without AI.

moritzwarhier 3y ago

I see your point, but the thought experiment still claims that AI risk does not require consciousness, "malicious intent" or a "breakout" of the machine in some sense, no? All of the PCM scenario - including the part you quoted - are described as a means to the ulterior goal (paperclips). Self-preservance just happens to become a logical part of if. The preconditions for an existential risks are not explained by the thought experiment, it just says that there's no need to assume "consious decisions" or consciousness for it, by providing a counterexample. So you might disagree with this, but the PCM thought experiment does not assume pain, feelings etc in AI. Of course this is only a thought experiment, and it assumes literal AGI. I'm also not arguing that LLMs are AGI. What I wanted to point out is already mentioned by many others: existential risk (or great harm) doesn't need a suprising "conscious decision", or consciousness. I mean, machines can cause unpredictable harm already if not operated correctly, all without AI.