3 ms·

> We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them. Mostly if the "obviousness"

by patrec 3y ago

> We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them. Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you? > The problem is, lots of "obvious" things have turned out to be very wrong. A much bigger problem is that lots more "counter-intuitive" things that people like to believe because they elevate them over the unwashed masses have turned and continue to turn out to be very wrong and that this does not prevent them from forming the basis for important policy decisions. I'm all for questioning even what appears intuitively obvious (especially if much rides on getting it right, as presumably it does here). But frankly, of the many bizarre reasons I have heard why we should not worry about AI the claim that it seems far too obvious that we should must be the single most perverse one yet. > Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious. Maybe your appraisal of what counts as evidence is defective? For example, there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans. And with AIs often not just reaching but far surpassing human ability. I'm happy to provide examples. Can you explain to me why you think this is does not count, in any way, as evidence that AIs have the potential to reach a level of capability that renders them quite dangerous?

revelio 3y ago
> Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you? The social taboo here is saying that a position taken by lots of highly educated people is nonsense because they're all locked in a dumb purity spiral that leads to motivated reasoning. This is actually one of societies biggest taboos! Look at what happens to people who make that argument publicly under their own name in other contexts; they tend to get fired and cancelled really fast. > there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans That sword cuts both ways! There have been lots of predictions in the last decade that AI will contribute novel and hithertofore unknown solutions to things like climate change or curing cancer. Try getting GPT-4 to spit out a novel research-quality solution to anything, even a simple product design problem, and you'll find it can't. > the claim that it seems far too obvious that we should They're not arguing that. They're saying that AI risk proponents don't actually have good arguments, which is why they so regularly fall back on "it's so obvious we shouldn't need to explain why it's important". If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.
- patrec 3y ago
  OK, I completely agree that if you feel that I invoked "obviousness" in an attempt of browbeating you and the GP with what in fact is a social taboo, you should be extra skeptical (I'm not sure that was the point the GP was trying to make though). > If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong. It doesn't though, does it? There's strong empirical evidence that AI systems are making rapid progress in many domains that previously only humans were good at, and a pace that basically surprised almost everyone. I gave a list of arguments in another thread why AI is uniquely powerful and dangerous. Which of these do you disagree with and why?
  - revelio 3y ago
    I didn't see your other post I think, but here's my response to the list of AI risks on the website we're discussing: https://news.ycombinator.com/item?id=36123082#36129011 https://news.ycombinator.com/item?id=36123082#36129011 Arguments like yours are very subjective. What is "rapid". What is "surprising". I don't find them particularly surprising myself - cool and awesome - but I was being amazed by language modelling ten years ago! The quality kept improving every year. It was clear that if that kept up eventually we'd have language models that could speak to like people. So the idea of a surprising change of pace doesn't really hold up under close inspection. LLM capabilities do seem to scale linearly, with the idea of emergent abilities coming under robust attack lately. To the extent big LLMs are surprising to a lot of people this has happened primarily due to throwing a previously implausible quantity of money at building them, and OpenAI releasing one of them from their lab prison that other companies were keeping them in, not due to any major new breakthrough in the underlying tech. The progress is linear but the visibility of that progress was not. The transformers paper was 5 years ago and GPT-4 is basically an optimization of that tech combined with RL, just executed very carefully and competently. Transformers in turn were an improvement over prior language models that could speak like a human, they just weren't as good at it. > It doesn't though, does it? It does. Arguments that consist of "everyone knows that" are also called rumours or folk wisdom. It's fine to adopt widely held beliefs if those beliefs rest on something solid, but what we have here is a pure argument from authority. This letter is literally one sentence long and the only reason anyone cares is the list of signatories. It's very reliant on the observer believing that these people have some unique insight into AI risk that nobody else has, but there's no evidence of that and many signers aren't even AI researchers to begin with.

by patrec 3y ago

revelio 3y ago

> Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you? The social taboo here is saying that a position taken by lots of highly educated people is nonsense because they're all locked in a dumb purity spiral that leads to motivated reasoning. This is actually one of societies biggest taboos! Look at what happens to people who make that argument publicly under their own name in other contexts; they tend to get fired and cancelled really fast. > there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans That sword cuts both ways! There have been lots of predictions in the last decade that AI will contribute novel and hithertofore unknown solutions to things like climate change or curing cancer. Try getting GPT-4 to spit out a novel research-quality solution to anything, even a simple product design problem, and you'll find it can't. > the claim that it seems far too obvious that we should They're not arguing that. They're saying that AI risk proponents don't actually have good arguments, which is why they so regularly fall back on "it's so obvious we shouldn't need to explain why it's important". If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.

patrec 3y ago
OK, I completely agree that if you feel that I invoked "obviousness" in an attempt of browbeating you and the GP with what in fact is a social taboo, you should be extra skeptical (I'm not sure that was the point the GP was trying to make though). > If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong. It doesn't though, does it? There's strong empirical evidence that AI systems are making rapid progress in many domains that previously only humans were good at, and a pace that basically surprised almost everyone. I gave a list of arguments in another thread why AI is uniquely powerful and dangerous. Which of these do you disagree with and why?
- revelio 3y ago
  I didn't see your other post I think, but here's my response to the list of AI risks on the website we're discussing: https://news.ycombinator.com/item?id=36123082#36129011 https://news.ycombinator.com/item?id=36123082#36129011 Arguments like yours are very subjective. What is "rapid". What is "surprising". I don't find them particularly surprising myself - cool and awesome - but I was being amazed by language modelling ten years ago! The quality kept improving every year. It was clear that if that kept up eventually we'd have language models that could speak to like people. So the idea of a surprising change of pace doesn't really hold up under close inspection. LLM capabilities do seem to scale linearly, with the idea of emergent abilities coming under robust attack lately. To the extent big LLMs are surprising to a lot of people this has happened primarily due to throwing a previously implausible quantity of money at building them, and OpenAI releasing one of them from their lab prison that other companies were keeping them in, not due to any major new breakthrough in the underlying tech. The progress is linear but the visibility of that progress was not. The transformers paper was 5 years ago and GPT-4 is basically an optimization of that tech combined with RL, just executed very carefully and competently. Transformers in turn were an improvement over prior language models that could speak like a human, they just weren't as good at it. > It doesn't though, does it? It does. Arguments that consist of "everyone knows that" are also called rumours or folk wisdom. It's fine to adopt widely held beliefs if those beliefs rest on something solid, but what we have here is a pure argument from authority. This letter is literally one sentence long and the only reason anyone cares is the list of signatories. It's very reliant on the observer believing that these people have some unique insight into AI risk that nobody else has, but there's no evidence of that and many signers aren't even AI researchers to begin with.