4 ms·

Anthropic is the AI doomer / safetyism lab, and Hinton is one of the patron saints of 'rationalist' AI doomerism. AI doomerism is psychologically attractive to

by novok 5mo ago

Anthropic is the AI doomer / safetyism lab, and Hinton is one of the patron saints of 'rationalist' AI doomerism. AI doomerism is psychologically attractive to "people with autistic cognitive traits, including dichotomous (black-and-white) thinking, intolerance of uncertainty, and a tendency toward catastrophizing". They are pascal's mugging themselves, to ironically use one of their terms. It's fundamentally a cognitive distortion.

ben_w 5mo ago
I'm reminded of a comic about global warming, "What if it's a big hoax and we create a better world for nothing?": https://climateactionreserve.org/blog/2012/08/31/environmental-cartoons-by-joel-pett/ https://climateactionreserve.org/blog/2012/08/31/environment... "What if AI doom is all fear-mongering, and we create AI less prone to make up dangerous stuff or mistake buggy goals for real ones" (which is what alignment is) "for nothing?" Even if Yudkowsky is autistic, you're muddling the condition. Autistic people have a *practical* intolerance of uncertainty in the moment (everything unexpected from a noise to a missed turn can be a jump-scare in their day-to-day activities), but often they're absolutely fine with intellectual uncertainty, unconventional ideas, abstract ambiguity, nonconformity, etc. Indeed, one of Yudkowsky's main things is Bayesianism, i.e. being precise about uncertainty. Yudkowsky's reported P(doom) is somewhere around 90%, which is very much in the realm of "we might eventually be able to figure this out, *but we're not even close to ready so for the love of everything slow down so we can figure this all out*"; the book title comes from a long tradition of authors noticing you need to beat readers over the head with your point for them to notice it. Anthropic (like at least also OpenAI), appears to think they can solve the problems that Yudkowsky has found. They're a lot more optimistic than him, but they take these problems seriously. For his work on AI, Hinton got a Nobel prize in Physics, a Turing Award, the inaugural Rumelhart Prize, a Princess of Asturias Award, a VinFuture Prize, and a Queen Elizabeth Prize for Engineering. Calling him a "patron saint" of "doomerism" is like calling Paul Krugman (Nobel laureate in Economics) a patron saint of "Trump Derangement Syndrome" on the basis of what he says in his YouTube channel: a smart person's considered opinions are worth listening to even if you have not got time for the details, because you can be sure someone else has considered the details and will absolutely be responding to even an i missing a dot. A Pascal's mugging would be more like S-risk (S stands for suffering) than doom risk: https://en.wikipedia.org/wiki/Risk_of_astronomical_suffering https://en.wikipedia.org/wiki/Risk_of_astronomical_suffering
- roenxi 5mo ago
  > I'm reminded of a comic about global warming, "What if it's a big hoax and we create a better world for nothing?": https://climateactionreserve.org/blog/2012/08/31/environment https://climateactionreserve.org/blog/2012/08/31/environment... The people who've made the biggest contribution to creating a better world over the last 50 years have been the Chinese; powered largely by coal and petroleum. And in one of the most ironic results in the 21st century, they're now the leaders in solar panel production on the back of the largest investment in fossil fuel energy in global history. The comic ran into the same problem as the climate change movement in general - they proposed ideas that generally made people worse off. And if measured in terms of CO2 emissions achieved nothing except pushing wealth creation to Asia. Which, in fairness, is probably appreciated by the Asians.
  - ben_w 5mo ago
    Mm, there is that. The unfortunate comparable here is that all the people who care about making sure their AI is safe, regardless of what they mean by that, are beaten to the market by the people who don't.
  - keyringlight 5mo ago
    Something that has been largely forgotten about is that it used to be routine to see pictures of smoggy Chinese and Asian cities, this was a problem for them that they solved. I can't help thinking we can't get this kind of preventative action on any large scale, we need to have severe issues first and that's not accounting for longer term/cumulative effects.
    - leonidasrup 5mo ago
      "Over the past years, the government has implemented various methods to improve the air quality in Northern China. Sandstorms, which were quite common 15 years ago, are now rarely seen in Beijing’s spring thanks to afforestation projects on China’s northern borders. The license-plate lottery system was introduced in Beijing to restrict the growth of private vehicles. Large trucks were not allowed to enter certain areas in Beijing. Above all, the coal consumption in Beijing has been restricted by shutting down industrial sites and improving heating systems. Beijing’s efforts to improve air quality has also been highly praised by the UN as a successful model for other cities. However, there is also criticism pointing out that the improvement of Beijing’s air quality is based on the sacrifice of surrounding provinces (including Hebei), as many factories were moved from Beijing to other regions." https://www.statista.com/statistics/690823/china-annual-pm25-particle-levels-beijing/ https://www.statista.com/statistics/690823/china-annual-pm25... CO2 emissions are a different kind of "pollution". They are not visible and diffuse quickly over the whole Earth.

by novok 5mo ago

ben_w 5mo ago

I'm reminded of a comic about global warming, "What if it's a big hoax and we create a better world for nothing?": https://climateactionreserve.org/blog/2012/08/31/environmental-cartoons-by-joel-pett/ https://climateactionreserve.org/blog/2012/08/31/environment... "What if AI doom is all fear-mongering, and we create AI less prone to make up dangerous stuff or mistake buggy goals for real ones" (which is what alignment is) "for nothing?" Even if Yudkowsky is autistic, you're muddling the condition. Autistic people have a *practical* intolerance of uncertainty in the moment (everything unexpected from a noise to a missed turn can be a jump-scare in their day-to-day activities), but often they're absolutely fine with intellectual uncertainty, unconventional ideas, abstract ambiguity, nonconformity, etc. Indeed, one of Yudkowsky's main things is Bayesianism, i.e. being precise about uncertainty. Yudkowsky's reported P(doom) is somewhere around 90%, which is very much in the realm of "we might eventually be able to figure this out, *but we're not even close to ready so for the love of everything slow down so we can figure this all out*"; the book title comes from a long tradition of authors noticing you need to beat readers over the head with your point for them to notice it. Anthropic (like at least also OpenAI), appears to think they can solve the problems that Yudkowsky has found. They're a lot more optimistic than him, but they take these problems seriously. For his work on AI, Hinton got a Nobel prize in Physics, a Turing Award, the inaugural Rumelhart Prize, a Princess of Asturias Award, a VinFuture Prize, and a Queen Elizabeth Prize for Engineering. Calling him a "patron saint" of "doomerism" is like calling Paul Krugman (Nobel laureate in Economics) a patron saint of "Trump Derangement Syndrome" on the basis of what he says in his YouTube channel: a smart person's considered opinions are worth listening to even if you have not got time for the details, because you can be sure someone else has considered the details and will absolutely be responding to even an i missing a dot. A Pascal's mugging would be more like S-risk (S stands for suffering) than doom risk: https://en.wikipedia.org/wiki/Risk_of_astronomical_suffering https://en.wikipedia.org/wiki/Risk_of_astronomical_suffering

roenxi 5mo ago
> I'm reminded of a comic about global warming, "What if it's a big hoax and we create a better world for nothing?": https://climateactionreserve.org/blog/2012/08/31/environment https://climateactionreserve.org/blog/2012/08/31/environment... The people who've made the biggest contribution to creating a better world over the last 50 years have been the Chinese; powered largely by coal and petroleum. And in one of the most ironic results in the 21st century, they're now the leaders in solar panel production on the back of the largest investment in fossil fuel energy in global history. The comic ran into the same problem as the climate change movement in general - they proposed ideas that generally made people worse off. And if measured in terms of CO2 emissions achieved nothing except pushing wealth creation to Asia. Which, in fairness, is probably appreciated by the Asians.
- ben_w 5mo ago
  Mm, there is that. The unfortunate comparable here is that all the people who care about making sure their AI is safe, regardless of what they mean by that, are beaten to the market by the people who don't.
- keyringlight 5mo ago
  Something that has been largely forgotten about is that it used to be routine to see pictures of smoggy Chinese and Asian cities, this was a problem for them that they solved. I can't help thinking we can't get this kind of preventative action on any large scale, we need to have severe issues first and that's not accounting for longer term/cumulative effects.
  - leonidasrup 5mo ago
    "Over the past years, the government has implemented various methods to improve the air quality in Northern China. Sandstorms, which were quite common 15 years ago, are now rarely seen in Beijing’s spring thanks to afforestation projects on China’s northern borders. The license-plate lottery system was introduced in Beijing to restrict the growth of private vehicles. Large trucks were not allowed to enter certain areas in Beijing. Above all, the coal consumption in Beijing has been restricted by shutting down industrial sites and improving heating systems. Beijing’s efforts to improve air quality has also been highly praised by the UN as a successful model for other cities. However, there is also criticism pointing out that the improvement of Beijing’s air quality is based on the sacrifice of surrounding provinces (including Hebei), as many factories were moved from Beijing to other regions." https://www.statista.com/statistics/690823/china-annual-pm25-particle-levels-beijing/ https://www.statista.com/statistics/690823/china-annual-pm25... CO2 emissions are a different kind of "pollution". They are not visible and diffuse quickly over the whole Earth.