4 ms·

As others have pointed out, there are many on this list (Bruce Schneier, for example) who do not stand to benefit from AI marketing or regulatory capture. Anyo

As others have pointed out, there are many on this list (Bruce Schneier, for example) who do not stand to benefit from AI marketing or regulatory capture. Anyone upvoting this comment should take a long look at the names on this letter and realize that many are not conflicted. Many signers of this letter are more politically sophisticated than the average HN commenter, also. So sure, maybe they're getting rolled by marketers. But also, maybe you're getting rolled by suspicion or bias against the claim they're making.

veerd 3y ago
Agreed. It's difficult for me to see how the regulatory capture arguments apply to Geoffrey Hinton and Yoshua Bengio (!!). Both of them are criticizing their own life's work and the source of their prestige. That has to be emotionally painful. They aren't doing it for fun. I totally understand not agreeing with AI x-risk concerns on an object level, but I find the casual dismissal bizarre.
- verdverm 3y ago
  Hinton has invested in multiple AI companies: https://www.crunchbase.com/person/geoffrey-hinton https://www.crunchbase.com/person/geoffrey-hinton
- logicchains 3y ago
  [flagged]
holoduke 3y ago
No. Its pretty obvious what is happening. The openai statements are pure self interest based. Nothing ethical. They lost that not a long time ago. And Sam Altman? He sold his soul to the devil. He is a lying sob.
- esafak 3y ago
  This is not an OpenAI statement.
RandomLensman 3y ago
Who on the list is an expert on existential risk (and perhaps even beyond academia)?
- staunton 3y ago
  Who in the world is an expert on existential risk? It's kind of hard to have empirically tested knowledge about that sort of thing.
  - RandomLensman 3y ago
    Pretty sure there are people looking into nuclear deterrence, bioterrorism defense, planetary defense etc. (We didn't have a nuclear war or some bioweapon killing everyone, for example, despite warnings). There are people studying how previous societies got into existential risk situations, too. We also have a huge amount of socio-economic modelling going into climate change, for example. So I'd say there should be quite a few around.
- holmesworcester 3y ago
  Signatory Jan Taallin has founded an x-risk organization focused on AI, biotech, nuclear weapons, and climate change: https://futureoflife.org/ https://futureoflife.org/
- veerd 3y ago
  Most people that study AI existential risk specifically are studying it due to concerns about AI x-risk. So the list of relevant AI x-risk experts will be subject to massive selection effects. If instead you want to consider the highest status/most famous people working on AI in general, then the list of signatories here is a pretty good summary. From my flawed perspective as a casual AI enthusiast, Yann LeCun and Jürgen Schmidhuber are the most glaring omissions (and both have publicly stated their lack of concern about AI x-risk). Of course, the highest status people aren't necessarily the most relevant people. Unfortunately, it's more difficult for me to judge relevance than fame.
verdverm 3y ago
Pretty much anyone can sign it, also notable people like Grimes, not sure why her signature carries weight on this
- arisAlexis 3y ago
  She knew who Rokko was before you. Seriously, there are some people that have been thinking about this stuff for many years.
evrydayhustling 3y ago
> Anyone upvoting this comment should take a long look at the names on this letter and realize that many are not conflicted. The concern is that the most informed names, and those spearheading the publicity around these letters, are the most conflicted. Also, you can't scan bio lines for the affiliations that impact this kind of statement. I'm not disputing that there are honest reasons for concern, but besides job titles there are sponsorships, friendships, self publicity, and a hundred other reasons for smart, "politically sophisticated" people to look the other way on the fact that this statement will be used as a lobbying tool. Almost everyone, certainly including myself, can agree that there should be active dialog about AI dangers. The dialog is happening! But by failing to make specifics or suggestions (in order to widen the tentpole and avoid the embarrassment of the last letter), they have produced an artifact of generalized fear, which can and will be used by opportunists of all stripes. Signatories should consider that they are empowering SOMEBODY, but most will have little say in who that is.
AlexandrB 3y ago
I definitely agree that names like Hinton, Schneier, and Norvig add a lot of weight here. The involvement of OpenAI muddies the water a lot though and it's not at all clear what is meant by "risk of extinction". It sounds scary, but what's the mechanism? The safe.ai website lists 8 risks, but these are quite vague as well, with many alluding to disruption of social order as the primary harm. If safe.ai knows something we don't, I wish they could communicate it more clearly. I also find it somewhat telling that something like "massive wealth disparity" or "massive unemployment" are not on the list, when this is a surefire way to create a highly unstable society and a far more immediate risk than AI going rogue. Risk #5 (below) sort of alludes to it, but misses the mark by pointing towards a hypothetical "regime" instead of companies like OpenAI. > Value Lock-In > Highly competent systems could give small groups of people a tremendous amount of power, leading to a lock-in of oppressive systems. > AI imbued with particular values may determine the values that are propagated into the future. Some argue that the exponentially increasing compute and data barriers to entry make AI a centralizing force. As time progresses, the most powerful AI systems may be designed by and available to fewer and fewer stakeholders. This may enable, for instance, regimes to enforce narrow values through pervasive surveillance and oppressive censorship. Overcoming such a regime could be unlikely, especially if we come to depend on it. Even if creators of these systems know their systems are self-serving or harmful to others, they may have incentives to reinforce their power and avoid distributing control.

veerd 3y ago

Agreed. It's difficult for me to see how the regulatory capture arguments apply to Geoffrey Hinton and Yoshua Bengio (!!). Both of them are criticizing their own life's work and the source of their prestige. That has to be emotionally painful. They aren't doing it for fun. I totally understand not agreeing with AI x-risk concerns on an object level, but I find the casual dismissal bizarre.

verdverm 3y ago
Hinton has invested in multiple AI companies: https://www.crunchbase.com/person/geoffrey-hinton https://www.crunchbase.com/person/geoffrey-hinton
logicchains 3y ago
[flagged]

holoduke 3y ago

No. Its pretty obvious what is happening. The openai statements are pure self interest based. Nothing ethical. They lost that not a long time ago. And Sam Altman? He sold his soul to the devil. He is a lying sob.

esafak 3y ago
This is not an OpenAI statement.

RandomLensman 3y ago

Who on the list is an expert on existential risk (and perhaps even beyond academia)?

staunton 3y ago
Who in the world is an expert on existential risk? It's kind of hard to have empirically tested knowledge about that sort of thing.
- RandomLensman 3y ago
  Pretty sure there are people looking into nuclear deterrence, bioterrorism defense, planetary defense etc. (We didn't have a nuclear war or some bioweapon killing everyone, for example, despite warnings). There are people studying how previous societies got into existential risk situations, too. We also have a huge amount of socio-economic modelling going into climate change, for example. So I'd say there should be quite a few around.
holmesworcester 3y ago
Signatory Jan Taallin has founded an x-risk organization focused on AI, biotech, nuclear weapons, and climate change: https://futureoflife.org/ https://futureoflife.org/
veerd 3y ago
Most people that study AI existential risk specifically are studying it due to concerns about AI x-risk. So the list of relevant AI x-risk experts will be subject to massive selection effects. If instead you want to consider the highest status/most famous people working on AI in general, then the list of signatories here is a pretty good summary. From my flawed perspective as a casual AI enthusiast, Yann LeCun and Jürgen Schmidhuber are the most glaring omissions (and both have publicly stated their lack of concern about AI x-risk). Of course, the highest status people aren't necessarily the most relevant people. Unfortunately, it's more difficult for me to judge relevance than fame.

verdverm 3y ago

Pretty much anyone can sign it, also notable people like Grimes, not sure why her signature carries weight on this

arisAlexis 3y ago
She knew who Rokko was before you. Seriously, there are some people that have been thinking about this stuff for many years.

evrydayhustling 3y ago

> Anyone upvoting this comment should take a long look at the names on this letter and realize that many are not conflicted. The concern is that the most informed names, and those spearheading the publicity around these letters, are the most conflicted. Also, you can't scan bio lines for the affiliations that impact this kind of statement. I'm not disputing that there are honest reasons for concern, but besides job titles there are sponsorships, friendships, self publicity, and a hundred other reasons for smart, "politically sophisticated" people to look the other way on the fact that this statement will be used as a lobbying tool. Almost everyone, certainly including myself, can agree that there should be active dialog about AI dangers. The dialog is happening! But by failing to make specifics or suggestions (in order to widen the tentpole and avoid the embarrassment of the last letter), they have produced an artifact of generalized fear, which can and will be used by opportunists of all stripes. Signatories should consider that they are empowering SOMEBODY, but most will have little say in who that is.

AlexandrB 3y ago

I definitely agree that names like Hinton, Schneier, and Norvig add a lot of weight here. The involvement of OpenAI muddies the water a lot though and it's not at all clear what is meant by "risk of extinction". It sounds scary, but what's the mechanism? The safe.ai website lists 8 risks, but these are quite vague as well, with many alluding to disruption of social order as the primary harm. If safe.ai knows something we don't, I wish they could communicate it more clearly. I also find it somewhat telling that something like "massive wealth disparity" or "massive unemployment" are not on the list, when this is a surefire way to create a highly unstable society and a far more immediate risk than AI going rogue. Risk #5 (below) sort of alludes to it, but misses the mark by pointing towards a hypothetical "regime" instead of companies like OpenAI. > Value Lock-In > Highly competent systems could give small groups of people a tremendous amount of power, leading to a lock-in of oppressive systems. > AI imbued with particular values may determine the values that are propagated into the future. Some argue that the exponentially increasing compute and data barriers to entry make AI a centralizing force. As time progresses, the most powerful AI systems may be designed by and available to fewer and fewer stakeholders. This may enable, for instance, regimes to enforce narrow values through pervasive surveillance and oppressive censorship. Overcoming such a regime could be unlikely, especially if we come to depend on it. Even if creators of these systems know their systems are self-serving or harmful to others, they may have incentives to reinforce their power and avoid distributing control.