5 ms·

NIST staffers revolt against expected appointment of Paul Cristiano to NIST

48 points by joe_the_user 3y ago

h2odragon 3y ago
[flagged]
- joe_the_user 3y ago
  Apparently NIST employees dislike Christiano's beliefs and associations. It doesn't seem justified to label their or Christiano's beliefs as "religious". If any disagreement with a common position is "religious war", then all such disagreements are arbitrary and religion itself is just as arbitrary.
  - lmeyerov 3y ago
    There are immediate cybercrime etc activities happening around AI -- eg, content abuse is rampant rn -- so someone advising on active standards that explicitly doesn't (relatively) care about these, especially at the federal standards leadership level, is insane. That's the job! Internally to gov, depts deploying AI are terrible at data security even before LLMs hoover & regurgitate everything. Most senior leaders don't understand this stuff, like why secure RAG indexing is hard. (Some are sharp, just too few.) That's even before other basics like ai vs human decision making for diff tasks. Getting a future-ethics person asleep at the wheel on the on-fire problems right as deployments & ATOs are getting figured out in every agency.. ouch.
    - mcguire 3y ago
      It is a great way to prevent those currently deploying and using AI from being impeded by any icky current concerns.
      - lmeyerov 3y ago
        yeah that's beyond my paygrade :) At least for our work, it just takes 1 hat out of the ring for those that we would be otherwise paying attention to, but afaict doesn't really change much otherwise. There are parallel efforts across the DOD etc to set AI compliance standards for internal policies. Likewise, as usual, legal innovators at the state level are also doing legislation to govern commerce, like the recent California thing. Precedent will still get set... just not by NIST? Ex: Sounds like this person would not be focused on basic efforts to update practical aspects of NIST SP 800-53 for telling teams how it is safe to do AI in the cloud. So either teams will default to NO -- we've seen this as a common reaction in different govs + banks -- or need to point to something like the IC/DOD AI policies. Or it'll continue business as usual for the folks doing the work.
  - kragen 3y ago
    the particular positions christiano is being criticized for are positions on questions such as the following: - what are the correct standards of right and wrong, if any exist? - what is the ultimate purpose of human existence? - what, if anything, will bring about the extinguishment of the human race? - what sorts of immaterial more-than-human consciousnesses exist, or can exist, and what are the relationships between them and humanity? it doesn't seem especially controversial or metaphorical to me to describe beliefs about such topics as 'religious', particularly when they are coupled with associations organized around those beliefs. buddhism is routinely described as a 'religion' even though it doesn't involve the worship of any deities and is often described by buddhists as entailing atheism
    - burkaman 3y ago
      Where did that list come from? I don't see anything like that in the article.
      - kragen 3y ago
        it came from reading about effective altruism and longtermism since before they got those names
        burkaman 3y ago
        Ok I see, I interpreted the phrase "association with" to be more about the people he's associating with than the beliefs, but the article is incredibly vague so there's no way for us to know what these criticisms actually are.
        kragen 3y ago
        the criticisms are that other people have different positions on those questions; it should hardly be surprising that people with conflicting positions on questions like the correct standards of right and wrong will criticize one another oh and also some bad people are effective altruists, which is something everyone agrees on, including the other effective altruists
artninja1988 3y ago
Related to the EA cult: AI doomsayers funded by billionaires ramp up lobbying https://www.politico.com/news/2024/02/23/ai-safety-washington-lobbying-00142783 https://www.politico.com/news/2024/02/23/ai-safety-washingto...
burkaman 3y ago
It seems concerning that he would be in charge of determining AI policy when he was a high-level OpenAI employee for 4 years and presumably has a huge financial interest in their success (https://www.levels.fyi/blog/openai-compensation.html https://www.levels.fyi/blog/openai-compensation.html).
- mitthrowaway2 3y ago
  Maybe; hopefully he can sell whatever stake he has in OpenAI to avoid conflict-of-interest concerns. Personally I can hardly think of any better person for the role.
  - burkaman 3y ago
    That would be great, but I don't think federal appointees are ever forced or even asked to divest anything. At most he'd just have to disclose the conflict of interest.
  - capital_guy 3y ago
    I can think of an entire body of people who should be excluded from such a role; people who spent multiple years working at OpenAI.
- olliej 3y ago
  Yeah being appointed to such a body should come with a mandatory “you must completely divest from the industry you will be regulating, include and aggregate fund or similar”, alongside “any compensation, hiring, or rehiring contracts and agreements offered prior to or during tenure are explicitly invalid”, and a “you cannot work on the board or executive of any business you I. The relevant industries for five years post departure, or at or with any previous employers for 10 years”
  - rokkitmensch 3y ago
    A policy as likely to stick in our marvelous environment of endemic corruption as banning trading by members of the legislative branch and their families. "Rules for thee, not for me."
jll29 3y ago
I had not heard about that movement, but here is their own introduction: https://www.effectivealtruism.org/articles/introduction-to-effective-altruism https://www.effectivealtruism.org/articles/introduction-to-e... At first glance, terms like "long-term" and "altruism" resonate with me, as I care much more whether we can avoid the heat death in 500 million years than about what happens tomorrow. Also the main "doctrine" seems to be one I can agree with, namely to find "unusually effective ways" of helping so that with the same ressources, more positive impact can be achieved. (I did the same when I had to decide which charity to support - since I am not a millionaire, but experienced at research, I focused on identifying highly effective charities to avoid wasting my humble charitable contributions.) However, where things get problematic is when utilitarian ethics get adopted, a paradigm considered problematic in philosophy because it tries to quantify good in order to measure how to maximize it, and the most good to the most people can often result in outcomes that people of good will agree are unethical, namely the suppression or extermination of minorities. For example, the link above compares pandemic deaths with terrorism deaths and concludes that pandemic deaths are the big ticket numbers and in comparison terrorism deaths are small. While the numbers reported are possibly statistically correct, I would personally object if one drew the conclusion from them that a small number of deaths does not matter, or that they mattered less than larger number of casualties. Utilitarian ethics, which is often appealing to engineering minds who are used to think in terms of optimizing a variable, has issues with protecting minorities and with justice [1]. From a recent article in the WSJ, it appears that NIST have larger problems than a potentially strange leader: the building substance of the Gaithersburg campus is rotten to the core, to the extent that equipment got destroyed from dampness or water damage etc. A complete overhaul seem due, yet plans exist to cut -- rather than expand -- its funding. [1] https://www.scu.edu/ethics/ethics-resources/ethical-decision-making/calculating-consequences-the-utilitarian-approach/ https://www.scu.edu/ethics/ethics-resources/ethical-decision...
- akira2501 3y ago
  > as I care much more whether we can avoid the heat death in 500 million years than about what happens tomorrow. Isn't this just an egotistical mechanism that allows the bearer to freely justify any means they please by the supposed loftiness of their stated ends?
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
sfink 3y ago
This article would be better if it reported on the topic of the article, rather than stopping at stating the topic. Seriously, the whole thing could easily fit in 240 characters. NIST staffers are revolting against his appointment... why, exactly? Ok, it also says he's associated with EA, and EA people have done bad things. Is that what the NIST staffers are saying? What are the staffers saying, anyway? If you're going to claim to be reporting on them, maybe at least try talking to them. I got pretty much nothing from this article other than the headline. It both piqued and frustrated my curiousity. Perhaps I need to read more about how EA is being problematic in practice. The high-level description seems fine, but then again, every single religion, cult, movement, rebellion, revolution, etc. could be described as "do good stuff" at a high level (or at least "don't do bad stuff"—Don't Be Evil, anyone?) The part between "do good stuff" and "therefore we must commit genocide" is kinda important.
deleted 3y ago
[deleted]
hiddencost 3y ago
Their estimations of how likely we are to have a Dyson sphere by 2040 should be sufficient to clarify that this movement has no interest in rigor.

48 points by joe_the_user 3y ago

h2odragon 3y ago

[flagged]

joe_the_user 3y ago
Apparently NIST employees dislike Christiano's beliefs and associations. It doesn't seem justified to label their or Christiano's beliefs as "religious". If any disagreement with a common position is "religious war", then all such disagreements are arbitrary and religion itself is just as arbitrary.
- lmeyerov 3y ago
  There are immediate cybercrime etc activities happening around AI -- eg, content abuse is rampant rn -- so someone advising on active standards that explicitly doesn't (relatively) care about these, especially at the federal standards leadership level, is insane. That's the job! Internally to gov, depts deploying AI are terrible at data security even before LLMs hoover & regurgitate everything. Most senior leaders don't understand this stuff, like why secure RAG indexing is hard. (Some are sharp, just too few.) That's even before other basics like ai vs human decision making for diff tasks. Getting a future-ethics person asleep at the wheel on the on-fire problems right as deployments & ATOs are getting figured out in every agency.. ouch.
  - mcguire 3y ago
    It is a great way to prevent those currently deploying and using AI from being impeded by any icky current concerns.
    - lmeyerov 3y ago
      yeah that's beyond my paygrade :) At least for our work, it just takes 1 hat out of the ring for those that we would be otherwise paying attention to, but afaict doesn't really change much otherwise. There are parallel efforts across the DOD etc to set AI compliance standards for internal policies. Likewise, as usual, legal innovators at the state level are also doing legislation to govern commerce, like the recent California thing. Precedent will still get set... just not by NIST? Ex: Sounds like this person would not be focused on basic efforts to update practical aspects of NIST SP 800-53 for telling teams how it is safe to do AI in the cloud. So either teams will default to NO -- we've seen this as a common reaction in different govs + banks -- or need to point to something like the IC/DOD AI policies. Or it'll continue business as usual for the folks doing the work.
- kragen 3y ago
  the particular positions christiano is being criticized for are positions on questions such as the following: - what are the correct standards of right and wrong, if any exist? - what is the ultimate purpose of human existence? - what, if anything, will bring about the extinguishment of the human race? - what sorts of immaterial more-than-human consciousnesses exist, or can exist, and what are the relationships between them and humanity? it doesn't seem especially controversial or metaphorical to me to describe beliefs about such topics as 'religious', particularly when they are coupled with associations organized around those beliefs. buddhism is routinely described as a 'religion' even though it doesn't involve the worship of any deities and is often described by buddhists as entailing atheism
  - burkaman 3y ago
    Where did that list come from? I don't see anything like that in the article.
    - kragen 3y ago
      it came from reading about effective altruism and longtermism since before they got those names
      - burkaman 3y ago
        Ok I see, I interpreted the phrase "association with" to be more about the people he's associating with than the beliefs, but the article is incredibly vague so there's no way for us to know what these criticisms actually are.
        kragen 3y ago
        the criticisms are that other people have different positions on those questions; it should hardly be surprising that people with conflicting positions on questions like the correct standards of right and wrong will criticize one another oh and also some bad people are effective altruists, which is something everyone agrees on, including the other effective altruists

artninja1988 3y ago

Related to the EA cult: AI doomsayers funded by billionaires ramp up lobbying https://www.politico.com/news/2024/02/23/ai-safety-washington-lobbying-00142783 https://www.politico.com/news/2024/02/23/ai-safety-washingto...

burkaman 3y ago

It seems concerning that he would be in charge of determining AI policy when he was a high-level OpenAI employee for 4 years and presumably has a huge financial interest in their success (https://www.levels.fyi/blog/openai-compensation.html https://www.levels.fyi/blog/openai-compensation.html).

mitthrowaway2 3y ago
Maybe; hopefully he can sell whatever stake he has in OpenAI to avoid conflict-of-interest concerns. Personally I can hardly think of any better person for the role.
- burkaman 3y ago
  That would be great, but I don't think federal appointees are ever forced or even asked to divest anything. At most he'd just have to disclose the conflict of interest.
- capital_guy 3y ago
  I can think of an entire body of people who should be excluded from such a role; people who spent multiple years working at OpenAI.
olliej 3y ago
Yeah being appointed to such a body should come with a mandatory “you must completely divest from the industry you will be regulating, include and aggregate fund or similar”, alongside “any compensation, hiring, or rehiring contracts and agreements offered prior to or during tenure are explicitly invalid”, and a “you cannot work on the board or executive of any business you I. The relevant industries for five years post departure, or at or with any previous employers for 10 years”
- rokkitmensch 3y ago
  A policy as likely to stick in our marvelous environment of endemic corruption as banning trading by members of the legislative branch and their families. "Rules for thee, not for me."

jll29 3y ago

I had not heard about that movement, but here is their own introduction: https://www.effectivealtruism.org/articles/introduction-to-effective-altruism https://www.effectivealtruism.org/articles/introduction-to-e... At first glance, terms like "long-term" and "altruism" resonate with me, as I care much more whether we can avoid the heat death in 500 million years than about what happens tomorrow. Also the main "doctrine" seems to be one I can agree with, namely to find "unusually effective ways" of helping so that with the same ressources, more positive impact can be achieved. (I did the same when I had to decide which charity to support - since I am not a millionaire, but experienced at research, I focused on identifying highly effective charities to avoid wasting my humble charitable contributions.) However, where things get problematic is when utilitarian ethics get adopted, a paradigm considered problematic in philosophy because it tries to quantify good in order to measure how to maximize it, and the most good to the most people can often result in outcomes that people of good will agree are unethical, namely the suppression or extermination of minorities. For example, the link above compares pandemic deaths with terrorism deaths and concludes that pandemic deaths are the big ticket numbers and in comparison terrorism deaths are small. While the numbers reported are possibly statistically correct, I would personally object if one drew the conclusion from them that a small number of deaths does not matter, or that they mattered less than larger number of casualties. Utilitarian ethics, which is often appealing to engineering minds who are used to think in terms of optimizing a variable, has issues with protecting minorities and with justice [1]. From a recent article in the WSJ, it appears that NIST have larger problems than a potentially strange leader: the building substance of the Gaithersburg campus is rotten to the core, to the extent that equipment got destroyed from dampness or water damage etc. A complete overhaul seem due, yet plans exist to cut -- rather than expand -- its funding. [1] https://www.scu.edu/ethics/ethics-resources/ethical-decision-making/calculating-consequences-the-utilitarian-approach/ https://www.scu.edu/ethics/ethics-resources/ethical-decision...

akira2501 3y ago
> as I care much more whether we can avoid the heat death in 500 million years than about what happens tomorrow. Isn't this just an egotistical mechanism that allows the bearer to freely justify any means they please by the supposed loftiness of their stated ends?

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

sfink 3y ago

This article would be better if it reported on the topic of the article, rather than stopping at stating the topic. Seriously, the whole thing could easily fit in 240 characters. NIST staffers are revolting against his appointment... why, exactly? Ok, it also says he's associated with EA, and EA people have done bad things. Is that what the NIST staffers are saying? What are the staffers saying, anyway? If you're going to claim to be reporting on them, maybe at least try talking to them. I got pretty much nothing from this article other than the headline. It both piqued and frustrated my curiousity. Perhaps I need to read more about how EA is being problematic in practice. The high-level description seems fine, but then again, every single religion, cult, movement, rebellion, revolution, etc. could be described as "do good stuff" at a high level (or at least "don't do bad stuff"—Don't Be Evil, anyone?) The part between "do good stuff" and "therefore we must commit genocide" is kinda important.

deleted 3y ago

[deleted]

hiddencost 3y ago

Their estimations of how likely we are to have a Dyson sphere by 2040 should be sufficient to clarify that this movement has no interest in rigor.