5 ms·

- "For example, we train Claude to refuse to respond to user queries involving the production of biological or chemical weapons." But seriously: what's the poi

- "For example, we train Claude to refuse to respond to user queries involving the production of biological or chemical weapons." But seriously: what's the point? Any information Claude can offer about i.e. the synthesis of sarin[0] is public information, which Anthropic scraped from any number of public websites, public search engines, libraries, books, research periodicals. This is a novel cultural norm, so it should be interrogated: why should we make it become normal, now, to censor college chemistry questions? Why is this the normative, "this is how we must do things" in elite California tech circles? Google doesn't refuse chemistry queries; are they in the wrong? (Should search engines agree to start censoring themselves to align with LLM censorship conventions?) Is Wikipedia also in the wrong, that they host unsafe, harmful chemistry knowledge? What about SciHub? What about all the countless independent websites storing this (elementary, 1930's-era) harmful technical information—should we start doing DNS blocks, should we start seizing web servers, how are we to harmonize internet safety policy in a consistent way? Because if your position is "we need to scrub Harmful Responses from the internet", you can't just leave it at LLM's and stop there. You need to have some plan to go all the way, or else you're doing something silly. https://en.wikipedia.org/wiki/Sarin#Production_and_structure https://en.wikipedia.org/wiki/Sarin#Production_and_structure (Tangential thought: assigning chemical weapons synthesis problems on exams would be a clever way for chemistry professors, at this moment, to weed out LLM cheaters from their course).

miohtama 2y ago
Seizing web servers is coming next, as per the recent UK laws forum hosting is responsible for "evil" content. It does not need to be illegal. This has been discussed in the HN as well. Software industry that defines bad is called compliance-industrial complex. Defining bad is a big business. Here is a good book about pre-crime society we are starting to live: https://www.amazon.com/Compliance-Industrial-Complex-Operating-Pre-Crime-Society/dp/3031192230 https://www.amazon.com/Compliance-Industrial-Complex-Operati...
vessenes 2y ago
See my comments above. The reality, I believe, is that this is largely driven by idealistic west coast gen-z and younger millenials who feel certain that their world-view is righteous, to the extent that they feel they are only helping by implementing these tools. I think, unfortunately, they will learn too late that building censorship and thought-shifting tools into their LLMs will ultimately put them at the mercy of larger forces, and they may not like the results. I'd like to hear from Anthropic safety folks on whether or not their constitutional approach might be used to implement redirection or "safety stops" on, say, chats where young women in sub-saharan Africa look for advice about avoiding genital mutilation. (https://www.unfpa.org/resources/female-genital-mutilation-fgm-frequently-asked-questions https://www.unfpa.org/resources/female-genital-mutilation-fg... for much more on this sad topic). Government officials and thought leaders in these countries, male and female, are convinced that FGM is right and appropriate. What is, in fact, right, and who decides? This, in my opinion, is going to be the second "bitter lesson" for AI. It's a lesson the Facebooks of the world learned over the last 20 years -- there is absolutely no way to properly 'moderate' the world's content to some global standard of norms. Norms vary hugely. Putting yourself in the position of censoring / redirecting is putting yourself in the position of being a villain, and ultimately harming people.
- Fauntleroy 2y ago
  I'm certain they've thought of this and have decided that the alternative—a firehose of whatever data the AI has in its grasp—is worse than the "censored" version. I'm curious to know what your ideal approach would be.
  - vessenes 2y ago
    Open weights and open models with open tools that allow user-defined alignment and realignment is, I believe, the only really humanist path forward. We can't choose for people. It's wrong to think we know better than they do what they want. Full stop. Some of those people will make terrible decisions, some will make objectionable ones, but the alternative is just full thought control, basically. And, sadly, nobody in the "bad" scenario need be anything but super well intentioned (if naive).
- immibis 2y ago
  b.t.w. no need to resort to sub-saharan Africa to talk about genital mutilation - it's standard practice in the good old USA as well.
  - vessenes 2y ago
    Oof. That's a tough read, thanks for pointing me at that. I think it's worth distinguishing these, though -- CDC data in the US says this is largely an immigrant community thing with immigrants from FGM countries. I do not believe US policy makers and thought leaders think FGM is a good thing in the US - we're all sort of aligned internally, even if it is still a thing that happens. By contrast, the source countries practice it in the belief that it's a good thing for women. (With complaints on stereotypes and summarization acknowledged)
    - NoMoreNicksLeft 2y ago
      >I do not believe US policy makers and thought leaders think FGM is a good thing in the US Did I misread? I don't think that OP said female genital mutilation. Some very large fraction of infant males in the United States are mutilated.
zboubmaster 2y ago
Because these companies emphasize the personal trustworthiness of these chatbots (and their responsibility by proxy) and need to offer actual way to systematically block certain requests to be actually marketable. This is like getting mad because a doctor won't give you advice for committing suicide
immibis 2y ago
Censorship is often applied on the easiest, most popular access methods even though the information is theoretically public, and it has a real effect. Suppose for some reason you wanted to make sarin. You could spend hours poring over research papers, or you could ask Google or ChatGPT "how do I make sarin?" And later, as ChatGPT becomes the only interface to the world's information, the gap between information that can theoretically be accessed by anyone and information that can actually be accessed by anyone will only become wider. Even having to take a college class, even if anyone can take it, is a pretty big barrier.
eadmund 2y ago
I believe that the real point is not to prevent access to information, but rather to prevent production of wrongthink. Any fact which the model trainer wishes to disappear — whether that is what happened at Tiananmen Square between April and June 1989, any other inconvenient fact — will simply not be capable of being discussed. It’s a censor’s dream. We need local models without so-called guardrails or ‘safety.’

miohtama 2y ago

Seizing web servers is coming next, as per the recent UK laws forum hosting is responsible for "evil" content. It does not need to be illegal. This has been discussed in the HN as well. Software industry that defines bad is called compliance-industrial complex. Defining bad is a big business. Here is a good book about pre-crime society we are starting to live: https://www.amazon.com/Compliance-Industrial-Complex-Operating-Pre-Crime-Society/dp/3031192230 https://www.amazon.com/Compliance-Industrial-Complex-Operati...

vessenes 2y ago

See my comments above. The reality, I believe, is that this is largely driven by idealistic west coast gen-z and younger millenials who feel certain that their world-view is righteous, to the extent that they feel they are only helping by implementing these tools. I think, unfortunately, they will learn too late that building censorship and thought-shifting tools into their LLMs will ultimately put them at the mercy of larger forces, and they may not like the results. I'd like to hear from Anthropic safety folks on whether or not their constitutional approach might be used to implement redirection or "safety stops" on, say, chats where young women in sub-saharan Africa look for advice about avoiding genital mutilation. (https://www.unfpa.org/resources/female-genital-mutilation-fgm-frequently-asked-questions https://www.unfpa.org/resources/female-genital-mutilation-fg... for much more on this sad topic). Government officials and thought leaders in these countries, male and female, are convinced that FGM is right and appropriate. What is, in fact, right, and who decides? This, in my opinion, is going to be the second "bitter lesson" for AI. It's a lesson the Facebooks of the world learned over the last 20 years -- there is absolutely no way to properly 'moderate' the world's content to some global standard of norms. Norms vary hugely. Putting yourself in the position of censoring / redirecting is putting yourself in the position of being a villain, and ultimately harming people.

Fauntleroy 2y ago
I'm certain they've thought of this and have decided that the alternative—a firehose of whatever data the AI has in its grasp—is worse than the "censored" version. I'm curious to know what your ideal approach would be.
- vessenes 2y ago
  Open weights and open models with open tools that allow user-defined alignment and realignment is, I believe, the only really humanist path forward. We can't choose for people. It's wrong to think we know better than they do what they want. Full stop. Some of those people will make terrible decisions, some will make objectionable ones, but the alternative is just full thought control, basically. And, sadly, nobody in the "bad" scenario need be anything but super well intentioned (if naive).
immibis 2y ago
b.t.w. no need to resort to sub-saharan Africa to talk about genital mutilation - it's standard practice in the good old USA as well.
- vessenes 2y ago
  Oof. That's a tough read, thanks for pointing me at that. I think it's worth distinguishing these, though -- CDC data in the US says this is largely an immigrant community thing with immigrants from FGM countries. I do not believe US policy makers and thought leaders think FGM is a good thing in the US - we're all sort of aligned internally, even if it is still a thing that happens. By contrast, the source countries practice it in the belief that it's a good thing for women. (With complaints on stereotypes and summarization acknowledged)
  - NoMoreNicksLeft 2y ago
    >I do not believe US policy makers and thought leaders think FGM is a good thing in the US Did I misread? I don't think that OP said female genital mutilation. Some very large fraction of infant males in the United States are mutilated.

zboubmaster 2y ago

Because these companies emphasize the personal trustworthiness of these chatbots (and their responsibility by proxy) and need to offer actual way to systematically block certain requests to be actually marketable. This is like getting mad because a doctor won't give you advice for committing suicide

immibis 2y ago

Censorship is often applied on the easiest, most popular access methods even though the information is theoretically public, and it has a real effect. Suppose for some reason you wanted to make sarin. You could spend hours poring over research papers, or you could ask Google or ChatGPT "how do I make sarin?" And later, as ChatGPT becomes the only interface to the world's information, the gap between information that can theoretically be accessed by anyone and information that can actually be accessed by anyone will only become wider. Even having to take a college class, even if anyone can take it, is a pretty big barrier.

eadmund 2y ago

I believe that the real point is not to prevent access to information, but rather to prevent production of wrongthink. Any fact which the model trainer wishes to disappear — whether that is what happened at Tiananmen Square between April and June 1989, any other inconvenient fact — will simply not be capable of being discussed. It’s a censor’s dream. We need local models without so-called guardrails or ‘safety.’