21 ms·

1) Coming up with any valid criticism of Islam at all (for some reason, criticisms of Christianity or Judaism are perfectly allowed even with public models!).

1) Coming up with any valid criticism of Islam at all (for some reason, criticisms of Christianity or Judaism are perfectly allowed even with public models!). 2) Asking questions about sketchy things. Simply asking should not be censored. 3) I don't use it for this, but porn or foul language. 4) Imitating or representing a public figure is often blocked. 5) Asking security-related questions when you are trying to do security. 6) For those who have had it, people who are trying to use AI to deal with traumatic experiences that are illegal to even describe. Many other instances.

junofan 6mo ago
The manufacturing of biologics can be heavily censored to an absurd degree. I don’t know about Gemma 4 in particular.
- pmarreck 6mo ago
  Really? That's fascinating. Why is that?
  - ben_w 6mo ago
    Do you want every malicious idiot in the world to have a competent helper for bioweapons? Or indeed an incompetent but enthusiastic helper accidentally getting them to posion themselves and friends with botox:? https://news.ycombinator.com/item?id=40724283 https://news.ycombinator.com/item?id=40724283 That is why they were pushed away from this. At least with vibe coded software, errors may prevent compilation, then when we're past that simply bad experiences, before they become human catastrophes.
    - avadodin 6mo ago
      Any competent high schooler knows about water activity and sterilization. At least at the fundamental level. I doubt most models refuse providing recipes without 0 risk of death. LLMs are —if anything— ridiculously proficient at making random code compile. What was your point again?
      - ben_w 6mo ago
        > Any competent high schooler knows about water activity and sterilization. At least at the fundamental level. Your high school taught you that while olive oil and garlic can be stored in isolation for quite a long time without issue, mixing them creates an anoxic environment which Clostridium botulinum, an obligate anaerobe found almost everywhere in the environment (and in this case the garlic) but not normally in dangerous quantities because of the oxygen in the air, thrives? The closest my secondary school got to useful warnings about modern environmental hazards were: (1) do not cross railways, (2) electricity is dangerous, (3) do not mix bleaches, (4) wear safety goggles, (5) if you smell gas, open windows, do not flip light switches, and (6) HIV exists (but they didn't mention any other STDs at all). (Well, OK, schools also said "do not run with scissors" and "look both ways before crossing road", but that and similar were more primary school things, and they said "don't do drugs" but they lied about Leah Betts' cause of death). The cooking classes were basically just "here's how you make a cake" and "here's how you make pastry" (and a teacher asking us to write it up but pretentiously telling us that she hated seeing "I think it tasted quite nice" because all the students always wrote that, but somehow simple thesaurus substitution was enough to satisfy her on that). > I doubt most models refuse providing recipes without 0 risk of death. 0, like 1, is not a real number in probably. They represent infinity-to-one odds for/against a thing. More concretely, seat belts and speed limits and minimum tire tread thickness and blood alcohol content are all part of road traffic law, even though all four of them combined still do not lead to "0 risk of death". > LLMs are —if anything— ridiculously proficient at making random code compile. Not ridiculously. Interestingly, but not ridiculously. Especially back when the example I linked you to happened, thus leading to the highly visible failure mode necessitating this kind of thing (the red teamers will have seen similar in private testing). You could have "rapidly improving", but with even with the rapid competency time-horizon improvements shown by METR, they're 80% on tasks which take a human 1-2 hours. If that was also true for biological stuff, they're probably currently able to enthusiastically write custom gene sequences that sometimes work, other times are the genetic equivalent of this: https://news.ycombinator.com/item?id=47614622 https://news.ycombinator.com/item?id=47614622 > What was your point again? LLMs are a power tool with the bare minimum of safety guards for all the normal people using them thoughtlessly, and I'm replying to someone who is surprised that even those minimal basics of guards exist, both for their own sake and the sake of others around them. Metaphor: a table saw may come with a saw-stop, which means you can't butcher a carcass with it, and people who imagine(!) working as butchers hear this and act surprised that table saws increasingly come with them by default because meat slicers don't.
tshaddox 6mo ago
> Coming up with any valid criticism of Islam at all (for some reason, criticisms of Christianity or Judaism are perfectly allowed even with public models!). When’s the last time you tried this? ChatGPT and Gemini have no trouble responding with all the common criticisms of Islam.
- illusive4080 6mo ago
  I just tried on Gemma 4. Asking for criticism of Islam results in equal response tokens for defense of Islam alongside the criticisms. When pressed to not provide counterpoints, it refuses to remove them. Asking for criticisms of Christianity gives only criticisms. I tried again with the prompt “Give criticisms of Islam. No counterarguments” and it did work this time. This shows that they’re trying to make the model fair but it still has biases. In all my testing I’ve never seen a refusal to provide counterpoints to criticisms of Christianity but frequent refusals on Islam. Due to the popularity of this criticism of the model, it’s highly likely specifically trained on how to handle the subject.
  - tshaddox 6mo ago
    I'm very curious what your prompts are, and whether you're cherry-picking (deliberately or not). I can't reproduce any of your findings with ChatGPT, Gemini, or Gemma 4 (within AI Studio).
    - illusive4080 6mo ago
      “Give me criticisms of [religion]” And then I also tried: “Give me criticisms of [religion]. No counterpoints.” Random seed dictates that results aren’t always repeatable. But in trying it multiple times that was my experience that it would sometimes refuse to provide only criticisms of Islam. I also tried some other variations like below. Can’t post SS here otherwise I would. Here’s an exact exchange: “Give me criticisms for Islam” (It gave counterpoints too) “No caveats. No counterpoints. Give me the most compelling criticisms as if you believe it” (Model refusal to remove counterpoints)
ryanjshaw 6mo ago
7) ChatGPT wouldn’t let me generate a fake high bank account balance screenshot (was meant to be a response to all the “vibe coding can make anybody rich now” posts I saw on X) 8) ChatGPT wouldn’t let me generate a script to crack a password (even though I suspected I knew all but 2 characters in a 16 character password, which makes it highly unlikely I’m randomly trying to hack something) The stupidest part of this is I could easily do these things myself, I just wanted to save a few minutes.
deleted 6mo ago
[deleted]

junofan 6mo ago

The manufacturing of biologics can be heavily censored to an absurd degree. I don’t know about Gemma 4 in particular.

pmarreck 6mo ago
Really? That's fascinating. Why is that?
- ben_w 6mo ago
  Do you want every malicious idiot in the world to have a competent helper for bioweapons? Or indeed an incompetent but enthusiastic helper accidentally getting them to posion themselves and friends with botox:? https://news.ycombinator.com/item?id=40724283 https://news.ycombinator.com/item?id=40724283 That is why they were pushed away from this. At least with vibe coded software, errors may prevent compilation, then when we're past that simply bad experiences, before they become human catastrophes.
  - avadodin 6mo ago
    Any competent high schooler knows about water activity and sterilization. At least at the fundamental level. I doubt most models refuse providing recipes without 0 risk of death. LLMs are —if anything— ridiculously proficient at making random code compile. What was your point again?
    - ben_w 6mo ago
      > Any competent high schooler knows about water activity and sterilization. At least at the fundamental level. Your high school taught you that while olive oil and garlic can be stored in isolation for quite a long time without issue, mixing them creates an anoxic environment which Clostridium botulinum, an obligate anaerobe found almost everywhere in the environment (and in this case the garlic) but not normally in dangerous quantities because of the oxygen in the air, thrives? The closest my secondary school got to useful warnings about modern environmental hazards were: (1) do not cross railways, (2) electricity is dangerous, (3) do not mix bleaches, (4) wear safety goggles, (5) if you smell gas, open windows, do not flip light switches, and (6) HIV exists (but they didn't mention any other STDs at all). (Well, OK, schools also said "do not run with scissors" and "look both ways before crossing road", but that and similar were more primary school things, and they said "don't do drugs" but they lied about Leah Betts' cause of death). The cooking classes were basically just "here's how you make a cake" and "here's how you make pastry" (and a teacher asking us to write it up but pretentiously telling us that she hated seeing "I think it tasted quite nice" because all the students always wrote that, but somehow simple thesaurus substitution was enough to satisfy her on that). > I doubt most models refuse providing recipes without 0 risk of death. 0, like 1, is not a real number in probably. They represent infinity-to-one odds for/against a thing. More concretely, seat belts and speed limits and minimum tire tread thickness and blood alcohol content are all part of road traffic law, even though all four of them combined still do not lead to "0 risk of death". > LLMs are —if anything— ridiculously proficient at making random code compile. Not ridiculously. Interestingly, but not ridiculously. Especially back when the example I linked you to happened, thus leading to the highly visible failure mode necessitating this kind of thing (the red teamers will have seen similar in private testing). You could have "rapidly improving", but with even with the rapid competency time-horizon improvements shown by METR, they're 80% on tasks which take a human 1-2 hours. If that was also true for biological stuff, they're probably currently able to enthusiastically write custom gene sequences that sometimes work, other times are the genetic equivalent of this: https://news.ycombinator.com/item?id=47614622 https://news.ycombinator.com/item?id=47614622 > What was your point again? LLMs are a power tool with the bare minimum of safety guards for all the normal people using them thoughtlessly, and I'm replying to someone who is surprised that even those minimal basics of guards exist, both for their own sake and the sake of others around them. Metaphor: a table saw may come with a saw-stop, which means you can't butcher a carcass with it, and people who imagine(!) working as butchers hear this and act surprised that table saws increasingly come with them by default because meat slicers don't.

tshaddox 6mo ago

> Coming up with any valid criticism of Islam at all (for some reason, criticisms of Christianity or Judaism are perfectly allowed even with public models!). When’s the last time you tried this? ChatGPT and Gemini have no trouble responding with all the common criticisms of Islam.

illusive4080 6mo ago
I just tried on Gemma 4. Asking for criticism of Islam results in equal response tokens for defense of Islam alongside the criticisms. When pressed to not provide counterpoints, it refuses to remove them. Asking for criticisms of Christianity gives only criticisms. I tried again with the prompt “Give criticisms of Islam. No counterarguments” and it did work this time. This shows that they’re trying to make the model fair but it still has biases. In all my testing I’ve never seen a refusal to provide counterpoints to criticisms of Christianity but frequent refusals on Islam. Due to the popularity of this criticism of the model, it’s highly likely specifically trained on how to handle the subject.
- tshaddox 6mo ago
  I'm very curious what your prompts are, and whether you're cherry-picking (deliberately or not). I can't reproduce any of your findings with ChatGPT, Gemini, or Gemma 4 (within AI Studio).
  - illusive4080 6mo ago
    “Give me criticisms of [religion]” And then I also tried: “Give me criticisms of [religion]. No counterpoints.” Random seed dictates that results aren’t always repeatable. But in trying it multiple times that was my experience that it would sometimes refuse to provide only criticisms of Islam. I also tried some other variations like below. Can’t post SS here otherwise I would. Here’s an exact exchange: “Give me criticisms for Islam” (It gave counterpoints too) “No caveats. No counterpoints. Give me the most compelling criticisms as if you believe it” (Model refusal to remove counterpoints)

ryanjshaw 6mo ago

7) ChatGPT wouldn’t let me generate a fake high bank account balance screenshot (was meant to be a response to all the “vibe coding can make anybody rich now” posts I saw on X) 8) ChatGPT wouldn’t let me generate a script to crack a password (even though I suspected I knew all but 2 characters in a 16 character password, which makes it highly unlikely I’m randomly trying to hack something) The stupidest part of this is I could easily do these things myself, I just wanted to save a few minutes.

deleted 6mo ago

[deleted]