8 ms·

The reason it's worse is basically because it's more 'safe' (not racist, etc). That of course sounds insane, and doesn't mean that safety shouldn't be strived

by chaxor 3y ago

The reason it's worse is basically because it's more 'safe' (not racist, etc). That of course sounds insane, and doesn't mean that safety shouldn't be strived for, etc - but there's an explanation as to how this occurs. It occurs because the system essentially does a latent classification of problems into 'acceptable' or 'not acceptable' to respond to. When this is done, a decent amount of information is lost regarding how to represent these latent spaces that may be completely unrelated (making nefarious materials, or spouting hate speech are now in the same 'bucket' for the decoder). This degradation was observed quite early on with the tikz unicorn benchmark, which improved with training, and then degraded when fine-tuning to be more safe was applied.

enumjorge 3y ago
Do you have any sources for that? How would making ChatGPT less likely to return a racist answer or hate speech affect its ability to return code? After a question has been classified into a coding problem, presumably ChatGPT servers could now continue to solve the problem as usual. Maybe running ChatGPT is really expensive, and they nerfed in order to reign in costs. That would explain why the answers we get are less useful, across-the-board. That may not be the reason after all, but my point is that it’s really hard to tell from the outside. There’s this narrative out there that “woke-ism” is ruining everything in tech, and I feel like some people here are being a little too eager to superimpose that narrative when we don’t really have insight into what openAI is doing.
- ushtaritk421 3y ago
  Maybe the problem is analogous to what Orwell describes here: "Even a single taboo can have an all-round crippling effect upon the mind, because there is always the danger that any thought which is freely followed up may lead to the forbidden thought." https://www.orwellfoundation.com/the-orwell-foundation/orwell/essays-and-other-works/the-prevention-of-literature/ https://www.orwellfoundation.com/the-orwell-foundation/orwel...
  - enumjorge 3y ago
    This is what I'm talking about though. The fact that you're quoting Orwell suggests that you're having an emotional response to this topic, not a logical one. We're not talking about the human mind here. ChatGPT is not a simulation of human thought. At it's core it's statistics telling you what the answer to your question ought to look like. You're applying an observation about apples to oranges.
    - qumpis 3y ago
      Why? Constraints on the reward model of LLMs restrict their generation space, so GP's quote applies
- Tostino 3y ago
  There are a lot of people who are entirely okay with the censorship but think it should be done in a different layer than the main LLM itself, as not to hurt the cognitive performance. Alignment is just fine-tuning... any type of fine tuning is possible to teach unwanted skills, and/or catastrophically forget previously learned skills. That is likely what is going on here, from what I can tell from the reading i've done into it. Most are arguing for a specific "censorship" model on the input/output of the main LLM.
- deleted 3y ago
  [deleted]
- tzekid 3y ago
  Here's the full talk[0] from a Microsoft lead researcher that worked with the early / uncensored version of GPT-4. Simplified, tuning it for censorship heavily limits the dimensionality the model can move in to find an answer which means worse results in general for some reason. [0]: https://www.youtube.com/watch?v=qbIk7-JPB2c https://www.youtube.com/watch?v=qbIk7-JPB2c
deleted 3y ago
[deleted]
geysersam 3y ago
You have no idea what you're talking about. Why would such a classification step remove any information about typical "benign" queries? It's a lot more likely they just nerfed the model because it's expensive to run.
lobocinza 3y ago
[flagged]
fhools 3y ago
There was a talk by a researcher where he was saying that they could see the progress being made on chatgpt by how much success it had with drawing a unicorn in latex. What stuck out to me was he said that the safer the model got the worst it got at drawing a unicorn.
- rob137 3y ago
  He also claimed that it initially beat 100% of humans on mock Google / Amazon coding interviews. Hard to imagine that now.
torginus 3y ago
It seems strange that safety training not pertaining to the subject matter makes the AI dumber - I suspect the safety is some kind of system prompt - It would take some context, but I'm not sure how "don't be racist" affect its binary-search writing skills negatively.
tmaly 3y ago
How soon before a competitor overtakes them because of their safety settings?
- kvetching 3y ago
  It's inevitable. When Sam asked a crowd how many people wanted an open source version of GPT-7 the moment they finished training it, and nearly everyone raised their hand. People will virtue signal, people will attempt regulatory capture, but deep down everyone wants a non-lobotomized model, and there will be thousands working to create one.
censor_me 3y ago
[dead]
mamp 3y ago
Yes, a real “Flowers for Algernon” vibe
moffkalast 3y ago
GPT 4 Lemongrab Mode: Everything is unacceptable.
weinzierl 3y ago
If I run a model like LLaMA locally would it be subject to the same restrictions? In other words is the safety baked into the model or a separate step separate from the main model?
- itsiadam 3y ago
  Both approaches are valid, but I would hope they are using a separate model to validate responses, rather than crippling the base model(s). In OpenAI's case, we don't know for sure, but it seems like a combination of both, resulting in lower quality responses overall. I imagine LLaMA was fed highly-vetted training data, as opposed to being "fixed" afterwards.
- jerpint 3y ago
  LLaMa was not fine tuned on human interactions , so it shouldn’t be subject to the same effect, but it also means it’s not nearly as good at having conversations. It’s much better at completing sentences
foreverobama 3y ago
[dead]
roenxi 3y ago
They're up against a pretty difficult barrier - if we had a perfect all-knowing oracle it might easily have opinions that are racist. Statistics alone suggest there will be racist truths. We're dealing with groups of people who are observably different from each other in correlated ways. GPT would need to reach a convincing balance of lying and honesty if it is supposed to navigate that challenge. It'd have to be deeply embedded in a particular culture to even know what 'racism' means; everyone has a different opinion.
- SpaceManNabs 3y ago
  > Statistics alone suggest there will be racist truths such as?
- buggythebug 3y ago
  Can you expand on the last sentence of your first paragraph?
  - redthrowaway 3y ago
    Crime stats, average IQ across groups, stereotype accuracy, etc. What's interesting to me is not the above, which is naughty in the anglosphere, but the question of the unknown unknowns that could be as bad or worse in other cultural contexts. There are probably enough people of Indian descent involved in GPT's development that they could guide it past some of the caste landmines, but what about a country like Turkey? We know they have massive internal divisions, but do we know what would exacerbate them and how to avoid them? What about Iran, or South Africa, or Brazil? We RLHF the piss out of LLMs to ensure they don't say things that make white college graduates in San Francisco ornery, but I'd suggest the much greater risk lies in accidentally spawning scissor statements in cultures you don't know how to begin to parse to figure out what to avoid.
    - a_cardboard_box 3y ago
      > Crime stats, average IQ across groups, stereotype accuracy, etc. If you measured these stats for Irish Americans in 1865 you'd also see high crime and low IQ. If you measure these stats with recent black immigrants from Africa, you see low crime and high IQ. These statistical differences are not caused by race. An all-knowing oracle wouldn't need to hold "opinions that are racist" to understand them.
ksherlock 3y ago
Not to get too far off topic, but that reminds me of a quote: "Unix was not designed to stop you from doing stupid things, because that would also stop you from doing clever things." -- Doug Gwyn Or maybe it's: "C is a language that doesn't get in your way. It doesn't stop you from doing dumb things, but it also doesn't stop you from doing clever things." -- Dennis Ritchie. I asked Bard for a source on those quotes and it couldn't find one for the first. Wikiquotes sources it to "Introducing Regular Expressions" by Michael Fitzgerald and that does include it as a quote but it's not the source of the quote, it's just a nice quote at the start of the chapter. For the second, Bard claims to be from a 1990 interview and is on page 21 of "The Art of Unix Programming" by Brian Kernighan and Rob Pike. There is a book called "The At of Unix Programming" (2003) but it's by Eric Raymond and I could not find the quote in the book. Pike and Kernighan have two books, "The Practice of Programming" (1999) and "The Unix Programming Environment" (1984). Neither contain that quote.
- SV_BubbleTime 3y ago
  Don’t ask an LLM objective things. Ask them subjective. They are language models, not fact models.
fallingknife 3y ago
[flagged]
- wing-_-nuts 3y ago
  It's one thing to communicate an unpopular idea in a civil manner. It's quite another to be offensive. Now, I will admit there are some people out there that cannot separate their feelings for an idea, and their feelings for the person communicating it. I can't really help that person. What I have noticed is those who express your sentiment are often looking for license to be uncivil and offensive first, and their 'ideas' are merely a tool you use to do that. That I judge. I think that's mostly what others judge too.
haolez 3y ago
I think it's more likely that they nerfed it due to scaling pains.

by chaxor 3y ago

enumjorge 3y ago

Do you have any sources for that? How would making ChatGPT less likely to return a racist answer or hate speech affect its ability to return code? After a question has been classified into a coding problem, presumably ChatGPT servers could now continue to solve the problem as usual. Maybe running ChatGPT is really expensive, and they nerfed in order to reign in costs. That would explain why the answers we get are less useful, across-the-board. That may not be the reason after all, but my point is that it’s really hard to tell from the outside. There’s this narrative out there that “woke-ism” is ruining everything in tech, and I feel like some people here are being a little too eager to superimpose that narrative when we don’t really have insight into what openAI is doing.

ushtaritk421 3y ago
Maybe the problem is analogous to what Orwell describes here: "Even a single taboo can have an all-round crippling effect upon the mind, because there is always the danger that any thought which is freely followed up may lead to the forbidden thought." https://www.orwellfoundation.com/the-orwell-foundation/orwell/essays-and-other-works/the-prevention-of-literature/ https://www.orwellfoundation.com/the-orwell-foundation/orwel...
- enumjorge 3y ago
  This is what I'm talking about though. The fact that you're quoting Orwell suggests that you're having an emotional response to this topic, not a logical one. We're not talking about the human mind here. ChatGPT is not a simulation of human thought. At it's core it's statistics telling you what the answer to your question ought to look like. You're applying an observation about apples to oranges.
  - qumpis 3y ago
    Why? Constraints on the reward model of LLMs restrict their generation space, so GP's quote applies
Tostino 3y ago
There are a lot of people who are entirely okay with the censorship but think it should be done in a different layer than the main LLM itself, as not to hurt the cognitive performance. Alignment is just fine-tuning... any type of fine tuning is possible to teach unwanted skills, and/or catastrophically forget previously learned skills. That is likely what is going on here, from what I can tell from the reading i've done into it. Most are arguing for a specific "censorship" model on the input/output of the main LLM.
deleted 3y ago
[deleted]
tzekid 3y ago
Here's the full talk[0] from a Microsoft lead researcher that worked with the early / uncensored version of GPT-4. Simplified, tuning it for censorship heavily limits the dimensionality the model can move in to find an answer which means worse results in general for some reason. [0]: https://www.youtube.com/watch?v=qbIk7-JPB2c https://www.youtube.com/watch?v=qbIk7-JPB2c

deleted 3y ago

[deleted]

geysersam 3y ago

You have no idea what you're talking about. Why would such a classification step remove any information about typical "benign" queries? It's a lot more likely they just nerfed the model because it's expensive to run.

lobocinza 3y ago

[flagged]

fhools 3y ago

There was a talk by a researcher where he was saying that they could see the progress being made on chatgpt by how much success it had with drawing a unicorn in latex. What stuck out to me was he said that the safer the model got the worst it got at drawing a unicorn.

rob137 3y ago
He also claimed that it initially beat 100% of humans on mock Google / Amazon coding interviews. Hard to imagine that now.

torginus 3y ago

It seems strange that safety training not pertaining to the subject matter makes the AI dumber - I suspect the safety is some kind of system prompt - It would take some context, but I'm not sure how "don't be racist" affect its binary-search writing skills negatively.

tmaly 3y ago

How soon before a competitor overtakes them because of their safety settings?

kvetching 3y ago
It's inevitable. When Sam asked a crowd how many people wanted an open source version of GPT-7 the moment they finished training it, and nearly everyone raised their hand. People will virtue signal, people will attempt regulatory capture, but deep down everyone wants a non-lobotomized model, and there will be thousands working to create one.

censor_me 3y ago

[dead]

mamp 3y ago

Yes, a real “Flowers for Algernon” vibe

moffkalast 3y ago

GPT 4 Lemongrab Mode: Everything is unacceptable.

weinzierl 3y ago

If I run a model like LLaMA locally would it be subject to the same restrictions? In other words is the safety baked into the model or a separate step separate from the main model?

itsiadam 3y ago
Both approaches are valid, but I would hope they are using a separate model to validate responses, rather than crippling the base model(s). In OpenAI's case, we don't know for sure, but it seems like a combination of both, resulting in lower quality responses overall. I imagine LLaMA was fed highly-vetted training data, as opposed to being "fixed" afterwards.
jerpint 3y ago
LLaMa was not fine tuned on human interactions , so it shouldn’t be subject to the same effect, but it also means it’s not nearly as good at having conversations. It’s much better at completing sentences

foreverobama 3y ago

[dead]

roenxi 3y ago

They're up against a pretty difficult barrier - if we had a perfect all-knowing oracle it might easily have opinions that are racist. Statistics alone suggest there will be racist truths. We're dealing with groups of people who are observably different from each other in correlated ways. GPT would need to reach a convincing balance of lying and honesty if it is supposed to navigate that challenge. It'd have to be deeply embedded in a particular culture to even know what 'racism' means; everyone has a different opinion.

SpaceManNabs 3y ago
> Statistics alone suggest there will be racist truths such as?
buggythebug 3y ago
Can you expand on the last sentence of your first paragraph?
- redthrowaway 3y ago
  Crime stats, average IQ across groups, stereotype accuracy, etc. What's interesting to me is not the above, which is naughty in the anglosphere, but the question of the unknown unknowns that could be as bad or worse in other cultural contexts. There are probably enough people of Indian descent involved in GPT's development that they could guide it past some of the caste landmines, but what about a country like Turkey? We know they have massive internal divisions, but do we know what would exacerbate them and how to avoid them? What about Iran, or South Africa, or Brazil? We RLHF the piss out of LLMs to ensure they don't say things that make white college graduates in San Francisco ornery, but I'd suggest the much greater risk lies in accidentally spawning scissor statements in cultures you don't know how to begin to parse to figure out what to avoid.
  - a_cardboard_box 3y ago
    > Crime stats, average IQ across groups, stereotype accuracy, etc. If you measured these stats for Irish Americans in 1865 you'd also see high crime and low IQ. If you measure these stats with recent black immigrants from Africa, you see low crime and high IQ. These statistical differences are not caused by race. An all-knowing oracle wouldn't need to hold "opinions that are racist" to understand them.

ksherlock 3y ago

Not to get too far off topic, but that reminds me of a quote: "Unix was not designed to stop you from doing stupid things, because that would also stop you from doing clever things." -- Doug Gwyn Or maybe it's: "C is a language that doesn't get in your way. It doesn't stop you from doing dumb things, but it also doesn't stop you from doing clever things." -- Dennis Ritchie. I asked Bard for a source on those quotes and it couldn't find one for the first. Wikiquotes sources it to "Introducing Regular Expressions" by Michael Fitzgerald and that does include it as a quote but it's not the source of the quote, it's just a nice quote at the start of the chapter. For the second, Bard claims to be from a 1990 interview and is on page 21 of "The Art of Unix Programming" by Brian Kernighan and Rob Pike. There is a book called "The At of Unix Programming" (2003) but it's by Eric Raymond and I could not find the quote in the book. Pike and Kernighan have two books, "The Practice of Programming" (1999) and "The Unix Programming Environment" (1984). Neither contain that quote.

SV_BubbleTime 3y ago
Don’t ask an LLM objective things. Ask them subjective. They are language models, not fact models.

fallingknife 3y ago

[flagged]

wing-_-nuts 3y ago
It's one thing to communicate an unpopular idea in a civil manner. It's quite another to be offensive. Now, I will admit there are some people out there that cannot separate their feelings for an idea, and their feelings for the person communicating it. I can't really help that person. What I have noticed is those who express your sentiment are often looking for license to be uncivil and offensive first, and their 'ideas' are merely a tool you use to do that. That I judge. I think that's mostly what others judge too.

haolez 3y ago

I think it's more likely that they nerfed it due to scaling pains.