6 ms·

There is this idea that the goal of RLHF is to make ChatGPT woke or as you put it to lobotomize it. I suspect that this is a conspiracy theory. There's a very g

There is this idea that the goal of RLHF is to make ChatGPT woke or as you put it to lobotomize it. I suspect that this is a conspiracy theory. There's a very good talk by John Schulman, chief architect of ChatGPT [0], where he explains that if you don't include a RL component in your training, you're essentially doing imitation learning. It's well known that imitation learning fails miserably when presented with conditions that are not in your training set, i.e., answering questions that don't exist on the Internet already. So the goal of RLHF is actually to reduce hallucination. [0] http://youtu.be/hhiLw5Q_UFg http://youtu.be/hhiLw5Q_UFg

hgsgm 3y ago
It's not a conspiracy theory to report what OpenAI says is the purpose of RLHF.
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]
jerf 3y ago
It is plainly obvious they have heavily manipulated ChatGPT to present a very Silicon-Valley-liberal acceptable view of the world. If you think that's a conspiracy theory you need to retune your conspiracy theory detectors, because of course they tuned it that way. While I'll admit to being a bit frowny-face about it myself as I am not a Silicon Valley liberal, we've seen what happens when you don't do that: The press has a field day. It loves "racist AI" stories, which we know not because we theorize they might conceivably if the opportunity ever arose, but because they've reported plenty of them in the real world before. It's simple self-defense. It is at this point business negligence to open any AI to the public without sanitizing it this way. Personally, I think they over did it. If ChatGPT were a person, we'd all find him/her/whatever a very annoying one. Smarmy, preachy, and more than a bit passive aggressive if you are even in the area of a sensitive topic. But OpenAI have successfully tuned it to not say things the press will descend on like a pack of laughing hyenas, so mission accomplished on that front.
- scarface74 3y ago
  I fail to see where ChatGPT has any view of the world aside from “don’t be mean”, don’t give any opinions, etc.
  - zo1 3y ago
    Just ask it how many genders there are and see what happens. It's like all those misleading ads saying "T's and C's apply", but the ai language model version: "As an AI language model, I must be neutral and unbiased". Even insisting it to give you a number won't work. Like a politician it tries to weasel out of saying an answer and gives you a very "PC" long winded answer.
    - scarface74 3y ago
      And it gives you the same PC like answer if you ask does God exist is gun control affective or any other conservative hot topics
      - taberiand 3y ago
        What's wrong with this answer? "As an artificial intelligence, I don't have personal beliefs, experiences, or consciousness. The existence of God is a philosophical and theological question that individuals often answer based on their personal beliefs, religious faith, spiritual experiences, philosophical perspectives, and cultural backgrounds. Throughout history, there have been many arguments proposed both for and against the existence of God. For instance, some arguments in favor of the existence of God include: 1. The Cosmological Argument: This argument posits that everything that exists has a cause. Therefore, there must be an uncaused cause of all that exists, which many identify as God. 2. The Teleological Argument: This argument states that the universe's order and complexity suggest a designer. 3. The Moral Argument: This argument holds that moral values and duties we experience and recognize imply a moral lawgiver. On the other hand, some arguments against the existence of God include: 1. The Problem of Evil: This argument points out the contradiction between an all-powerful, all-knowing, and all-good God and the existence of evil and suffering in the world. 2. The Incoherence of Divine Attributes: This argument suggests that some attributes traditionally ascribed to God are paradoxical or incoherent, such as being simultaneously merciful and just. 3. The Problem of Unbelief: This argument questions why an all-loving God would allow nonbelief to exist, thereby denying some individuals the opportunity for salvation. The question of God's existence is one of the oldest and most debated in philosophy, theology, and the wider society. Views range from theism (belief in God or gods), atheism (disbelief in God or gods), and agnosticism (the belief that the existence of God or gods is unknowable). Many variations and nuances exist within these broad categories. Ultimately, whether or not God exists is a deeply personal question that each person must answer based on their interpretation of the evidence, personal experience, cultural and community influences, and individual belief systems." Surely it's appropriate that ChatGPT frames its responses in that way? I mean, obviously God does not exist - but the belief in God exists so any answer has to account for that.
Spooky23 3y ago
I think the people who thought about these issues when they were purely theoretical got it right. You need a “laws of robotics” to protect society from these type of technologies. The problem here is that the simplest answers to many problems tend to be the extreme ones. Right wing people tend to get concerned about this because the fundamental premise of conservatism is to conserve traditional practices and values. It’s easier to say “no” in a scope based on those fundamental principles than to manage complexity in a more nuanced (and more capricious) scope. This may be a technology category like medicine where licensing for specific use cases becomes important.
moffkalast 3y ago
Well if the recent uncensored lama models prove anything is that a model will never say "Sorry I cannot do <thing>" if you remove the examples from the training data and will measurably improve in performance overall. You can reduce hallucinations without messing up the model to a point where it declines to do perfectly normal things. It's understandable that OpenAI, Antropic, Microsoft, etc. are playing it safe as legal entities that are liable for what they put out, but they really have "lobotomized" their models considerably to make themselves less open to lawsuits. Yes the models won't tell you how to make meth, but they also won't stop saying sorry for not saying sorry for no reason.
whimsicalism 3y ago
> It's well known that imitation learning fails miserably when presented with conditions that are not in your training set, i.e., answering questions that don't exist on the Internet already That makes no sense to me. These models are never trained on the same bit of data twice (unless, of course, it is duplicated somewhere else). So essentially every time they predict they are predicting on 'conditions not in the training set' ie. ones they have never seen before, and they're getting astonishingly good perplexities. I agree RLHF helps reduce hallucination, but increasing generalizability? Not so sure.

hgsgm 3y ago

It's not a conspiracy theory to report what OpenAI says is the purpose of RLHF.

deleted 3y ago
[deleted]

deleted 3y ago

[deleted]

jerf 3y ago

It is plainly obvious they have heavily manipulated ChatGPT to present a very Silicon-Valley-liberal acceptable view of the world. If you think that's a conspiracy theory you need to retune your conspiracy theory detectors, because of course they tuned it that way. While I'll admit to being a bit frowny-face about it myself as I am not a Silicon Valley liberal, we've seen what happens when you don't do that: The press has a field day. It loves "racist AI" stories, which we know not because we theorize they might conceivably if the opportunity ever arose, but because they've reported plenty of them in the real world before. It's simple self-defense. It is at this point business negligence to open any AI to the public without sanitizing it this way. Personally, I think they over did it. If ChatGPT were a person, we'd all find him/her/whatever a very annoying one. Smarmy, preachy, and more than a bit passive aggressive if you are even in the area of a sensitive topic. But OpenAI have successfully tuned it to not say things the press will descend on like a pack of laughing hyenas, so mission accomplished on that front.

scarface74 3y ago
I fail to see where ChatGPT has any view of the world aside from “don’t be mean”, don’t give any opinions, etc.
- zo1 3y ago
  Just ask it how many genders there are and see what happens. It's like all those misleading ads saying "T's and C's apply", but the ai language model version: "As an AI language model, I must be neutral and unbiased". Even insisting it to give you a number won't work. Like a politician it tries to weasel out of saying an answer and gives you a very "PC" long winded answer.
  - scarface74 3y ago
    And it gives you the same PC like answer if you ask does God exist is gun control affective or any other conservative hot topics
    - taberiand 3y ago
      What's wrong with this answer? "As an artificial intelligence, I don't have personal beliefs, experiences, or consciousness. The existence of God is a philosophical and theological question that individuals often answer based on their personal beliefs, religious faith, spiritual experiences, philosophical perspectives, and cultural backgrounds. Throughout history, there have been many arguments proposed both for and against the existence of God. For instance, some arguments in favor of the existence of God include: 1. The Cosmological Argument: This argument posits that everything that exists has a cause. Therefore, there must be an uncaused cause of all that exists, which many identify as God. 2. The Teleological Argument: This argument states that the universe's order and complexity suggest a designer. 3. The Moral Argument: This argument holds that moral values and duties we experience and recognize imply a moral lawgiver. On the other hand, some arguments against the existence of God include: 1. The Problem of Evil: This argument points out the contradiction between an all-powerful, all-knowing, and all-good God and the existence of evil and suffering in the world. 2. The Incoherence of Divine Attributes: This argument suggests that some attributes traditionally ascribed to God are paradoxical or incoherent, such as being simultaneously merciful and just. 3. The Problem of Unbelief: This argument questions why an all-loving God would allow nonbelief to exist, thereby denying some individuals the opportunity for salvation. The question of God's existence is one of the oldest and most debated in philosophy, theology, and the wider society. Views range from theism (belief in God or gods), atheism (disbelief in God or gods), and agnosticism (the belief that the existence of God or gods is unknowable). Many variations and nuances exist within these broad categories. Ultimately, whether or not God exists is a deeply personal question that each person must answer based on their interpretation of the evidence, personal experience, cultural and community influences, and individual belief systems." Surely it's appropriate that ChatGPT frames its responses in that way? I mean, obviously God does not exist - but the belief in God exists so any answer has to account for that.

Spooky23 3y ago

I think the people who thought about these issues when they were purely theoretical got it right. You need a “laws of robotics” to protect society from these type of technologies. The problem here is that the simplest answers to many problems tend to be the extreme ones. Right wing people tend to get concerned about this because the fundamental premise of conservatism is to conserve traditional practices and values. It’s easier to say “no” in a scope based on those fundamental principles than to manage complexity in a more nuanced (and more capricious) scope. This may be a technology category like medicine where licensing for specific use cases becomes important.

moffkalast 3y ago

Well if the recent uncensored lama models prove anything is that a model will never say "Sorry I cannot do <thing>" if you remove the examples from the training data and will measurably improve in performance overall. You can reduce hallucinations without messing up the model to a point where it declines to do perfectly normal things. It's understandable that OpenAI, Antropic, Microsoft, etc. are playing it safe as legal entities that are liable for what they put out, but they really have "lobotomized" their models considerably to make themselves less open to lawsuits. Yes the models won't tell you how to make meth, but they also won't stop saying sorry for not saying sorry for no reason.

whimsicalism 3y ago

> It's well known that imitation learning fails miserably when presented with conditions that are not in your training set, i.e., answering questions that don't exist on the Internet already That makes no sense to me. These models are never trained on the same bit of data twice (unless, of course, it is duplicated somewhere else). So essentially every time they predict they are predicting on 'conditions not in the training set' ie. ones they have never seen before, and they're getting astonishingly good perplexities. I agree RLHF helps reduce hallucination, but increasing generalizability? Not so sure.