4 ms·

How is it possible for OpenAI to use their users to train their model? I will sometimes as ChatGPT a couple of questions, and sometimes it gives me useful resp

How is it possible for OpenAI to use their users to train their model? I will sometimes as ChatGPT a couple of questions, and sometimes it gives me useful responses and I’m done, and other times it gives me useless responses and I’m done. Presumably OpenAI would need a way of discerning the useful output from the useless output in order to do this, but I’m not providing any feedback about which is which when I use the product.

choilive 2y ago
They do.. they literally have thumbs up/down buttons on the messages.
- AmericanChopper 2y ago
  Only on some of their interfaces, but even then that seems like a rather unreliable method to perform RLHF. I’ve never clicked a thumbs up/down on a ChatGTP output, but I don’t imagine people are validating the output very thoroughly before providing that feedback…
- fsndz 2y ago
  and sometimes they generate two answers and the same time and users have to choose one
bobsmooth 2y ago
I thank chatgpt when it does a good job. I like to think I'm helping to improve the model.
RevEng 2y ago
Training data can be anything. They scrape the entire internet, plenty of which is inaccurate or poorly written. That doesn't prevent it from being useful training data because the point of an LLM trained for text generation is to predict what someone would write. Every question you ask and your responses to their responses is valuable data that isn't already available on the public internet. Even if some of it is unreliable or even intentionally adversarial, on average the responses will be useful for training. This is training data that they have exclusive access to it.
- fsndz 2y ago
  Typically, when people aren't satisfied with an answer, they continue asking questions or prompting further, which makes the whole exchange valuable even without the use of a thumbs-up or thumbs-down button. That's the secret weapon of OpenAI, and a part of their 'moat'
  - AmericanChopper 2y ago
    This is what I’m skeptical of, because my experience with LLMs doesn’t align with this description at all. Even if a response is good, I will typically give it follow up prompts to get more details, or answer other questions that the (potentially high quality) response raised. If the response is bad I might try some follow ups to see if it improves. In either case, I’m submitting a prompt, I might accept the answer immediately, give up immediately, accept the answer after further prompting, or give up after further prompting. With my own use there is no correlation between the number of prompts I submit, and the quality of the responses given. If this is the metric OpenAI is using to perform crowdsourced RLHF, then the reinforcement is going to be garbage.
    - fsndz 2y ago
      The chain of thought that is apparent in the conversation is what's really interesting and what OpenAI exploits. OpenAI does not have to evaluate the quality because the conversation itself is already valuable. Whether the conversation was good or bad can easily be inferred from the back-and-forth. That's the key: they just have to take all those conversations and fine-tune their models with them.
    - ben_w 2y ago
      What about the nature of your follow up responses? Do you say "No, not x, y"? Or perhaps "Now Baz the Foos"?
visarga 2y ago
> Presumably OpenAI would need a way of discerning the useful output from the useless output You are the validator, using real world testing. But not in one-off interactions, when you exchange multiple rounds with the model. After the whole chat session is finished, the model can rank their answers in context, having hindsight. They can just look at the later outcomes and observe which idea was good or bad. Especially bad ideas would elicit a response from the user, trying to iterate on them until solved. Basically the LLM is exploring the world through indirect agency. They act through users, and collect outcomes also through users. More complex projects can even be spread out over many days in many sessions, LLM providers need to comb through the logs to see "idea -> outcome" chains. This whole feedback collection process stops working in one-round interactions, like many we have in the API. But in the chatroom a significant number of interactions continue after the first response, or patterns emerge across multiple sessions, as new one-off interactions relate to past ones. Scale this to 200M users, they have a huge amount of personal experience and interactivity to offer to the model. An experience flywheel that could be spinning once a year, or once a month, or even daily, absorbing new experience from users and serving it back to the users.
- AmericanChopper 2y ago
  > Especially bad ideas would elicit a response from the user, trying to iterate on them until solved. This is the assumption that’s been repeated a few times now in response to my comment, but this is the assumption that seems wrong to me. If I get a good output from an LLM, I probably also want more output on the same topic, or to try refine it somehow. If I get a bad output I will probably try to iterate on it a bit. From the perspective of the LLM, these two interactions look the same.

choilive 2y ago

They do.. they literally have thumbs up/down buttons on the messages.

AmericanChopper 2y ago
Only on some of their interfaces, but even then that seems like a rather unreliable method to perform RLHF. I’ve never clicked a thumbs up/down on a ChatGTP output, but I don’t imagine people are validating the output very thoroughly before providing that feedback…
fsndz 2y ago
and sometimes they generate two answers and the same time and users have to choose one

bobsmooth 2y ago

I thank chatgpt when it does a good job. I like to think I'm helping to improve the model.

RevEng 2y ago

Training data can be anything. They scrape the entire internet, plenty of which is inaccurate or poorly written. That doesn't prevent it from being useful training data because the point of an LLM trained for text generation is to predict what someone would write. Every question you ask and your responses to their responses is valuable data that isn't already available on the public internet. Even if some of it is unreliable or even intentionally adversarial, on average the responses will be useful for training. This is training data that they have exclusive access to it.

fsndz 2y ago
Typically, when people aren't satisfied with an answer, they continue asking questions or prompting further, which makes the whole exchange valuable even without the use of a thumbs-up or thumbs-down button. That's the secret weapon of OpenAI, and a part of their 'moat'
- AmericanChopper 2y ago
  This is what I’m skeptical of, because my experience with LLMs doesn’t align with this description at all. Even if a response is good, I will typically give it follow up prompts to get more details, or answer other questions that the (potentially high quality) response raised. If the response is bad I might try some follow ups to see if it improves. In either case, I’m submitting a prompt, I might accept the answer immediately, give up immediately, accept the answer after further prompting, or give up after further prompting. With my own use there is no correlation between the number of prompts I submit, and the quality of the responses given. If this is the metric OpenAI is using to perform crowdsourced RLHF, then the reinforcement is going to be garbage.
  - fsndz 2y ago
    The chain of thought that is apparent in the conversation is what's really interesting and what OpenAI exploits. OpenAI does not have to evaluate the quality because the conversation itself is already valuable. Whether the conversation was good or bad can easily be inferred from the back-and-forth. That's the key: they just have to take all those conversations and fine-tune their models with them.
  - ben_w 2y ago
    What about the nature of your follow up responses? Do you say "No, not x, y"? Or perhaps "Now Baz the Foos"?

visarga 2y ago

> Presumably OpenAI would need a way of discerning the useful output from the useless output You are the validator, using real world testing. But not in one-off interactions, when you exchange multiple rounds with the model. After the whole chat session is finished, the model can rank their answers in context, having hindsight. They can just look at the later outcomes and observe which idea was good or bad. Especially bad ideas would elicit a response from the user, trying to iterate on them until solved. Basically the LLM is exploring the world through indirect agency. They act through users, and collect outcomes also through users. More complex projects can even be spread out over many days in many sessions, LLM providers need to comb through the logs to see "idea -> outcome" chains. This whole feedback collection process stops working in one-round interactions, like many we have in the API. But in the chatroom a significant number of interactions continue after the first response, or patterns emerge across multiple sessions, as new one-off interactions relate to past ones. Scale this to 200M users, they have a huge amount of personal experience and interactivity to offer to the model. An experience flywheel that could be spinning once a year, or once a month, or even daily, absorbing new experience from users and serving it back to the users.

AmericanChopper 2y ago
> Especially bad ideas would elicit a response from the user, trying to iterate on them until solved. This is the assumption that’s been repeated a few times now in response to my comment, but this is the assumption that seems wrong to me. If I get a good output from an LLM, I probably also want more output on the same topic, or to try refine it somehow. If I get a bad output I will probably try to iterate on it a bit. From the perspective of the LLM, these two interactions look the same.