12 ms·

Think about Copilot taking code from GitHub. If you start letting it contribute potentially shaky code back to GitHub, it's going to learn more incorrect patter

Think about Copilot taking code from GitHub. If you start letting it contribute potentially shaky code back to GitHub, it's going to learn more incorrect patterns on its next harvest. It's a feedback loop. Anyone who wants accurate AI answers would not want the AI itself to contribute to the learning pool. The same is true for S.E. and ChatGPT. ChatGPT doesn't understand why something works or doesn't; it just infers the next word or sentence from the previous language. It's not factually accurate, but it draws from material that contains facts. If it's allowed to contribute back to its source data, it gets worse as a model.

echelon 4y ago
I'm confident we're better than this. Human evaluators can filter out the (for now) intermediate bad results. All of our AI models and products will climb a quality gradient year by year to the point where this will be increasingly less necessary. To your other point, you can use machine data to bootstrap a "real" model. My team has done this to incredible effect.
- noduerme 4y ago
  You may be right in the long run, but it's definitely a gradient and the capacity for AI answers to be pasted thoughtlessly far outweighs the human capacity to correct it on a site that already creaks under the weight of insufficiently answered questions. I think if you ran such a site, you'd want to stay a little behind the curve of intermediate false but truthy-sounding responses.
pledess 4y ago
Would it maybe help to extend https://developers.google.com/search/blog/2019/09/evolving-nofollow-new-ways-to-identify https://developers.google.com/search/blog/2019/09/evolving-n... with new options, like rel="aigc" or rel="noTrainingOfYourLanguageModel" ?
nl 4y ago
> Think about Copilot taking code from GitHub. If you start letting it contribute potentially shaky code back to GitHub, it's going to learn more incorrect patterns on its next harvest. It's a feedback loop. ChatGPT (and presumably soon CoPilot) learn via * Reinforcement Learning from Human Feedback*[1] on top of the raw language model. This takes low numbers of high quality examples to improve the quality, and avoids the "junk training data" entropy problem you identify. The OpenAI Codex text-davinci-002 and text-davinci-003 models have been trained for code generation using this human feedback process[2]. [1] https://huggingface.co/blog/rlhf https://huggingface.co/blog/rlhf [2] https://beta.openai.com/docs/model-index-for-researchers https://beta.openai.com/docs/model-index-for-researchers
- noduerme 4y ago
  >> ChatGPT (and presumably soon CoPilot) learn via * Reinforcement Learning from Human Feedback* Technically, they aren't trained that way, their opposite recognition AI is trained that way. That's less than ideal when their original set of training data aren't drawn from their own output, only from a pool of humans influenced by their output. It becomes suicidal to the AI when it begins consuming its own output through what it considers to be trusted channels. [edit] The ancient coder phrase "garbage in, garbage out" has never been more significant than now in the context of what's fed into neural nets and ML algos. I think it's basically silly hubris sprouting from a lack of underpinning technical understanding when people assert these models will train themselves without much more extreme means of assessing their output than can be provided by a few minimum wage workers somewhere.
  - nl 4y ago
    > Technically, they aren't trained that way, their opposite recognition AI is trained that way. No, they are trained that way. These aren't GANs or anything resembling that. There is a reinforcement model that is build from expert human guidance which is then used to fine-tune the LM. From the HuggingFace explainer page I linked above: [there are] three core steps: 1. Pretraining a language model (LM), 2. gathering data and training a reward model, and 3. fine-tuning the LM with reinforcement learning. Read https://arxiv.org/abs/2203.02155 https://arxiv.org/abs/2203.02155 if you want all the details (although the HF explainer is easier to follow). From the abstract of that paper: > Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. There is no "opposite recognition AI".

echelon 4y ago

I'm confident we're better than this. Human evaluators can filter out the (for now) intermediate bad results. All of our AI models and products will climb a quality gradient year by year to the point where this will be increasingly less necessary. To your other point, you can use machine data to bootstrap a "real" model. My team has done this to incredible effect.

noduerme 4y ago
You may be right in the long run, but it's definitely a gradient and the capacity for AI answers to be pasted thoughtlessly far outweighs the human capacity to correct it on a site that already creaks under the weight of insufficiently answered questions. I think if you ran such a site, you'd want to stay a little behind the curve of intermediate false but truthy-sounding responses.

pledess 4y ago

Would it maybe help to extend https://developers.google.com/search/blog/2019/09/evolving-nofollow-new-ways-to-identify https://developers.google.com/search/blog/2019/09/evolving-n... with new options, like rel="aigc" or rel="noTrainingOfYourLanguageModel" ?

nl 4y ago

> Think about Copilot taking code from GitHub. If you start letting it contribute potentially shaky code back to GitHub, it's going to learn more incorrect patterns on its next harvest. It's a feedback loop. ChatGPT (and presumably soon CoPilot) learn via * Reinforcement Learning from Human Feedback*[1] on top of the raw language model. This takes low numbers of high quality examples to improve the quality, and avoids the "junk training data" entropy problem you identify. The OpenAI Codex text-davinci-002 and text-davinci-003 models have been trained for code generation using this human feedback process[2]. [1] https://huggingface.co/blog/rlhf https://huggingface.co/blog/rlhf [2] https://beta.openai.com/docs/model-index-for-researchers https://beta.openai.com/docs/model-index-for-researchers

noduerme 4y ago
>> ChatGPT (and presumably soon CoPilot) learn via * Reinforcement Learning from Human Feedback* Technically, they aren't trained that way, their opposite recognition AI is trained that way. That's less than ideal when their original set of training data aren't drawn from their own output, only from a pool of humans influenced by their output. It becomes suicidal to the AI when it begins consuming its own output through what it considers to be trusted channels. [edit] The ancient coder phrase "garbage in, garbage out" has never been more significant than now in the context of what's fed into neural nets and ML algos. I think it's basically silly hubris sprouting from a lack of underpinning technical understanding when people assert these models will train themselves without much more extreme means of assessing their output than can be provided by a few minimum wage workers somewhere.
- nl 4y ago
  > Technically, they aren't trained that way, their opposite recognition AI is trained that way. No, they are trained that way. These aren't GANs or anything resembling that. There is a reinforcement model that is build from expert human guidance which is then used to fine-tune the LM. From the HuggingFace explainer page I linked above: [there are] three core steps: 1. Pretraining a language model (LM), 2. gathering data and training a reward model, and 3. fine-tuning the LM with reinforcement learning. Read https://arxiv.org/abs/2203.02155 https://arxiv.org/abs/2203.02155 if you want all the details (although the HF explainer is easier to follow). From the abstract of that paper: > Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. There is no "opposite recognition AI".