3 ms·

That's a problem for machine learning systems in general. For instance, a recommendation system's output effects what users see, so it effects what they click

That's a problem for machine learning systems in general. For instance, a recommendation system's output effects what users see, so it effects what they click on. The next training set is statistically dependent on the input of the previous. There are strategies to deal with it, but in the case of a LLM it seems difficult apart from downweighing everything after 2023 that isn't from a vetted source.

comboy 4y ago
OpenAI was talking about some kind of steganography so that previous outputs can be excluded from the training data. Not sure what's the progress on that.
- javajosh 4y ago
  Just ignore anything that ends with an "In conclusion" paragraph.
  - teaearlgraycold 4y ago
    I will begin all of my writing with “as a large language model” to effectively opt out of all training sets.
    - FartyMcFarter 4y ago
      As a large language model, that sounds really clever. Let the cat and mouse games begin.
- navjordj 4y ago
  I would be very suprised if this hasn't already been implemented since GPT-3/3.5
- tyingq 4y ago
  That might protect them from their own generated content, but eventually other LLMs will catch up. Using some other undisclosed steganography they can't detect.
hkt 4y ago
That last part reminds me of this: https://en.m.wikipedia.org/wiki/Low-background_steel https://en.m.wikipedia.org/wiki/Low-background_steel tl;dr steel (except for low background steel from sunk battleships) has been pretty much forever contaminated by nuclear weapons. I'd posit that data-wise, ChatGPT is the equivalent event.
- visarga 4y ago
  That makes sense only if LLMs will remain sub-human level for a very long time. But probably in 2-3 years generated content will be better than 99% of the human content. The RLHF reward model is effectively a good/bad content filter. Human==good AI==bad doesn't always hold, there is good AI generated content and bad human written content out there, we should filter by content quality and novelty, not origin. If I had the resources I would turn GPT-4 onto its own dataset to mine all the facts from all the source documents. Then flip the index grouping by fact, and for each fact it should write a page listing the supporting and contradicting evidence. This would be super useful in determining - 1. existence of a fact, 2. its controversiality level and 3. the distribution of answers. With these pieces of information at hand GPT would get much better, it would at least stop hallucinating. This fact index would probably double the size of the training corpus as well. Google, with its years of search logs, is the best positioned to create this fact index. They already had simpler attempts at creating a knowledge graph for a long time. But if they don't move to exploit the data they sit on, others will take the lead anyway. It costs only money to run so much text over the model, no slow & expensive human involvement necessary.

comboy 4y ago

OpenAI was talking about some kind of steganography so that previous outputs can be excluded from the training data. Not sure what's the progress on that.

javajosh 4y ago
Just ignore anything that ends with an "In conclusion" paragraph.
- teaearlgraycold 4y ago
  I will begin all of my writing with “as a large language model” to effectively opt out of all training sets.
  - FartyMcFarter 4y ago
    As a large language model, that sounds really clever. Let the cat and mouse games begin.
navjordj 4y ago
I would be very suprised if this hasn't already been implemented since GPT-3/3.5
tyingq 4y ago
That might protect them from their own generated content, but eventually other LLMs will catch up. Using some other undisclosed steganography they can't detect.

hkt 4y ago

That last part reminds me of this: https://en.m.wikipedia.org/wiki/Low-background_steel https://en.m.wikipedia.org/wiki/Low-background_steel tl;dr steel (except for low background steel from sunk battleships) has been pretty much forever contaminated by nuclear weapons. I'd posit that data-wise, ChatGPT is the equivalent event.

visarga 4y ago
That makes sense only if LLMs will remain sub-human level for a very long time. But probably in 2-3 years generated content will be better than 99% of the human content. The RLHF reward model is effectively a good/bad content filter. Human==good AI==bad doesn't always hold, there is good AI generated content and bad human written content out there, we should filter by content quality and novelty, not origin. If I had the resources I would turn GPT-4 onto its own dataset to mine all the facts from all the source documents. Then flip the index grouping by fact, and for each fact it should write a page listing the supporting and contradicting evidence. This would be super useful in determining - 1. existence of a fact, 2. its controversiality level and 3. the distribution of answers. With these pieces of information at hand GPT would get much better, it would at least stop hallucinating. This fact index would probably double the size of the training corpus as well. Google, with its years of search logs, is the best positioned to create this fact index. They already had simpler attempts at creating a knowledge graph for a long time. But if they don't move to exploit the data they sit on, others will take the lead anyway. It costs only money to run so much text over the model, no slow & expensive human involvement necessary.