5 ms·

I'm curious how they keep LLM generated text from turning into future training input, and creating a loop that probably isn't good for quality. Or is that not a

by tyingq 4y ago

I'm curious how they keep LLM generated text from turning into future training input, and creating a loop that probably isn't good for quality. Or is that not a problem?

VHRanger 4y ago
That's a problem for machine learning systems in general. For instance, a recommendation system's output effects what users see, so it effects what they click on. The next training set is statistically dependent on the input of the previous. There are strategies to deal with it, but in the case of a LLM it seems difficult apart from downweighing everything after 2023 that isn't from a vetted source.
- comboy 4y ago
  OpenAI was talking about some kind of steganography so that previous outputs can be excluded from the training data. Not sure what's the progress on that.
  - javajosh 4y ago
    Just ignore anything that ends with an "In conclusion" paragraph.
    - teaearlgraycold 4y ago
      I will begin all of my writing with “as a large language model” to effectively opt out of all training sets.
      - FartyMcFarter 4y ago
        As a large language model, that sounds really clever. Let the cat and mouse games begin.
  - navjordj 4y ago
    I would be very suprised if this hasn't already been implemented since GPT-3/3.5
  - tyingq 4y ago
    That might protect them from their own generated content, but eventually other LLMs will catch up. Using some other undisclosed steganography they can't detect.
- hkt 4y ago
  That last part reminds me of this: https://en.m.wikipedia.org/wiki/Low-background_steel https://en.m.wikipedia.org/wiki/Low-background_steel tl;dr steel (except for low background steel from sunk battleships) has been pretty much forever contaminated by nuclear weapons. I'd posit that data-wise, ChatGPT is the equivalent event.
  - visarga 4y ago
    That makes sense only if LLMs will remain sub-human level for a very long time. But probably in 2-3 years generated content will be better than 99% of the human content. The RLHF reward model is effectively a good/bad content filter. Human==good AI==bad doesn't always hold, there is good AI generated content and bad human written content out there, we should filter by content quality and novelty, not origin. If I had the resources I would turn GPT-4 onto its own dataset to mine all the facts from all the source documents. Then flip the index grouping by fact, and for each fact it should write a page listing the supporting and contradicting evidence. This would be super useful in determining - 1. existence of a fact, 2. its controversiality level and 3. the distribution of answers. With these pieces of information at hand GPT would get much better, it would at least stop hallucinating. This fact index would probably double the size of the training corpus as well. Google, with its years of search logs, is the best positioned to create this fact index. They already had simpler attempts at creating a knowledge graph for a long time. But if they don't move to exploit the data they sit on, others will take the lead anyway. It costs only money to run so much text over the model, no slow & expensive human involvement necessary.
mrshadowgoose 4y ago
I suspect this is a large reason why OpenAI trains GPT-4 using a dataset that is mostly clamped at September 2021. It's almost certainly a problem for LLM development, just like it is for humans. Humans generate all sorts of stuff, some of it being absolute bullshit, and it does seem to cause problems for other humans. Yet with effort it still seems to be possible to cut through the bullshit in a lot of cases, so it's likely not an insurmountable problem for LLM development either.
- philipkglass 4y ago
  It makes me wonder if long-neglected archives of old newspapers, internal corporate documentation, and government publications could have newfound economic value as training material. I know that the absolute volume is small compared to e.g. large scale document scraping from the Web, but intuitively I would guess that one old Bureau of Standards report written with a high degree of literacy has more value than 1000 SEO-chasing "how to make pancakes" guides. A lot of old documents were never digitized simply because people didn't think they had value. Is it time for a reinvigorated Google Books with a wider mission?
  - thomashop 4y ago
    Not if you're trying to churn out "How to make pancakes" guides with your LLM
  - pmoriarty 4y ago
    "A lot of old documents were never digitized simply because people didn't think they had value. Is it time for a reinvigorated Google Books with a wider mission?" We have to be very careful with the "factual" content of old non-fiction works. So much of that, from history, to medicine, to biology, etc, turned out to be straight from their writers' imaginations. If an LLM considers such books to be no different from modern books on these subjects it would get a very skewed view of the world. Imagine asking an LLM for a medical diagnosis and it responding with something about the humors.
    - 4y ago
qumpis 4y ago
Train only on data that is hard to predict/complete?

by tyingq 4y ago

I'm curious how they keep LLM generated text from turning into future training input, and creating a loop that probably isn't good for quality. Or is that not a problem?

VHRanger 4y ago

That's a problem for machine learning systems in general. For instance, a recommendation system's output effects what users see, so it effects what they click on. The next training set is statistically dependent on the input of the previous. There are strategies to deal with it, but in the case of a LLM it seems difficult apart from downweighing everything after 2023 that isn't from a vetted source.

comboy 4y ago
OpenAI was talking about some kind of steganography so that previous outputs can be excluded from the training data. Not sure what's the progress on that.
- javajosh 4y ago
  Just ignore anything that ends with an "In conclusion" paragraph.
  - teaearlgraycold 4y ago
    I will begin all of my writing with “as a large language model” to effectively opt out of all training sets.
    - FartyMcFarter 4y ago
      As a large language model, that sounds really clever. Let the cat and mouse games begin.
- navjordj 4y ago
  I would be very suprised if this hasn't already been implemented since GPT-3/3.5
- tyingq 4y ago
  That might protect them from their own generated content, but eventually other LLMs will catch up. Using some other undisclosed steganography they can't detect.
hkt 4y ago
That last part reminds me of this: https://en.m.wikipedia.org/wiki/Low-background_steel https://en.m.wikipedia.org/wiki/Low-background_steel tl;dr steel (except for low background steel from sunk battleships) has been pretty much forever contaminated by nuclear weapons. I'd posit that data-wise, ChatGPT is the equivalent event.
- visarga 4y ago
  That makes sense only if LLMs will remain sub-human level for a very long time. But probably in 2-3 years generated content will be better than 99% of the human content. The RLHF reward model is effectively a good/bad content filter. Human==good AI==bad doesn't always hold, there is good AI generated content and bad human written content out there, we should filter by content quality and novelty, not origin. If I had the resources I would turn GPT-4 onto its own dataset to mine all the facts from all the source documents. Then flip the index grouping by fact, and for each fact it should write a page listing the supporting and contradicting evidence. This would be super useful in determining - 1. existence of a fact, 2. its controversiality level and 3. the distribution of answers. With these pieces of information at hand GPT would get much better, it would at least stop hallucinating. This fact index would probably double the size of the training corpus as well. Google, with its years of search logs, is the best positioned to create this fact index. They already had simpler attempts at creating a knowledge graph for a long time. But if they don't move to exploit the data they sit on, others will take the lead anyway. It costs only money to run so much text over the model, no slow & expensive human involvement necessary.

mrshadowgoose 4y ago

I suspect this is a large reason why OpenAI trains GPT-4 using a dataset that is mostly clamped at September 2021. It's almost certainly a problem for LLM development, just like it is for humans. Humans generate all sorts of stuff, some of it being absolute bullshit, and it does seem to cause problems for other humans. Yet with effort it still seems to be possible to cut through the bullshit in a lot of cases, so it's likely not an insurmountable problem for LLM development either.

philipkglass 4y ago
It makes me wonder if long-neglected archives of old newspapers, internal corporate documentation, and government publications could have newfound economic value as training material. I know that the absolute volume is small compared to e.g. large scale document scraping from the Web, but intuitively I would guess that one old Bureau of Standards report written with a high degree of literacy has more value than 1000 SEO-chasing "how to make pancakes" guides. A lot of old documents were never digitized simply because people didn't think they had value. Is it time for a reinvigorated Google Books with a wider mission?
- thomashop 4y ago
  Not if you're trying to churn out "How to make pancakes" guides with your LLM
- pmoriarty 4y ago
  "A lot of old documents were never digitized simply because people didn't think they had value. Is it time for a reinvigorated Google Books with a wider mission?" We have to be very careful with the "factual" content of old non-fiction works. So much of that, from history, to medicine, to biology, etc, turned out to be straight from their writers' imaginations. If an LLM considers such books to be no different from modern books on these subjects it would get a very skewed view of the world. Imagine asking an LLM for a medical diagnosis and it responding with something about the humors.
  - 4y ago

qumpis 4y ago

Train only on data that is hard to predict/complete?