3 ms·

It's this sort of thing that makes me bullish on OpenAI, oddly enough. I keep hearing rumours that it's possible to watermark LLM output with a sort of unique k

It's this sort of thing that makes me bullish on OpenAI, oddly enough. I keep hearing rumours that it's possible to watermark LLM output with a sort of unique key such that the resulting text still sounds just as fluent, but it can be detected with 100% accuracy given more than a few words and the key. I also hear that OpenAI have mastered this technology. If that's true then it leads to the possibility of converting first mover advantage to absolute market dominance: 1. Spend lots of money to give a high quality LLM away for free (ChatGPT). Rapidly achieve market dominance by doing so. 2. Watermark its output without making a detector publicly available. 3. Users flood the internet with watermarked text. 4. Now you can generate fresh training data without circular corruption because almost all the LLM output on the internet is yours. 5. Your competitors incorporate LLM output into their training inputs because they can't detect it. Their quality suffers in ways yours don't. 6. Users notice the quality gap and continue to use your model. Competitors can't close it because they have no way to do so. 7. Profit! This sort of loop happened with Google where using the click stream to do ranking yielded a virtuous cycle that competitors struggled to beat, and then sites using robots.txt to ban non-Googlebots made it worse. If this equivalent actually happens then OpenAI can end up having an absolute monopoly on LLM technology.

kmeisthax 3y ago
This is plausible, but I can put some bounds on the scope of the conspiracy. OpenAI has already released their own AI text classifiers[0], and they are nowhere near good enough to suggest the use of stylometric[1] watermarking. Furthermore, if such a watermark did exist, it would be detectable by anyone with a large enough corpus of known AI text. It probably would also be easier to detect (have lower false classification rates) than the unintentional stylo in ChatGPT output that most AI plagiarism detection is currently identifying. I could see OpenAI archiving literally every line of text that their models generated, and then doing normal full-text search on their text corpus to remove their own output from it. But there's still one other flaw: the existence of other models. GPT-3+ are not a monopoly. Google has BERT, Facebook has LLaMA, and the FOSS community has BLOOM, StableLM, and CerebrasGPT. While these models may or may not be GPT-4 quality, they will still be generating text without OpenAI's watermark, meaning that OpenAI won't know to filter for it. So even if they did plan to use watermarking as a way to invisibly scrape human content while polluting other companies' training sets, they wouldn't be able to succeed on such a plan. [0] https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text https://openai.com/blog/new-ai-classifier-for-indicating-ai-... [1] Stylometry is the process of fingerprinting word usage to determine authorship. This was used, for example, to unmask "Robert Galbraith" as the pen name of J.K. Rowling, before she made it too easy by just using her mystery novels as a way to rant about trans people and Twitter. More famously, the Unabomber was unmasked as Ted Kaczynski using the same means. Anonymity is a polite fiction and how you write is personally identifying information.
ChatGTP 3y ago
This assumes you can actually watermark the text, which sounds like total bullshit ?

kmeisthax 3y ago

This is plausible, but I can put some bounds on the scope of the conspiracy. OpenAI has already released their own AI text classifiers[0], and they are nowhere near good enough to suggest the use of stylometric[1] watermarking. Furthermore, if such a watermark did exist, it would be detectable by anyone with a large enough corpus of known AI text. It probably would also be easier to detect (have lower false classification rates) than the unintentional stylo in ChatGPT output that most AI plagiarism detection is currently identifying. I could see OpenAI archiving literally every line of text that their models generated, and then doing normal full-text search on their text corpus to remove their own output from it. But there's still one other flaw: the existence of other models. GPT-3+ are not a monopoly. Google has BERT, Facebook has LLaMA, and the FOSS community has BLOOM, StableLM, and CerebrasGPT. While these models may or may not be GPT-4 quality, they will still be generating text without OpenAI's watermark, meaning that OpenAI won't know to filter for it. So even if they did plan to use watermarking as a way to invisibly scrape human content while polluting other companies' training sets, they wouldn't be able to succeed on such a plan. [0] https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text https://openai.com/blog/new-ai-classifier-for-indicating-ai-... [1] Stylometry is the process of fingerprinting word usage to determine authorship. This was used, for example, to unmask "Robert Galbraith" as the pen name of J.K. Rowling, before she made it too easy by just using her mystery novels as a way to rant about trans people and Twitter. More famously, the Unabomber was unmasked as Ted Kaczynski using the same means. Anonymity is a polite fiction and how you write is personally identifying information.

ChatGTP 3y ago

This assumes you can actually watermark the text, which sounds like total bullshit ?