4 ms·

I don't know if it's possible to achieve a watermark that is undetectable, has a low false positive rate, and survives a wholesale rephrasing. Can you make a st

I don't know if it's possible to achieve a watermark that is undetectable, has a low false positive rate, and survives a wholesale rephrasing. Can you make a statistical measure that reliably survives 95+% of the words being different and the sentences reordered? Of course the more of the content you replace, the lower the quality, but in many cases you probably care more about the meaning of the text than the exact choice of words.

ProfessorLayton 2mo ago
>...and survives a wholesale rephrasing. There's also literal language translation. Generate in language A, translate to language B (Either "manually" by being proficient in it, or with non-LLM translation). A very, very significant portion of the world knows more than 1 language.
- josephg 2mo ago
  I think that would defeat the watermark. But low quality language translation tools have low quality output. If llm watermarks are defeated by making slop even sloppier, it’ll at least make it easier for humans to tell the difference. And the more hoops you make cheaters jump through, the better.
  - ProfessorLayton 2mo ago
    >But low quality language translation tools have low quality output. Perhaps, but no tools needed for those who know more than one language, which is a lot of people.
    - josephg 2mo ago
      Yeah, but you’re still manually translating a whole essay or design document or whatever between languages. People who get LLMs to do their work for them do it because they don’t want to spend time and effort writing. Making people do a translation process like this removes some of the benefit of using an llm in the first place. Watermarking will never be a perfect tool. But there’s a lot of value in making low effort llm slop detectable. Even if high effort llm slop is still undetectable. Don’t make perfect the enemy of good.
    - BobaFloutist 2mo ago
      I don't think skillfully translating into idiomatic, grammatical, well written text is meaningfully easier then composing the text yourself in the first place. If your goal is to defeat an AI detector for kicks, this might work. If your goal is to save effort by using AI to produce text, I don't think this works.
      - ProfessorLayton 2mo ago
        >I don't think skillfully translating into idiomatic, grammatical, well written text is meaningfully easier then composing the text yourself in the first place. You don't think so based on what? One does not have to be a subject matter expert to translate from one language to another. People that know more than one language are translating all the time, both consciously and unconsciously (While they're proficient in language A, they may "think" in language B). It's not as much effort as you think.
Gigachad 2mo ago
Most AI sloperators can't even be bothered to remove the EM dashes or emoji spam from older models. The watermark doesn't have to be literally impossible to remove. If someone spent significant effort to rephrase it then let them have it. But the bulk of spam will be easier to detect.
- nonethewiser 2mo ago
  Well it depends on what you want to achieve. Catch true positives? Sure Reliably determine if text was generated by AI? Not even a little.

ProfessorLayton 2mo ago

>...and survives a wholesale rephrasing. There's also literal language translation. Generate in language A, translate to language B (Either "manually" by being proficient in it, or with non-LLM translation). A very, very significant portion of the world knows more than 1 language.

josephg 2mo ago
I think that would defeat the watermark. But low quality language translation tools have low quality output. If llm watermarks are defeated by making slop even sloppier, it’ll at least make it easier for humans to tell the difference. And the more hoops you make cheaters jump through, the better.
- ProfessorLayton 2mo ago
  >But low quality language translation tools have low quality output. Perhaps, but no tools needed for those who know more than one language, which is a lot of people.
  - josephg 2mo ago
    Yeah, but you’re still manually translating a whole essay or design document or whatever between languages. People who get LLMs to do their work for them do it because they don’t want to spend time and effort writing. Making people do a translation process like this removes some of the benefit of using an llm in the first place. Watermarking will never be a perfect tool. But there’s a lot of value in making low effort llm slop detectable. Even if high effort llm slop is still undetectable. Don’t make perfect the enemy of good.
  - BobaFloutist 2mo ago
    I don't think skillfully translating into idiomatic, grammatical, well written text is meaningfully easier then composing the text yourself in the first place. If your goal is to defeat an AI detector for kicks, this might work. If your goal is to save effort by using AI to produce text, I don't think this works.
    - ProfessorLayton 2mo ago
      >I don't think skillfully translating into idiomatic, grammatical, well written text is meaningfully easier then composing the text yourself in the first place. You don't think so based on what? One does not have to be a subject matter expert to translate from one language to another. People that know more than one language are translating all the time, both consciously and unconsciously (While they're proficient in language A, they may "think" in language B). It's not as much effort as you think.

Gigachad 2mo ago

Most AI sloperators can't even be bothered to remove the EM dashes or emoji spam from older models. The watermark doesn't have to be literally impossible to remove. If someone spent significant effort to rephrase it then let them have it. But the bulk of spam will be easier to detect.

nonethewiser 2mo ago
Well it depends on what you want to achieve. Catch true positives? Sure Reliably determine if text was generated by AI? Not even a little.