16 ms·

Yeah, but it's better than nothing. People underestimate the value of rules that only take malice and a little knowledge to break. And they tend to exaggerate

Yeah, but it's better than nothing. People underestimate the value of rules that only take malice and a little knowledge to break. And they tend to exaggerate that underestimation if they... don't like the rule.

JoshTriplett 2mo ago
Exactly. If this works on pull requests, for instance, it'd be really useful for projects trying to do a first-pass filter to close slop spam.
- hennell 2mo ago
  Can it work on code itself? Obviously if you get AI to write the PR description that could be watermarked, but code isn't going to like random unicode, and the sythID approach feels like it would fall apart in the quite strict syntax of most code.
  - pavon 2mo ago
    If a language supports unicode it will have a fairly permissive definition of whitespace, and it will be easy to generate permutations of the whitespace that meet the syntax requirements. Of course I would want my code formatting tool to normalize that all to plain 0x20 spaces. But it would still be a helpful "brown M&M" test of did you even read CONTRIBUTING and run the code formatter before submitting this PR?
    - JoshTriplett 2mo ago
      Code submitted to a project should follow the formatting standards of that project and its programming language, which doesn't leave much room for flexibility in whitespace. However, there are many ways to write comments, and many ways to write PR descriptions and commit messages, and often many choices of words to name identifiers, and other potential sources of bits of entropy with no functional impact. It doesn't take that many bits to encode a robust "AI was here" indicator.
bonoboTP 2mo ago
I even see the Gemini diamond watermark on so many fake social media profile pictures. You could ask an LLM to find a github project that removes/inpaints those watermarks and be done with it. Or you could just use the API, which doesn't stamp the visible diamond on it (just the invisible synthID). Most people are generally lazy. They upload text to LinkedIn full of "genuine", "honest" and "load-bearing".
- josephg 2mo ago
  > Most people are generally lazy. And the people who get LLMs to make content for them are often the laziest of us all! Some people can’t even be bothered to remove their prompt from the output. Or they submit work which ends with “is there anything else you want to know about ___?” Go ahead. Make tools to rip off the llm stenography. Many of the people who would proactively use tools like that are writing content by hand anyway.
pavon 2mo ago
Yes! How many people have been caught committing fraud because they used a modern word processor and fonts to forge old documents? How many leaks have occurred because people failed to redact documents correctly, despite there being easy to use tools for this very purpose? How many people neglected to strip sensitive EXIF information from images they share (before websites started doing it for them)? How many people flat out post evidence of their crimes on social media? Yes, this watermark will be easy to strip. It is still valuable for the vast majority of times where people just don't.
krapp 2mo ago
It's worse than nothing because it gives people the false impression that there exists a system they can trust to tell them what information is authentic and what isn't. The incentive for governments and other interests to undermine that system to push false narratives and disinformation is just too great.
nradov 2mo ago
How is this "better than nothing" considering that it's not even needed in the first place?

JoshTriplett 2mo ago

Exactly. If this works on pull requests, for instance, it'd be really useful for projects trying to do a first-pass filter to close slop spam.

hennell 2mo ago
Can it work on code itself? Obviously if you get AI to write the PR description that could be watermarked, but code isn't going to like random unicode, and the sythID approach feels like it would fall apart in the quite strict syntax of most code.
- pavon 2mo ago
  If a language supports unicode it will have a fairly permissive definition of whitespace, and it will be easy to generate permutations of the whitespace that meet the syntax requirements. Of course I would want my code formatting tool to normalize that all to plain 0x20 spaces. But it would still be a helpful "brown M&M" test of did you even read CONTRIBUTING and run the code formatter before submitting this PR?
  - JoshTriplett 2mo ago
    Code submitted to a project should follow the formatting standards of that project and its programming language, which doesn't leave much room for flexibility in whitespace. However, there are many ways to write comments, and many ways to write PR descriptions and commit messages, and often many choices of words to name identifiers, and other potential sources of bits of entropy with no functional impact. It doesn't take that many bits to encode a robust "AI was here" indicator.

bonoboTP 2mo ago

I even see the Gemini diamond watermark on so many fake social media profile pictures. You could ask an LLM to find a github project that removes/inpaints those watermarks and be done with it. Or you could just use the API, which doesn't stamp the visible diamond on it (just the invisible synthID). Most people are generally lazy. They upload text to LinkedIn full of "genuine", "honest" and "load-bearing".

josephg 2mo ago
> Most people are generally lazy. And the people who get LLMs to make content for them are often the laziest of us all! Some people can’t even be bothered to remove their prompt from the output. Or they submit work which ends with “is there anything else you want to know about ___?” Go ahead. Make tools to rip off the llm stenography. Many of the people who would proactively use tools like that are writing content by hand anyway.

pavon 2mo ago

Yes! How many people have been caught committing fraud because they used a modern word processor and fonts to forge old documents? How many leaks have occurred because people failed to redact documents correctly, despite there being easy to use tools for this very purpose? How many people neglected to strip sensitive EXIF information from images they share (before websites started doing it for them)? How many people flat out post evidence of their crimes on social media? Yes, this watermark will be easy to strip. It is still valuable for the vast majority of times where people just don't.

krapp 2mo ago

It's worse than nothing because it gives people the false impression that there exists a system they can trust to tell them what information is authentic and what isn't. The incentive for governments and other interests to undermine that system to push false narratives and disinformation is just too great.

nradov 2mo ago

How is this "better than nothing" considering that it's not even needed in the first place?