3 ms·

Worth pointing out that while watermarking is mathematically reliable, the scammers who are selling "AI detection" don't have the weight-level access that it re

• eitland 2y ago
  > watermarking is mathematically reliable So should accounting be. To a much higher degree. Yet I hope most of us are aware of the British Post Office scandal where what really should be accounting software falsely accused thousands of employees of theft, of which over 900 were convicted of theft, fraud and false accounting. If this can happen in something as utterly boring as an accounting system should be in this millennium, I don't think we should trust AI fraud detection in science and academia until we get a few decades of experience with it. (Do I think records from accounting can be used as evidence? Absolutely, given the right circumstances: we can know it hasn't been tampered with etc What I don't think however is that pattern matching or "watermarks" that indicates probability should be used as evidence. Especially not closed source systems with secret distributions and watermarking algorithms.)
  • zebomon 2y ago
    I agree with you completely. In the wild, there are too many variables to use watermarking to draw meaningful conclusions about any piece of text, no matter the word count. Scott Aaronson described well one of those variables, "the pineapple attack," in his Aug. 2023 talk at Simons [1]. Watermarking is illuminating to the ongoing study of language models' functionality, but it doesn't put the genie back in the bottle. 1. https://www.youtube.com/watch?v=2Kx9jbSMZqA https://www.youtube.com/watch?v=2Kx9jbSMZqA