4 ms·
It says they observed no difference in people clicking thumbs up or down. There are loads of other behavior that they didn't observe; like, say, switching to a
by gwd 2mo ago
It says they observed no difference in people clicking thumbs up or down. There are loads of other behavior that they didn't observe; like, say, switching to a different LLM.
- iainmerrick 2mo agoHow exactly do you propose they should keep track of quality, then, if not by A/B testing?
- gwd 2mo agoThe question here is not to what product management advice to the Gemini team. The discussion here is whether the watermarking is noticeable. The easy thing to do here would be to have 1000 questions, randomly assigning one half to an LLM with a watermark, and the other half without. Then show people pairs and say, "Which one seems watermarked?" (Or, "Which text seems more natural" or "Which is a better answer" or something like that.) If they come out equal, the watermark really is indiscernible, at least to most people.
- iainmerrick 2mo agoThey do exactly that "which is the better answer?" test -- I've seen it pop up a few times.
- summarybot 2mo agoIsn't "which one is watermarked?" a different question than "which one is better?" "Which diamonds are shinier, the blood diamond sourced ones or the ethically sourced ones?" ... that's not the same question as "which diamonds are blood diamonds" (to employ an extreme analogy) Concluding that no one could detect which ones were blood diamonds because they were "equally shiny" is not really correct now, is it?
- iainmerrick 2mo agoThat's true, but you don't typically explain what you're testing in this sort of (presumably) randomised trial. And the Daring Fireball article does complain that watermarking will reduce quality. If that's what you're trying to check, "which is better?" is the right question.
- Fogest 2mo agoAgreed, if I simply didn't like the style or words an AI was using in something it wrote, I would switch to a competitors and see what it can come with. I probably wouldn't hit the thumbs down on the Gemini response as it's not that the response is wrong, I just didn't like it. I usually reserve the thumb down for when the AI is wrong. Also, depending on what I am asking it, I often don't want to use the thumb down or up, as this may mean my conversation is going to have some kind of human review and depending on what I am asking for, I may not want to bring attention to my stuff.