3 ms·

The goal of the AI act is not to determine if an "oh yeah!" comment was AI generated. The target is long papers that falsely claim human review and can have rea

by simon84 1mo ago

The goal of the AI act is not to determine if an "oh yeah!" comment was AI generated. The target is long papers that falsely claim human review and can have real significant consequences. E.g. research paper, law makers, lawyers, state policies, notaries,... These are much longer content and thus statistically they will disclose a better guess at AI generated content. Asking another AI to paraphrase will not erase the mark (which they are unaware about) but rather cumulatively add their own mark and make it easier to detect. The problem is not to use AI, but to endorse the responsibility of the content you (as a human) deliver and somehow make sure that fake-news, biased content or unverified output is detected as early as possible.

avaer 1mo ago
You'd think that the humans being paid to review these things are actually reviewing them. Journals, laywers etc. are expensive. With the addition of AI, it should be easier than ever to review things on their merits. Maybe some part of it is that the deluge of slop is uncovering how poorly/sloppily these social institutions were working in the first place.
- truckyday 1mo ago
  [dead]
  - avaer 1mo ago
    Anecdote: I've had at two friends publish "papers" just by slapping their names on something they had literally nothing to do with, out of pure nepotism. This wouldn't be helped by an AI watermark. What would help is if the reviewer used AI to look up the authors and see the authorship claims are dubious. The papers are still up. I do think the academic publishing field is corrupt, which is why I'm not convinced it should be on AI providers to help bail it out of doing its one job (verification and trust).
    - maleldil 1mo ago
      Verifying authorship is an ambiguous task. How much involvement is necessary to qualify for authorship? If someone reads the paper and gives some small feedback, is that enough? What if they were present in one meeting and raised a question that turned out not to be interesting? What if they have no clue about the work but helped with data validation? I'd argue all of these could justify authorship, even if they're just in the middle of the author list. At least in NLP, which can be seen in the generally high number of authors in papers.
      - rcxdude 1mo ago
        The standards also vary by field. It's not unusual for supervisors to be the last author on any paper that a group publishes, even if they had basically no input into it directly. I've been listed on papers just because I designed and built the equipment that happened to be used for the experiments, even though I didn't do anything but a quick review of the actual paper. For most papers, unless there's an indication otherwise, it's generally only safe to assume the first author listed that actually did the bulk of the work and write-up, and the others are there mainly for having some potentially quite indirect contribution.
        maleldil 1mo ago
        I agree completely. That's why I pushed back on "nothing to do" from the OP. Maybe it was a small contribution, but calling it "corruption" is ridiculous.
- simon84 1mo ago
  I think you are very correct here! Now imagine how much worse it can get with AI in the way. It is as always (think cybersecurity) the cat and mouse game, what is a weapon is also a defense. You, the simple fact that you are on this very site, means that you are probably more educated to AI than most, so it is not necessarily you that will really benefit from any constraining framework. There are many people believing in many weird theories and those are more prone to be convinced by a nice narrative. AI did not introduce that, it just made it easier and available to anyone with any intent.
- maleldil 1mo ago
  Peer reviews is largely done for free. In some cases, an author needs to commit to reviewing someone else's work for their work to be reviewed. However, this has led to an increase in LLM usage for review generation, even if conference/journal guidelines forbid it. I've seen nonsensical reviews from people who obviously haven't read the paper beyond an LLM-generated summary. Area chairs are supposed to catch this, but I imagine they're using LLMs too.
- watwut 1mo ago
  > With the addition of AI, it should be easier than ever to review things on their merits. It makes it harder, because amount of bullshit goes up and it is easier to generate plausibly sounding bullshitm
wodenokoto 1mo ago
I don’t understand how asking another AI to paraphrase causes both watermarks to remain
- simon84 1mo ago
  From what I understand of the not-very-detailed text watermark, it comes up to masking the generated output with specific bias in the weights of the token generation (or something like this). Anthropic was explaining that it would survive edits (not full rewrite). So asking another AI to paraphrase will surely add its own, but since the bias is not disclosed, it will depend on how the 2nd AI is considering the existing tokens to steer its own. Fictive example: imagine the bias is to inject notion of colorfullness in the text regardless or its content: "I eat an apple" becomes "I eat a red apple" The 2nd AI is about injecting size component, so the text becomes " I eat a big red apple" There you get both watermarks. It is not that obvious obviously

by simon84 1mo ago

avaer 1mo ago

You'd think that the humans being paid to review these things are actually reviewing them. Journals, laywers etc. are expensive. With the addition of AI, it should be easier than ever to review things on their merits. Maybe some part of it is that the deluge of slop is uncovering how poorly/sloppily these social institutions were working in the first place.

truckyday 1mo ago
[dead]
- avaer 1mo ago
  Anecdote: I've had at two friends publish "papers" just by slapping their names on something they had literally nothing to do with, out of pure nepotism. This wouldn't be helped by an AI watermark. What would help is if the reviewer used AI to look up the authors and see the authorship claims are dubious. The papers are still up. I do think the academic publishing field is corrupt, which is why I'm not convinced it should be on AI providers to help bail it out of doing its one job (verification and trust).
  - maleldil 1mo ago
    Verifying authorship is an ambiguous task. How much involvement is necessary to qualify for authorship? If someone reads the paper and gives some small feedback, is that enough? What if they were present in one meeting and raised a question that turned out not to be interesting? What if they have no clue about the work but helped with data validation? I'd argue all of these could justify authorship, even if they're just in the middle of the author list. At least in NLP, which can be seen in the generally high number of authors in papers.
    - rcxdude 1mo ago
      The standards also vary by field. It's not unusual for supervisors to be the last author on any paper that a group publishes, even if they had basically no input into it directly. I've been listed on papers just because I designed and built the equipment that happened to be used for the experiments, even though I didn't do anything but a quick review of the actual paper. For most papers, unless there's an indication otherwise, it's generally only safe to assume the first author listed that actually did the bulk of the work and write-up, and the others are there mainly for having some potentially quite indirect contribution.
      - maleldil 1mo ago
        I agree completely. That's why I pushed back on "nothing to do" from the OP. Maybe it was a small contribution, but calling it "corruption" is ridiculous.
simon84 1mo ago
I think you are very correct here! Now imagine how much worse it can get with AI in the way. It is as always (think cybersecurity) the cat and mouse game, what is a weapon is also a defense. You, the simple fact that you are on this very site, means that you are probably more educated to AI than most, so it is not necessarily you that will really benefit from any constraining framework. There are many people believing in many weird theories and those are more prone to be convinced by a nice narrative. AI did not introduce that, it just made it easier and available to anyone with any intent.
maleldil 1mo ago
Peer reviews is largely done for free. In some cases, an author needs to commit to reviewing someone else's work for their work to be reviewed. However, this has led to an increase in LLM usage for review generation, even if conference/journal guidelines forbid it. I've seen nonsensical reviews from people who obviously haven't read the paper beyond an LLM-generated summary. Area chairs are supposed to catch this, but I imagine they're using LLMs too.
watwut 1mo ago
> With the addition of AI, it should be easier than ever to review things on their merits. It makes it harder, because amount of bullshit goes up and it is easier to generate plausibly sounding bullshitm

wodenokoto 1mo ago

I don’t understand how asking another AI to paraphrase causes both watermarks to remain

simon84 1mo ago
From what I understand of the not-very-detailed text watermark, it comes up to masking the generated output with specific bias in the weights of the token generation (or something like this). Anthropic was explaining that it would survive edits (not full rewrite). So asking another AI to paraphrase will surely add its own, but since the bias is not disclosed, it will depend on how the 2nd AI is considering the existing tokens to steer its own. Fictive example: imagine the bias is to inject notion of colorfullness in the text regardless or its content: "I eat an apple" becomes "I eat a red apple" The 2nd AI is about injecting size component, so the text becomes " I eat a big red apple" There you get both watermarks. It is not that obvious obviously