3 ms·

To state the obvious, detecting anything generated with a neural net by using another neural net is an arms race because the generator was trained explicitly to

To state the obvious, detecting anything generated with a neural net by using another neural net is an arms race because the generator was trained explicitly to fool a detector... If you get your detector working well enough today, next week is another story.

rat9988 7y ago
There is a fallacy in your reasoning as you discount one side's effort and focus on the others. >If you get your detector working well enough today, next week is another story. Yes, you need to do some work to be able to catch up. And you have no guarantee you'll succeed. > the generator was trained explicitly to fool a detector Here again you have some work to do to fool the detector, and you have no guarantee of success either each time the detector gets smarter. Moreover, the generator wasn't trained explicitly to fool a detector but to fool humans. On the other side, the detector was trained explicitly to unmask the generator.
- beagle3 7y ago
  > the generator wasn't trained explicitly to fool a detector but to fool humans. Yes, but “next week”’s one will be trained to fool humans AND this week’s Grover - which is still likely to be easy. Next week’s Grover will Be harder to get right against this model. But will likely be built. The faker model of two weeks from now will be harder to build (against the updated week Grover), and so on. There is no fallacy here. An attacker usually has easier time than a defender, because the defender must respond to every attack whereas an attacker can vary and pick every detail of their attack.
gwern 7y ago
Not in this case, as Grover/GPT-2 are pure max likelihood/predictive loss training; they don't use anything like use one Grover to finetune another. So no adversarial dynamics... yet.
- etaioinshrdlu 7y ago
  Very interesting. I was wrong to assume they were adversarially trained.
- tfha 7y ago
  Point is that we can't really rely on these tools at all, especially because there are already false positives and any rate of false positives makes the tools essentially worthless.

rat9988 7y ago

There is a fallacy in your reasoning as you discount one side's effort and focus on the others. >If you get your detector working well enough today, next week is another story. Yes, you need to do some work to be able to catch up. And you have no guarantee you'll succeed. > the generator was trained explicitly to fool a detector Here again you have some work to do to fool the detector, and you have no guarantee of success either each time the detector gets smarter. Moreover, the generator wasn't trained explicitly to fool a detector but to fool humans. On the other side, the detector was trained explicitly to unmask the generator.

beagle3 7y ago
> the generator wasn't trained explicitly to fool a detector but to fool humans. Yes, but “next week”’s one will be trained to fool humans AND this week’s Grover - which is still likely to be easy. Next week’s Grover will Be harder to get right against this model. But will likely be built. The faker model of two weeks from now will be harder to build (against the updated week Grover), and so on. There is no fallacy here. An attacker usually has easier time than a defender, because the defender must respond to every attack whereas an attacker can vary and pick every detail of their attack.

gwern 7y ago

Not in this case, as Grover/GPT-2 are pure max likelihood/predictive loss training; they don't use anything like use one Grover to finetune another. So no adversarial dynamics... yet.

etaioinshrdlu 7y ago
Very interesting. I was wrong to assume they were adversarially trained.
tfha 7y ago
Point is that we can't really rely on these tools at all, especially because there are already false positives and any rate of false positives makes the tools essentially worthless.