4 ms·

I think what will happen is that there effectively be an arms race between the bot generation and detection sides. As anomalies are discovered in the distributi

I think what will happen is that there effectively be an arms race between the bot generation and detection sides. As anomalies are discovered in the distribution of content generated by bots vs humans, bots will update to remove the anomaly. And researchers will try to find new anomalies. In the end as bots get better, the difference in the two distributions will be indistinguishable in practice.

marginalia_nu 3y ago
Possible, sure. Likely, maybe. But you made it sound inevitable? Of that I am less sure.
- vidarh 3y ago
  For any accessible detector you can automate the evasion. E.g, crudest variant would be: generate text run it through the detector finetune model w/result of the detector repeat until detector fails to detect. But every time the failure rate of the detector starts getting high, the other side has a lot of development work to do finding new patterns that won't also lead to lots of false positives. So while getting a point where it's totally impossible to detect may well take a long time, it'll be periods of the detector suddenly being hard to bypass for a while, and gradually getting easier, repeat. It's highly unlikely it'll ever be impossible to bypass for any length of time. The only way of keeping such a detector working for any length of time will be to make it expensive or very hard for adversaries to get access to, which means effectively relying on only giving access to trusted users.
  - _v7gu 3y ago
    Isn't this basically how autoencoder models are trained?
  - marginalia_nu 3y ago
    This assumes the ability to perfectly model this function, which is in a sense what we're setting out to demonstrate. It can certainly not be taken for granted.
    - vidarh 3y ago
      Firstly, it doesn't need to perfectly model it. It needs to model it to within a range where claiming to have detected it causes more false positives than acceptable, and given the range of quality of human output that's a pretty wide span. But secondly, LLMs wired up with a feedback loop and and appropriate IO can model Markov models (models, not chains) which are Turing complete, so a LLM wired up this way is clearly Turing complete as well. As such it's a question of training, not whether or not the model can model it.

marginalia_nu 3y ago

Possible, sure. Likely, maybe. But you made it sound inevitable? Of that I am less sure.

vidarh 3y ago
For any accessible detector you can automate the evasion. E.g, crudest variant would be: generate text run it through the detector finetune model w/result of the detector repeat until detector fails to detect. But every time the failure rate of the detector starts getting high, the other side has a lot of development work to do finding new patterns that won't also lead to lots of false positives. So while getting a point where it's totally impossible to detect may well take a long time, it'll be periods of the detector suddenly being hard to bypass for a while, and gradually getting easier, repeat. It's highly unlikely it'll ever be impossible to bypass for any length of time. The only way of keeping such a detector working for any length of time will be to make it expensive or very hard for adversaries to get access to, which means effectively relying on only giving access to trusted users.
- _v7gu 3y ago
  Isn't this basically how autoencoder models are trained?
- marginalia_nu 3y ago
  This assumes the ability to perfectly model this function, which is in a sense what we're setting out to demonstrate. It can certainly not be taken for granted.
  - vidarh 3y ago
    Firstly, it doesn't need to perfectly model it. It needs to model it to within a range where claiming to have detected it causes more false positives than acceptable, and given the range of quality of human output that's a pretty wide span. But secondly, LLMs wired up with a feedback loop and and appropriate IO can model Markov models (models, not chains) which are Turing complete, so a LLM wired up this way is clearly Turing complete as well. As such it's a question of training, not whether or not the model can model it.