3 ms·

As mentioned by other comments, we will likely end up in an arms race of adversarial neural networks. Which will be opposed by neural nets driven by manual cla

by bin0 7y ago

As mentioned by other comments, we will likely end up in an arms race of adversarial neural networks. Which will be opposed by neural nets driven by manual classification of spam/not spam entries. However, I think the spammers have a decisive advantage. What you are dealing with here is an asymmetric situation: while sites must spend money on people to do classification, the spammers have an oracle against which their neural nets can verify responses. This means improving their bots to fight new anti-spam tech is easier. I can't say what the solution is here.

jsnell 7y ago
The asymmetries cut both ways. The site can detect probing attempts, and if that happens switch into a mode where the captcha results are obfuscated. Let's use blog comments as an example: The normal mode of operation is to give the user a clear error message if they failed the captcha, and have the post go through if they passed the captcha. If the number of failures is higher than e.g. 10 in the last hour, successful captchas cause the comment to be put in a hold queue for an hour. Failed captchas cause the comment to be rejected. The attacker can't know if they passed the captcha until an hour has passed, which slows down their iteration a lot. You can then slow it down further with some randomization, while keeping the experience of real users the same. E.g. successful captchas go to the hold queue. Failed captchas are rejected 99% of time time. The remaining 1% of the time they go to the hold queue, but are auto-deleted after a random delay of 2h-12h. So you accept a small amount of temporarily visible spam as the price of obfuscating the signals.
- bin0 7y ago
  > The attacker can't know if they passed the captcha until an hour has passed. Naah. The attacker just does batch learning: dump a ton of comments, wait an hour, spend a few minutes training, rinse-and-repeat. Your users also won't tolerate tons of delays. > You can then slow it down further with some randomization. So now I wait a few extra hours, and have 99% accurate data. That's still pretty good, honestly. Ideally, a neural network can get a good model from less accurate data than 99%; just tweak the learning rate or use sgd, modify mini-batch size, etc. The advantage of the site, of course, is that legitimate businesses are likely more able to afford people with this sort of knowledge. The scammers are probably script kiddies who clone a template and tweak a config; they'd have real jobs if they knew more.

by bin0 7y ago

jsnell 7y ago

The asymmetries cut both ways. The site can detect probing attempts, and if that happens switch into a mode where the captcha results are obfuscated. Let's use blog comments as an example: The normal mode of operation is to give the user a clear error message if they failed the captcha, and have the post go through if they passed the captcha. If the number of failures is higher than e.g. 10 in the last hour, successful captchas cause the comment to be put in a hold queue for an hour. Failed captchas cause the comment to be rejected. The attacker can't know if they passed the captcha until an hour has passed, which slows down their iteration a lot. You can then slow it down further with some randomization, while keeping the experience of real users the same. E.g. successful captchas go to the hold queue. Failed captchas are rejected 99% of time time. The remaining 1% of the time they go to the hold queue, but are auto-deleted after a random delay of 2h-12h. So you accept a small amount of temporarily visible spam as the price of obfuscating the signals.

bin0 7y ago
> The attacker can't know if they passed the captcha until an hour has passed. Naah. The attacker just does batch learning: dump a ton of comments, wait an hour, spend a few minutes training, rinse-and-repeat. Your users also won't tolerate tons of delays. > You can then slow it down further with some randomization. So now I wait a few extra hours, and have 99% accurate data. That's still pretty good, honestly. Ideally, a neural network can get a good model from less accurate data than 99%; just tweak the learning rate or use sgd, modify mini-batch size, etc. The advantage of the site, of course, is that legitimate businesses are likely more able to afford people with this sort of knowledge. The scammers are probably script kiddies who clone a template and tweak a config; they'd have real jobs if they knew more.