You may want to use countermeasures that aren't so easily detected by the bot owners. You want to lengthen their feedback loop. They know instantly when you blo

You may want to use countermeasures that aren't so easily detected by the bot owners. You want to lengthen their feedback loop. They know instantly when you block them and can just move to a different server at amazon. For example you could put them in a tarpit, and gradually serve successive pages more and more slowly. Exponential backoff or something. You could also serve bad data back to them when they get so far into the tarpit. It all depends on how much work it's worth doing. Id love to hear more ideas, it's a fascinating question. So many sites have this problem, and its an endless cat and mouse game, perhaps there is a product opportunity in here somewhere.