3 ms·

Why not just rate limit every user to realistic human rates. You just punish anyone behaving like a bot.

• guerrilla 2y ago
  Sold. Pay by page retrieval rate.
• mrweasel 2y ago
  Because, as pointed out in another post about the same problem: Many of these scrappers make one or two requests from one IP and then move on.