3 ms·

I think sending back a `429 Too Many Requests` can solve this. Unless you want to block their IPs completely, which I doubt you'd want to do.

• gramakri 6y ago
  It says in the post he did this already
• napolux 6y ago
  I tried. It was ignored.
• klohto 6y ago
  > HTTP 429 -> ignored It's in the article
• jandrese 6y ago
  A well made crawler that would support 429 codes isn't likely to have this problem in the first place.