3 ms·

Rate limiting requires them to buy more IPs (which are not free) and slows down scraping.

• sunnybeetroot 3mo ago
  Yes, that’s what the big scrapers are doing (frontier AI labs). They have infinite money for this.