4 ms·
Rate limiters can also help scrape a site that doesn't like it, like Facebook. We use them to keep our scraping rate below the service's own rate limits. But t
by barfbagginus 2y ago
Rate limiters can also help scrape a site that doesn't like it, like Facebook. We use them to keep our scraping rate below the service's own rate limits.
But the simple algorithms are not at all covert. We assume companies can deploy rate limiting detectors. So we ask, are there rate limiting algorithms designed to evade detection?
In one solution, we train an HMM on the inter-request times of a human user. This gives us a much more covert rate limiter! But how does it perform against state of the art detectors?