4 ms·
Also the Facebook hit scraper. Which does not respect robots.txt and definitely is just scraping. AS blocks are the only really effective tool now, there are
by buro9 2y ago
Also the Facebook hit scraper.
Which does not respect robots.txt and definitely is just scraping.
AS blocks are the only really effective tool now, there are many scrapers that do not even respect user agent
- wiredfool 2y agoFacebook and someone who’s using a Firefox UA are the big hitters for me today, each at a sustained 6 req per second for the last 24 hours on one site. Today is actually pretty good, there’s some real looking UA traffic in the top 10.