4 ms·
Last months saw an example of a distributed database/api service scraping, using hundreds of thousands of IPs of around the world, and random user agents (I thi
by gmuslera 10y ago
Last months saw an example of a distributed database/api service scraping, using hundreds of thousands of IPs of around the world, and random user agents (I think that all started with Mozilla/5.0, but the rest were from random historical user agents), to avoid rate limits.
The OSs referenced in those user agents went back to early windows versions, OS/2 Warp and Maemo (to put 2 examples that hurt knowing that were fake).
If this site take as base sites/URLs that are somewhat hit by that kind of traffic the stats could be very misleading.
- herbst 10y agoYeah. Where is "vitaly rules" in this graph? I have medium sizes sites where that is 2-7% of the traffic right now.