2 ms·
I think those crawlers are just very generic: they basically operate like wget scripts, without much logic for avoiding sites that already offer clean data dump
by wslh 2y ago
I think those crawlers are just very generic: they basically operate like wget scripts, without much logic for avoiding sites that already offer clean data dumps.
- ldng 2y agoThat is not an excuse. Wikipedia isn't just any site.
- wslh 2y agoNot an excuse, a plausible explanation of what's actually happening.
- franktankbank 2y agoAlso plausibly they are trying to kill the site via soft ddos. Then they can sell a service based on all the data they scraped + unauditable censoring.