29 ms·
Doesn't Wikipedia offer full tarballs? This would imaginably put some downward pressure on scraper volume.
by jddj 1y ago
Doesn't Wikipedia offer full tarballs?
This would imaginably put some downward pressure on scraper volume.
- tolmasky 1y agoFrom the first paragraph in my comment: > You'd see them taking some additional steps to combat this. They haven't. Their CDN handles it just fine. They don't even both telling AI bots to just download the tarballs they specifically make available for this exact use case. Yes, they do. But they aren't in a rush to tell AI companies this, because again, this is not actually a super meaningful amount of traffic increase for them.