3 ms·
> The consequence will almost universal blocks otherwise! Who cares? They've already scraped the content by then.
by Vampiero 2y ago
> The consequence will almost universal blocks otherwise!
Who cares? They've already scraped the content by then.
- jsheard 2y agoBold to assume that an AI scraper won't come back to download everything again, just in case there's any new scraps of data to extract. OP mentioned in the other thread that this bot had pulled 3TB so far, and I doubt their git server actually has 3TB of unique data, so the bot is probably pulling the same data over and over again.
- xena 2y agoFWIW that includes other scrapers, Amazon's is just the one that showed up the most in the logs.
- _heimdall 2y agoIf they only needed a one-time scrape we really wouldn't be seeing noticeable not traffic today.
- seethenerdz 2y agoThat's the spirit!