3 ms·
A combination of shotscraper and metascraper; really more web previews than archives. And in a single thread, to different hostnames, maybe one every 10 seconds
by NelsonMinar 4y ago
A combination of shotscraper and metascraper; really more web previews than archives. And in a single thread, to different hostnames, maybe one every 10 seconds? Honestly surprised Akamai or anything even noticed. I fake my user agent now, lesson learned.