3 ms·
As it only does one URL at a time and uploads to S3 between requests, it shouldn't unduly load any reasonable system. I'll add an additional "BrowserCrawler" st
by spullara 16y ago
As it only does one URL at a time and uploads to S3 between requests, it shouldn't unduly load any reasonable system. I'll add an additional "BrowserCrawler" string to the user-agent, that seems very reasonable.
Update: JQuery can't set the User-Agent header on ajax requests it appears. I have instead set a new X-User-Agent header to BrowserCrawler.
- ryan-allen 16y agoAh, I'm familiar with S3 latencies. That'd work. Also, if you're uploading other people's content to public S3 buckets, you're very likely to be in violation of mosts site's copyright and TOS. You might want to mention that in the README. If we found some of our sites mirrored on the public internet we'd issue cease and desist notices.
- JoachimSchipper 16y agoWhat's this X-User-Agent stuff? The entire internet isn't going to special-case your hack. Just honour robots.txt.
- spullara 16y agoI don't that a individual storing web pages has any requirement to obey robots.txt. You should be able to click "Save Page" on anything you have access to. Obviously redistributing that is another can of worms.