3 ms·
What's this X-User-Agent stuff? The entire internet isn't going to special-case your hack. Just honour robots.txt.
by JoachimSchipper 16y ago
What's this X-User-Agent stuff? The entire internet isn't going to special-case your hack. Just honour robots.txt.
- spullara 16y agoI don't that a individual storing web pages has any requirement to obey robots.txt. You should be able to click "Save Page" on anything you have access to. Obviously redistributing that is another can of worms.