5 ms·
No mention of observing robots.txt?
by eli 16y ago
No mention of observing robots.txt?
- megamark16 16y agoIt was my understanding the urllib2 respects robots.txt automatically. I can't find much to back that up, but I really thought I read that somewhere reliable once.
- mcav 16y agoThis link corroborates that urllib2 respects robot.txt: http://stackoverflow.com/questions/3197299/urllib2-connection-timed-out-error http://stackoverflow.com/questions/3197299/urllib2-connectio...
- deleted 16y ago[deleted]
- arst 16y agoThat link is wrong. Python does ship with a robotparser module in the standard library that parses robots.txt files, but urllib2 does not use it out of the box. This can be easily confirmed using Wireshark or a quick glance at the source: http://hg.python.org/cpython/file/08b5e2c9112c/Lib/urllib2.py http://hg.python.org/cpython/file/08b5e2c9112c/Lib/urllib2.p....
- Groxx 16y agoIf you're scraping, and explicitly "behaving" like a human, you're probably grabbing things which aren't exposed in a robot-friendly format. I.e., technically, you should. Practically, you can't all the time.