3 ms·
This link corroborates that urllib2 respects robot.txt: http://stackoverflow.com/questions/3197299/urllib2-connection-timed-out-error http://stackoverflow.com/
by mcav 15y ago
This link corroborates that urllib2 respects robot.txt:
http://stackoverflow.com/questions/3197299/urllib2-connection-timed-out-error http://stackoverflow.com/questions/3197299/urllib2-connectio...
- deleted 15y ago[deleted]
- arst 15y agoThat link is wrong. Python does ship with a robotparser module in the standard library that parses robots.txt files, but urllib2 does not use it out of the box. This can be easily confirmed using Wireshark or a quick glance at the source: http://hg.python.org/cpython/file/08b5e2c9112c/Lib/urllib2.py http://hg.python.org/cpython/file/08b5e2c9112c/Lib/urllib2.p....