4 ms·Scraping the sh*t out of the interwebz – Part #15 points by lepunk 14y agoemilv 14y agoDon't forget to set a good user-agent header and follow the rules set forth by robots.txt!