4 ms·Simple web-crawler for Node.js2 points by DamonOehlman 14y agozoowar 14y agoShow some respect for the web and add robots.txt and sitemap.xml support.