3 ms·Show HN: Domain Crawler2 points by cblock811 12y agoAznHisoka 12y agodoes this obey robots.txt and have some sort of delay so it doesn't take down a site?cblock811 12y agoIt's a starting point for a crawler, and does not have limiters on robots.txt and sleeps yet. Users could easily add them in if want to crawl with it.