3 ms·
does this obey robots.txt and have some sort of delay so it doesn't take down a site?
by AznHisoka 12y ago
does this obey robots.txt and have some sort of delay so it doesn't take down a site?
- cblock811 12y agoIt's a starting point for a crawler, and does not have limiters on robots.txt and sleeps yet. Users could easily add them in if want to crawl with it.