2 ms·
It's a starting point for a crawler, and does not have limiters on robots.txt and sleeps yet. Users could easily add them in if want to crawl with it.
by cblock811 12y ago
It's a starting point for a crawler, and does not have limiters on robots.txt and sleeps yet. Users could easily add them in if want to crawl with it.