3 ms·
Well, most of the time spiders are idle while waiting for IO operations to complete. Main bottleneck is not CPU, but RAM and bandwidth.
by Dolores12 9y ago
Well, most of the time spiders are idle while waiting for IO operations to complete. Main bottleneck is not CPU, but RAM and bandwidth.
- rileymat2 9y agoIt depends on what analysis is being done on each page.
- Dolores12 9y agoWhat kind of analysis are you expecting from a crawler which main purpose is to grab a webpage?
- ivanhoe 9y agoMain purpose is to render the page, crawl the dom for the data, and then load the next page as fast as possible... so it's equivalent of having a thousand bookmarks and opening them all at the same time, and as soon each loads running the scraper script and reloading them with the new link. Try it for yourself, it's easy to script, and check the load...
- rileymat2 9y agoPresumably, there is some reason you are crawling.