3 ms·

There are a few things that I hope you just left out in your description... - There is no mention of implementing a crawl-delay. You should always wait for sev

by AcoonDe 13y ago

There are a few things that I hope you just left out in your description... - There is no mention of implementing a crawl-delay. You should always wait for several seconds (better yet, a minute) between requests to the same host. - Do you follow redirects when requesting the robots.txt? You should! Some sites send you a redirect to a different URL even for robots.txt. In most cases it is just a slightly different hostname, like www.domain.com instead of domain.com. But it can redirect you to somewhere completely different in some cases. - You probably don't want to crawl anything that ends with .jpg, .gif, and definitely not something like .avi, .wmv or .mkv. There are a LOT more file-extensions that you'll want to ignore. I agree with cmiles74 that using a database is probably a bad idea. For a sizeable crawl (say a billion pages) this database will get pretty damn big. I doubt that you will be able to get decent performance out of anything with "SQL" in its name for such a use-case, unless you throw a ton of hardware at it. Building your own specialized solution for this would probably be a lot faster and less resource-intensive.

at-fates-hands 13y ago
>>>You should always wait for several seconds (better yet, a minute) between requests to the same host. This a thousand times. I learned this lesson AFTER getting several angry emails from Admins and getting outright banned from one site for not having any delay between the requests in the first crawler I built.
ebol4 13y ago
>There are a LOT more file-extensions that you'll want to ignore. It'd probably be a good idea just to whitelist filetypes that you actually want to crawl, rather than trying to blacklist all the ones you don't.
drakaal 13y ago
For http://www.samuru.com http://www.samuru.com we store our crawl in Google's DataStore. It gets big. By not storing all of the page, (we do content extraction first) we save quite a bit. Typically the content has much less markup than the template does. We extract other things like the Opengraph image and social media links for author and store them independent of the content.
boyter 13y ago
Its also missing some form of lock to prevent slow requests cascading (to the same host or IP) when the next instance of your crawler comes along. I did this once where I had a crawler spin up in a cron job. No problem, unless the previous crawler was still running. Speaking from experience its far better to lock your crawler down from the beginning then to do it after you kill someones site. I did expose a few slow pages for some sites, but not before crashing the whole thing.

by AcoonDe 13y ago

at-fates-hands 13y ago

>>>You should always wait for several seconds (better yet, a minute) between requests to the same host. This a thousand times. I learned this lesson AFTER getting several angry emails from Admins and getting outright banned from one site for not having any delay between the requests in the first crawler I built.

ebol4 13y ago

>There are a LOT more file-extensions that you'll want to ignore. It'd probably be a good idea just to whitelist filetypes that you actually want to crawl, rather than trying to blacklist all the ones you don't.

drakaal 13y ago

For http://www.samuru.com http://www.samuru.com we store our crawl in Google's DataStore. It gets big. By not storing all of the page, (we do content extraction first) we save quite a bit. Typically the content has much less markup than the template does. We extract other things like the Opengraph image and social media links for author and store them independent of the content.

boyter 13y ago

Its also missing some form of lock to prevent slow requests cascading (to the same host or IP) when the next instance of your crawler comes along. I did this once where I had a crawler spin up in a cron job. No problem, unless the previous crawler was still running. Speaking from experience its far better to lock your crawler down from the beginning then to do it after you kill someones site. I did expose a few slow pages for some sites, but not before crashing the whole thing.