8 ms·

I wish there were more articles about determining the frequency at which one page should be crawled. Some pages never change, some change multiple times per min

by kaivi 11y ago

I wish there were more articles about determining the frequency at which one page should be crawled. Some pages never change, some change multiple times per minute, and we do not want to crawl them all equally often.

imaginenore 11y ago
It's relatively easy. First you crawl a page every hour. If it changed, you halve the time. If it hasn't, you double it. You set some limits, like once a minute to once a month. You can also adjust the multiplier, and instead of factor of 2 you use something like 1.2. This way you can adjust more precisely to the page's update time. Also headers and sitemap.XML can tell you how often the pages change.
petewailes 11y ago
This is a problem I've researched fairly extensively the last few months. My ideal solution looks something like: * Initial pull * Secondary pulls x time later, where x doubles each time, up to a maximum value, y y is the one that's tricky to define. For us, it's a value computed based on the frequency of update of similar URLs for that domain, the domain as a whole, similar content, and a few other bits and pieces. Essentially, our thinking is that if we can understand how alike any page is to another cluster of pages, we can use their average frequency of update to give reasonably likely initial values for x, and sensible thresholds for y. We also temper this with how much change there is, to determine whether the differences are something we care about. Obviously, should the system notice that if its change timings are particularly outside where it'd expect given the cohort assigned, it's then able to start moving around its comparison. An example would be a blog category page which updates so infrequently that it's particularly unusual, or a page with a lot of social feeds on it where there's a lot of flux constantly. Works pretty well, but if anyone's got a better solution I'd love to hear of it.
HaseebR7 11y ago
That is why you have sitemaps, for new websites with no sitemaps we could start with a default frequency, say, once a day. If it changes more often than that you can increase it to multiple times a day and so on. We can also look for the type of website to determine a starting frequency — for example an e-commerce websites ought to be crawled more frequently because new products, reviews, ratings could be added every few minutes whereas a personal blog may stay the same for weeks altogether.

by kaivi 11y ago

imaginenore 11y ago

It's relatively easy. First you crawl a page every hour. If it changed, you halve the time. If it hasn't, you double it. You set some limits, like once a minute to once a month. You can also adjust the multiplier, and instead of factor of 2 you use something like 1.2. This way you can adjust more precisely to the page's update time. Also headers and sitemap.XML can tell you how often the pages change.

petewailes 11y ago

This is a problem I've researched fairly extensively the last few months. My ideal solution looks something like: * Initial pull * Secondary pulls x time later, where x doubles each time, up to a maximum value, y y is the one that's tricky to define. For us, it's a value computed based on the frequency of update of similar URLs for that domain, the domain as a whole, similar content, and a few other bits and pieces. Essentially, our thinking is that if we can understand how alike any page is to another cluster of pages, we can use their average frequency of update to give reasonably likely initial values for x, and sensible thresholds for y. We also temper this with how much change there is, to determine whether the differences are something we care about. Obviously, should the system notice that if its change timings are particularly outside where it'd expect given the cohort assigned, it's then able to start moving around its comparison. An example would be a blog category page which updates so infrequently that it's particularly unusual, or a page with a lot of social feeds on it where there's a lot of flux constantly. Works pretty well, but if anyone's got a better solution I'd love to hear of it.

HaseebR7 11y ago

That is why you have sitemaps, for new websites with no sitemaps we could start with a default frequency, say, once a day. If it changes more often than that you can increase it to multiple times a day and so on. We can also look for the type of website to determine a starting frequency — for example an e-commerce websites ought to be crawled more frequently because new products, reviews, ratings could be added every few minutes whereas a personal blog may stay the same for weeks altogether.