2 ms·
Last-Modified/Etags are your friends for detecting modifications. Having said that, link orientation may not matter if they have to run a headless browser rend
by supriyo-biswas 3y ago
Last-Modified/Etags are your friends for detecting modifications.
Having said that, link orientation may not matter if they have to run a headless browser rendering such performance optimizations moot.
- marginalia_nu 3y agoI don't think it makes economic sense to crawl everything with headless browsers, given it's several orders of magnitude more expensive, and in most cases doesn't actually give you different results. (Many websites that seem like they'd not be crawlable with a dumb crawler, e.g. Medium, actually crawl just fine as long as you don't masquerade the U-A to look like Chrome) Makes a lot more sense to sample, identify and specifically crawl the sites where a classic retrieval doesn't work with headless browser, and do classic crawling elsewhere.