3 ms·
Or perhaps there are also heuristics in place to determine which strategy to follow, ie. heuristics to see whether executing JS would be worth it, would yield a
by dom0 10y ago
Or perhaps there are also heuristics in place to determine which strategy to follow, ie. heuristics to see whether executing JS would be worth it, would yield additional content. So, say, when crawling documentation, where JS doesn't give any of that (eg. Sphinx' JS search), it could decide - nah, not doing JS, not worth it.
I'd expect that there are also other heuristics and different strategies for crawling to better handle eg. content presented by one of the popular CMSes.
- valarauca1 10y agoheuristics to see whether executing JS would be worth it, would yield additional content. You are literally describing vanilla page rank. If a large number of links are found to a page, but that page doesn't contain the contents the link rate suggest it should contain... either link rate has failed, or JavaScript should be executed.