4 ms·

I'm convinced that Google has several Googlebots that are run depending on how popular a site is. That is, new and low traffic sites are crawled by less intell

I'm convinced that Google has several Googlebots that are run depending on how popular a site is. That is, new and low traffic sites are crawled by less intelligent bots, and as the site gets more visitors or better rankings, more complicated and resource intensive bots are deployed. How this might work with the most popular sites out there, the Amazons and Wikipedias of this world - I'm not so sure about that. If I were in charge, I'd be tempted to have customised bots and ranking weights for each of these exceptional sites. Sadly the chances of getting a real answer on this in my lifetime are close to zero.

manojlds 10y ago
You could join Google :)
- dx034 10y ago
  The chances of becoming a part of the search team are still very low. Even most Googlers won't know the exact details.
  - tr1ck5t3r 10y ago
    Parse your content, then to finalise your page ranking, listen to what former Defence Secretary Donald Rumsfeld had to say. I am a Robot. https://en.wikipedia.org/wiki/There_are_known_knowns https://en.wikipedia.org/wiki/There_are_known_knowns
  - lern_too_spel 10y ago
    But within Google, you could ask somebody and get the right answer.
    - user5994461 10y ago
      And outside Google, you could look for Google employees on linkedin who may be in position to have the answer and ask nicely ;)
  - tehlike 10y ago
    no it is not :) you can move around in the company.
ncouture 10y ago
You're at least partly right about sites being crawled differently depending on popularity. I think the factors may not be limited to popularity alone, but we see this behavior documented in the crawling rate documentation Google provides its users/clients so there is no reason it couldn't apply to other "expensive" actions their crawlers do. How this might work with the most popular sites out there? We see it in on-page answers that provide extracts of pages with the answers to questions asked in search phrases that include a reference to the document they were sourced from. Matt Cutts used to qualify sites like Wikipedia as "reputable" to the eyes of the search engine.
dom0 10y ago
Or perhaps there are also heuristics in place to determine which strategy to follow, ie. heuristics to see whether executing JS would be worth it, would yield additional content. So, say, when crawling documentation, where JS doesn't give any of that (eg. Sphinx' JS search), it could decide - nah, not doing JS, not worth it. I'd expect that there are also other heuristics and different strategies for crawling to better handle eg. content presented by one of the popular CMSes.
- valarauca1 10y ago
  heuristics to see whether executing JS would be worth it, would yield additional content. You are literally describing vanilla page rank. If a large number of links are found to a page, but that page doesn't contain the contents the link rate suggest it should contain... either link rate has failed, or JavaScript should be executed.
matt4077 10y ago
I recently put a little browser for themes for the hyper.app terminal online (https://hyperthemes.matthi.coffee https://hyperthemes.matthi.coffee). It was just something I used to try out Elm, and there's no reason to believe Google would regard it as anything different than a generic new site. If you look at Google's cached version, you can see that the JS is executed (although it fails trying to download the actual data): https://webcache.googleusercontent.com/search?q=cache:hN5yCkyIyzkJ:https://hyperthemes.matthi.coffee/+&cd=3&hl=en&ct=clnk https://webcache.googleusercontent.com/search?q=cache:hN5yCk... Edit: as has been pointed out below, the cached version is just the same as the original and the JS gets executed on your end. This doesn't show weather Google also executes it during its crawl.
- gildas 10y ago
  > If you look at Google's cached version, you can see that the JS is executed Correct me if I am wrong but when I look at the cached version of the homepage (http://webcache.googleusercontent.com/search?q=cache:hN5yCkyIyzkJ:https://hyperthemes.matthi.coffee/&num=1&hl=en&gl=fr&strip=0&vwsrc=1 http://webcache.googleusercontent.com/search?q=cache:hN5yCky...), I don't see that the JS has been interpreted.
  - matt4077 10y ago
    The "Network error" in the top left is a JS result. Also, since you linked to the source view, you'll see that there actually isn't anything in the body but the script being loaded, whereas "Full version" shows the user interface was correctly initialised.
    - gildas 10y ago
      Disable JavaScript in your browser when viewing the cached page. All you will see is an empty page.
      - matt4077 10y ago
        Now I'm feeling a bit stupid, will slowly walk away and hope I can get by with "it's Jan 1st and I didn't get much sleep" as an excuse...

manojlds 10y ago

You could join Google :)

dx034 10y ago
The chances of becoming a part of the search team are still very low. Even most Googlers won't know the exact details.
- tr1ck5t3r 10y ago
  Parse your content, then to finalise your page ranking, listen to what former Defence Secretary Donald Rumsfeld had to say. I am a Robot. https://en.wikipedia.org/wiki/There_are_known_knowns https://en.wikipedia.org/wiki/There_are_known_knowns
- lern_too_spel 10y ago
  But within Google, you could ask somebody and get the right answer.
  - user5994461 10y ago
    And outside Google, you could look for Google employees on linkedin who may be in position to have the answer and ask nicely ;)
- tehlike 10y ago
  no it is not :) you can move around in the company.

ncouture 10y ago

You're at least partly right about sites being crawled differently depending on popularity. I think the factors may not be limited to popularity alone, but we see this behavior documented in the crawling rate documentation Google provides its users/clients so there is no reason it couldn't apply to other "expensive" actions their crawlers do. How this might work with the most popular sites out there? We see it in on-page answers that provide extracts of pages with the answers to questions asked in search phrases that include a reference to the document they were sourced from. Matt Cutts used to qualify sites like Wikipedia as "reputable" to the eyes of the search engine.

dom0 10y ago

Or perhaps there are also heuristics in place to determine which strategy to follow, ie. heuristics to see whether executing JS would be worth it, would yield additional content. So, say, when crawling documentation, where JS doesn't give any of that (eg. Sphinx' JS search), it could decide - nah, not doing JS, not worth it. I'd expect that there are also other heuristics and different strategies for crawling to better handle eg. content presented by one of the popular CMSes.

valarauca1 10y ago
heuristics to see whether executing JS would be worth it, would yield additional content. You are literally describing vanilla page rank. If a large number of links are found to a page, but that page doesn't contain the contents the link rate suggest it should contain... either link rate has failed, or JavaScript should be executed.

matt4077 10y ago

I recently put a little browser for themes for the hyper.app terminal online (https://hyperthemes.matthi.coffee https://hyperthemes.matthi.coffee). It was just something I used to try out Elm, and there's no reason to believe Google would regard it as anything different than a generic new site. If you look at Google's cached version, you can see that the JS is executed (although it fails trying to download the actual data): https://webcache.googleusercontent.com/search?q=cache:hN5yCkyIyzkJ:https://hyperthemes.matthi.coffee/+&cd=3&hl=en&ct=clnk https://webcache.googleusercontent.com/search?q=cache:hN5yCk... Edit: as has been pointed out below, the cached version is just the same as the original and the JS gets executed on your end. This doesn't show weather Google also executes it during its crawl.

gildas 10y ago
> If you look at Google's cached version, you can see that the JS is executed Correct me if I am wrong but when I look at the cached version of the homepage (http://webcache.googleusercontent.com/search?q=cache:hN5yCkyIyzkJ:https://hyperthemes.matthi.coffee/&num=1&hl=en&gl=fr&strip=0&vwsrc=1 http://webcache.googleusercontent.com/search?q=cache:hN5yCky...), I don't see that the JS has been interpreted.
- matt4077 10y ago
  The "Network error" in the top left is a JS result. Also, since you linked to the source view, you'll see that there actually isn't anything in the body but the script being loaded, whereas "Full version" shows the user interface was correctly initialised.
  - gildas 10y ago
    Disable JavaScript in your browser when viewing the cached page. All you will see is an empty page.
    - matt4077 10y ago
      Now I'm feeling a bit stupid, will slowly walk away and hope I can get by with "it's Jan 1st and I didn't get much sleep" as an excuse...