4 ms·

Googlebot has done various amounts of JS parsing/execution for a while now. They've also issued similar webmaster guidelines in the past (e.g., don't use robot

Googlebot has done various amounts of JS parsing/execution for a while now. They've also issued similar webmaster guidelines in the past (e.g., don't use robots.txt to block crawling of scripts and styles). 2008: http://moz.com/ugc/new-reality-google-follows-links-in-javascript-4930 http://moz.com/ugc/new-reality-google-follows-links-in-javas... 2009: http://www.labnol.org/internet/search/googlebot-executes-javascript-on-web-pages/8040/ http://www.labnol.org/internet/search/googlebot-executes-jav... 2011: https://twitter.com/mattcutts/status/131425949597179904 https://twitter.com/mattcutts/status/131425949597179904 2012: http://www.thegooglecache.com/white-hat-seo/googlebots-javascript-interpreter-a-diagnostic/ http://www.thegooglecache.com/white-hat-seo/googlebots-javas... From the 2012 article: "Google is actually interpreting the Javascript it spiders. It is not merely trying to extract strings of text and it does appear to be nuanced enough to know what text is and is not added to the Document Object Model."

ChuckMcM 12y ago
I was surprised, its kind of old news. There is useful JS to look at when you're crawling a web page. Its also a useful way to detect malware on a page.
- thatthatis 12y ago
  My understanding is that the state before was: Content augmented with JS -- OK to index. Blank page with all content rendered by JS -- not indexed And what they're saying today is: "we are now going to be able to index single page apps that don't have any server rendered content" Perhaps I've missed something, but this is my interpretation: SPAs are now full citizens in the SEO world.
  - KMag 12y ago
    Google has understood pages that have had only client-side rendered content for years. I worked mostly on JavaScript execution in the indexing system from 2006 until 2010, as well as other "rich content" indexing. Certainly somewhere in the 2008 to 2009 timeframe we saw that the Chinese language version of the Wall Street Journal had a lot of pages in their archive where all of the non-boilerplate content was rendered via JavaScript. Since they didn't do this with their English content, it didn't seem to be an attempt to hide content from search engines, but much more likely a workaround for an older browser that wouldn't properly render Unicode, but with a JavaScript engine that would properly render Unicode. Sometime in the 2008 to 2009 timeframe, Google's indexing system started understanding text that was written into documents from JavaScript body onload handlers, and the Chinese Wall Street Journal archive content was exhibit A in my argument that my changes should be turned on in production. I'm sure they've increased the accuracy of the analysis since, but they've certainly been able to index content written by JavaScript for something like 5 years now. Edit: without giving away any Google secrets, here's a pretty good analysis of my work from 2008: http://moz.com/ugc/new-reality-google-follows-links-in-javascript-4930 http://moz.com/ugc/new-reality-google-follows-links-in-javas... Edit 2: Since the 2008-2009 timeframe, Google also notices when you use JavaScript to change a page's title. I caused a crash in Google's indexing system when I made a bad assumption about Google's HTML parser's handling of XHTML-style empty title tags <title/> and tried to construct negative-length std::strings from them. When your code runs on every single webpage that Google can find, you're certain to hit corner cases you didn't anticipate. I did test for empty <title></title>, but not <title/>, and made incorrect assumptions about the two pointers I'd get to the beginning and end of the title.
    - thatthatis 12y ago
      So would the more accurate interpretation be: google is starting to promote that it can and will index javascript single page apps? Thanks for sharing, and nice work btw.
      - KMag 12y ago
        There were a lot of caveats that both reduced fidelity and would have made announcements confusing back when I was at Google. Also, if one mentions a limitation in an announcement, a lot of the Search Engine Optimization community would be citing the announcement for more than 6 months. So, it's difficult and potentially counter-productive to have an announcement with lots of caveats. V8 and Chrome weren't even a glimmer in Google's eye back in 2006, so I hope they've largely replaced the code I was working on with something based on Chrome. As late as 2010, the DOM was a completely custom implementation that looked somewhat like Firefox, but with enough IE features to fool lots of other pages that would otherwise change their content to "You must run IE to view this page". (On a side note, as much as many people would like to see such pages heavily penalized and indexed as if the IE-only message were their only information, some of those pages are unique sources of invaluable information and users wouldn't be well served by such harsh treatment.)
    - juretriglav 12y ago
      Thanks for the insight! I know it's very off-topic, but seeing your comment about indexing title tags I have to ask if you have any idea what could be wrong here: http://stackoverflow.com/questions/23732242/how-to-get-google-to-index-a-dynamic-title-in-an-angular-js-app http://stackoverflow.com/questions/23732242/how-to-get-googl... Google seems to be ignoring the title changes our JS makes. Should we not have the title tag there in the first place and then add it in when the title is known?
  - davemel37 12y ago
    Everyone seems to be missing the point of this announcement. They are just saying that they are now adding a feature in Webmaster Tools to show you they see your site so you can diagnose problems.
    - thatthatis 12y ago
      Perhaps you haven't been privy to any of the discussions about: "what is the seo implication of SPAs?" Up until now people were pursuing isomorphic JavaScript so they could render server side and client side so the search engines would see their content. This "just a tool" lets webmasters see how google sees their pages, which means SPAs are becoming safe for SEO. That tool plus the song and dance about understanding the modern web is a pretty strong signal in an industry that operates heavily on rumors.
crucialfelix 12y ago
I think the difference is that they are officially announcing that they will do this at scale. Previously it was only a us geeks that knew this was happening, and only for some sites.
tonyhb 12y ago
Yeah, I remember about 2 years ago reading a blog post that analysed the origin of requests to an outdated API. Turns out, Google had crawled the site and was executing JS 17 (or so) days later. Hopefully this news is about them improving that two week lead time so that more modern websites have a better chance of being indexed.
- KMag 12y ago
  Within hours of being indexed, the JavaScript on the page would have been executed and the links extracted. Presumably these particular extracted links didn't look particularly interesting to the crawl scheduling algorithm, and sat in the crawl scheduling queue for 16 or 17 days. Source: I worked mostly on JavaScript execution in Google's indexing system from 2006 to 2010. I don't know the intricacies of the crawl scheduling, but I do know it's highly non-trivial.

ChuckMcM 12y ago

I was surprised, its kind of old news. There is useful JS to look at when you're crawling a web page. Its also a useful way to detect malware on a page.

thatthatis 12y ago
My understanding is that the state before was: Content augmented with JS -- OK to index. Blank page with all content rendered by JS -- not indexed And what they're saying today is: "we are now going to be able to index single page apps that don't have any server rendered content" Perhaps I've missed something, but this is my interpretation: SPAs are now full citizens in the SEO world.
- KMag 12y ago
  Google has understood pages that have had only client-side rendered content for years. I worked mostly on JavaScript execution in the indexing system from 2006 until 2010, as well as other "rich content" indexing. Certainly somewhere in the 2008 to 2009 timeframe we saw that the Chinese language version of the Wall Street Journal had a lot of pages in their archive where all of the non-boilerplate content was rendered via JavaScript. Since they didn't do this with their English content, it didn't seem to be an attempt to hide content from search engines, but much more likely a workaround for an older browser that wouldn't properly render Unicode, but with a JavaScript engine that would properly render Unicode. Sometime in the 2008 to 2009 timeframe, Google's indexing system started understanding text that was written into documents from JavaScript body onload handlers, and the Chinese Wall Street Journal archive content was exhibit A in my argument that my changes should be turned on in production. I'm sure they've increased the accuracy of the analysis since, but they've certainly been able to index content written by JavaScript for something like 5 years now. Edit: without giving away any Google secrets, here's a pretty good analysis of my work from 2008: http://moz.com/ugc/new-reality-google-follows-links-in-javascript-4930 http://moz.com/ugc/new-reality-google-follows-links-in-javas... Edit 2: Since the 2008-2009 timeframe, Google also notices when you use JavaScript to change a page's title. I caused a crash in Google's indexing system when I made a bad assumption about Google's HTML parser's handling of XHTML-style empty title tags <title/> and tried to construct negative-length std::strings from them. When your code runs on every single webpage that Google can find, you're certain to hit corner cases you didn't anticipate. I did test for empty <title></title>, but not <title/>, and made incorrect assumptions about the two pointers I'd get to the beginning and end of the title.
  - thatthatis 12y ago
    So would the more accurate interpretation be: google is starting to promote that it can and will index javascript single page apps? Thanks for sharing, and nice work btw.
    - KMag 12y ago
      There were a lot of caveats that both reduced fidelity and would have made announcements confusing back when I was at Google. Also, if one mentions a limitation in an announcement, a lot of the Search Engine Optimization community would be citing the announcement for more than 6 months. So, it's difficult and potentially counter-productive to have an announcement with lots of caveats. V8 and Chrome weren't even a glimmer in Google's eye back in 2006, so I hope they've largely replaced the code I was working on with something based on Chrome. As late as 2010, the DOM was a completely custom implementation that looked somewhat like Firefox, but with enough IE features to fool lots of other pages that would otherwise change their content to "You must run IE to view this page". (On a side note, as much as many people would like to see such pages heavily penalized and indexed as if the IE-only message were their only information, some of those pages are unique sources of invaluable information and users wouldn't be well served by such harsh treatment.)
  - juretriglav 12y ago
    Thanks for the insight! I know it's very off-topic, but seeing your comment about indexing title tags I have to ask if you have any idea what could be wrong here: http://stackoverflow.com/questions/23732242/how-to-get-google-to-index-a-dynamic-title-in-an-angular-js-app http://stackoverflow.com/questions/23732242/how-to-get-googl... Google seems to be ignoring the title changes our JS makes. Should we not have the title tag there in the first place and then add it in when the title is known?
- davemel37 12y ago
  Everyone seems to be missing the point of this announcement. They are just saying that they are now adding a feature in Webmaster Tools to show you they see your site so you can diagnose problems.
  - thatthatis 12y ago
    Perhaps you haven't been privy to any of the discussions about: "what is the seo implication of SPAs?" Up until now people were pursuing isomorphic JavaScript so they could render server side and client side so the search engines would see their content. This "just a tool" lets webmasters see how google sees their pages, which means SPAs are becoming safe for SEO. That tool plus the song and dance about understanding the modern web is a pretty strong signal in an industry that operates heavily on rumors.

crucialfelix 12y ago

I think the difference is that they are officially announcing that they will do this at scale. Previously it was only a us geeks that knew this was happening, and only for some sites.

tonyhb 12y ago

Yeah, I remember about 2 years ago reading a blog post that analysed the origin of requests to an outdated API. Turns out, Google had crawled the site and was executing JS 17 (or so) days later. Hopefully this news is about them improving that two week lead time so that more modern websites have a better chance of being indexed.

KMag 12y ago
Within hours of being indexed, the JavaScript on the page would have been executed and the links extracted. Presumably these particular extracted links didn't look particularly interesting to the crawl scheduling algorithm, and sat in the crawl scheduling queue for 16 or 17 days. Source: I worked mostly on JavaScript execution in Google's indexing system from 2006 to 2010. I don't know the intricacies of the crawl scheduling, but I do know it's highly non-trivial.