3 ms·

Google has understood pages that have had only client-side rendered content for years. I worked mostly on JavaScript execution in the indexing system from 2006

by KMag 12y ago

Google has understood pages that have had only client-side rendered content for years. I worked mostly on JavaScript execution in the indexing system from 2006 until 2010, as well as other "rich content" indexing. Certainly somewhere in the 2008 to 2009 timeframe we saw that the Chinese language version of the Wall Street Journal had a lot of pages in their archive where all of the non-boilerplate content was rendered via JavaScript. Since they didn't do this with their English content, it didn't seem to be an attempt to hide content from search engines, but much more likely a workaround for an older browser that wouldn't properly render Unicode, but with a JavaScript engine that would properly render Unicode. Sometime in the 2008 to 2009 timeframe, Google's indexing system started understanding text that was written into documents from JavaScript body onload handlers, and the Chinese Wall Street Journal archive content was exhibit A in my argument that my changes should be turned on in production. I'm sure they've increased the accuracy of the analysis since, but they've certainly been able to index content written by JavaScript for something like 5 years now. Edit: without giving away any Google secrets, here's a pretty good analysis of my work from 2008: http://moz.com/ugc/new-reality-google-follows-links-in-javascript-4930 http://moz.com/ugc/new-reality-google-follows-links-in-javas... Edit 2: Since the 2008-2009 timeframe, Google also notices when you use JavaScript to change a page's title. I caused a crash in Google's indexing system when I made a bad assumption about Google's HTML parser's handling of XHTML-style empty title tags <title/> and tried to construct negative-length std::strings from them. When your code runs on every single webpage that Google can find, you're certain to hit corner cases you didn't anticipate. I did test for empty <title></title>, but not <title/>, and made incorrect assumptions about the two pointers I'd get to the beginning and end of the title.

thatthatis 12y ago
So would the more accurate interpretation be: google is starting to promote that it can and will index javascript single page apps? Thanks for sharing, and nice work btw.
- KMag 12y ago
  There were a lot of caveats that both reduced fidelity and would have made announcements confusing back when I was at Google. Also, if one mentions a limitation in an announcement, a lot of the Search Engine Optimization community would be citing the announcement for more than 6 months. So, it's difficult and potentially counter-productive to have an announcement with lots of caveats. V8 and Chrome weren't even a glimmer in Google's eye back in 2006, so I hope they've largely replaced the code I was working on with something based on Chrome. As late as 2010, the DOM was a completely custom implementation that looked somewhat like Firefox, but with enough IE features to fool lots of other pages that would otherwise change their content to "You must run IE to view this page". (On a side note, as much as many people would like to see such pages heavily penalized and indexed as if the IE-only message were their only information, some of those pages are unique sources of invaluable information and users wouldn't be well served by such harsh treatment.)
juretriglav 12y ago
Thanks for the insight! I know it's very off-topic, but seeing your comment about indexing title tags I have to ask if you have any idea what could be wrong here: http://stackoverflow.com/questions/23732242/how-to-get-google-to-index-a-dynamic-title-in-an-angular-js-app http://stackoverflow.com/questions/23732242/how-to-get-googl... Google seems to be ignoring the title changes our JS makes. Should we not have the title tag there in the first place and then add it in when the title is known?

by KMag 12y ago

thatthatis 12y ago

So would the more accurate interpretation be: google is starting to promote that it can and will index javascript single page apps? Thanks for sharing, and nice work btw.

KMag 12y ago
There were a lot of caveats that both reduced fidelity and would have made announcements confusing back when I was at Google. Also, if one mentions a limitation in an announcement, a lot of the Search Engine Optimization community would be citing the announcement for more than 6 months. So, it's difficult and potentially counter-productive to have an announcement with lots of caveats. V8 and Chrome weren't even a glimmer in Google's eye back in 2006, so I hope they've largely replaced the code I was working on with something based on Chrome. As late as 2010, the DOM was a completely custom implementation that looked somewhat like Firefox, but with enough IE features to fool lots of other pages that would otherwise change their content to "You must run IE to view this page". (On a side note, as much as many people would like to see such pages heavily penalized and indexed as if the IE-only message were their only information, some of those pages are unique sources of invaluable information and users wouldn't be well served by such harsh treatment.)

juretriglav 12y ago

Thanks for the insight! I know it's very off-topic, but seeing your comment about indexing title tags I have to ask if you have any idea what could be wrong here: http://stackoverflow.com/questions/23732242/how-to-get-google-to-index-a-dynamic-title-in-an-angular-js-app http://stackoverflow.com/questions/23732242/how-to-get-googl... Google seems to be ignoring the title changes our JS makes. Should we not have the title tag there in the first place and then add it in when the title is known?