13 ms·

Understanding web pages better

236 points by dmnd 12y ago

Theodores 12y ago
This is great news. However, will you be able to feed the Googlebot a script to simulate user interaction and download all of the ajax content that normally needs clicks or mouseovers?
- cleverjake 12y ago
  no. however it already clicks things.
  - Touche 12y ago
    But how does it know when to take the snapshot? A clicked link can do all sorts of asynchronous things and Google is unaware when the new rendering is "finished".
    - kclay 12y ago
      Giving that chrome can debug ajax request with full call stack ,I wouldn't think it would be to hard to adapt to know these things.
    - ryanpetrich 12y ago
      When there are no outstanding HTTP requests, DOM events, CSS transitions or setTimeouts the page can be assumed to be rendered. Not all pages will enter this state, so some heuristics are likely used.
      - Touche 12y ago
        What you are describing is not possible with, for example, Phantom. HTTP request observation is, but AFAIK the others are not. I know it's likely that Google has something far more advanced / customized, but just wanted to point this out for anyone thinking about doing this themselves; it's a really hard problem to solve.
    - Theodores 12y ago
      "Snapshot" also implies that Google search results are likely to get a lot more visual. The tone of the article made me feel that. Does a 'snapshot' imply imagery in U.S. English?
      - ForHackernews 12y ago
        > Does a 'snapshot' imply imagery in U.S. English? Not necessarily. For instance, I would talk about a "database snapshot", meaning a "backup of a database at a particular moment in time". Obviously the word "snapshot" originates from photography, but to me, the connotation is much more "capture an exact copy at a moment" than "make an image".
        stan_rogers 12y ago
        (Unnecessary pedantry ahead.) Actually, it originates in hunting (fowling) and meant an opportunity shot taken without time to set up or consciously aim, relying on reaction time and trained instinct. In photography, it was a pejorative term applied to amateur photographs taken with essentially the same amount of thought and planning by people who owned those new Kodak gizmos that Mr. Eastman made, and it's pretty much retained that meaning.
        ForHackernews 12y ago
        Interesting: http://www.etymonline.com/index.php?term=snapshot http://www.etymonline.com/index.php?term=snapshot http://www.youtube.com/watch?v=Mv7atRAoU9s http://www.youtube.com/watch?v=Mv7atRAoU9s
mbrubeck 12y ago
Googlebot has done various amounts of JS parsing/execution for a while now. They've also issued similar webmaster guidelines in the past (e.g., don't use robots.txt to block crawling of scripts and styles). 2008: http://moz.com/ugc/new-reality-google-follows-links-in-javascript-4930 http://moz.com/ugc/new-reality-google-follows-links-in-javas... 2009: http://www.labnol.org/internet/search/googlebot-executes-javascript-on-web-pages/8040/ http://www.labnol.org/internet/search/googlebot-executes-jav... 2011: https://twitter.com/mattcutts/status/131425949597179904 https://twitter.com/mattcutts/status/131425949597179904 2012: http://www.thegooglecache.com/white-hat-seo/googlebots-javascript-interpreter-a-diagnostic/ http://www.thegooglecache.com/white-hat-seo/googlebots-javas... From the 2012 article: "Google is actually interpreting the Javascript it spiders. It is not merely trying to extract strings of text and it does appear to be nuanced enough to know what text is and is not added to the Document Object Model."
- ChuckMcM 12y ago
  I was surprised, its kind of old news. There is useful JS to look at when you're crawling a web page. Its also a useful way to detect malware on a page.
  - thatthatis 12y ago
    My understanding is that the state before was: Content augmented with JS -- OK to index. Blank page with all content rendered by JS -- not indexed And what they're saying today is: "we are now going to be able to index single page apps that don't have any server rendered content" Perhaps I've missed something, but this is my interpretation: SPAs are now full citizens in the SEO world.
    - KMag 12y ago
      Google has understood pages that have had only client-side rendered content for years. I worked mostly on JavaScript execution in the indexing system from 2006 until 2010, as well as other "rich content" indexing. Certainly somewhere in the 2008 to 2009 timeframe we saw that the Chinese language version of the Wall Street Journal had a lot of pages in their archive where all of the non-boilerplate content was rendered via JavaScript. Since they didn't do this with their English content, it didn't seem to be an attempt to hide content from search engines, but much more likely a workaround for an older browser that wouldn't properly render Unicode, but with a JavaScript engine that would properly render Unicode. Sometime in the 2008 to 2009 timeframe, Google's indexing system started understanding text that was written into documents from JavaScript body onload handlers, and the Chinese Wall Street Journal archive content was exhibit A in my argument that my changes should be turned on in production. I'm sure they've increased the accuracy of the analysis since, but they've certainly been able to index content written by JavaScript for something like 5 years now. Edit: without giving away any Google secrets, here's a pretty good analysis of my work from 2008: http://moz.com/ugc/new-reality-google-follows-links-in-javascript-4930 http://moz.com/ugc/new-reality-google-follows-links-in-javas... Edit 2: Since the 2008-2009 timeframe, Google also notices when you use JavaScript to change a page's title. I caused a crash in Google's indexing system when I made a bad assumption about Google's HTML parser's handling of XHTML-style empty title tags <title/> and tried to construct negative-length std::strings from them. When your code runs on every single webpage that Google can find, you're certain to hit corner cases you didn't anticipate. I did test for empty <title></title>, but not <title/>, and made incorrect assumptions about the two pointers I'd get to the beginning and end of the title.
superduper33 12y ago
careface
frik 12y ago
I was under the assumption that Googlebot already used a headless Chrome to index websites for some time. Google used Chrome to generated page preview pictures (at index-time) to show the search terms (mouse over, but this feature is no more, as it seems). Some websites that shows you the user agent displayed the Chrome user-agent in the preview picture, back then (~2 years ago). it was called Instant Preview: http://googlesystem.blogspot.co.at/2010/11/google-instant-previews.html http://googlesystem.blogspot.co.at/2010/11/google-instant-pr... details: https://sites.google.com/site/webmasterhelpforum/en/faq-instant-previews https://sites.google.com/site/webmasterhelpforum/en/faq-inst... Google removed this useful feature in 04/2013 :( As we’ve streamlined the results page, we’ve had to remove certain features, such as Instant Previews. -- https://productforums.google.com/forum/#!topic/websearch/AomrTzXMWDI%5B1-25-false%5D https://productforums.google.com/forum/#!topic/websearch/Aom...
- sp332 12y ago
  Especially for their own Blogger platform, which doesn't show anything with JS disabled.
- 27182818284 12y ago
  >Chrome to index websites for some time I always thought this was the secret reason Chrome was built. Build a better Googlebot and then, wait a minute, why not just release an awesome browser to get more people using our product at the same time? Forked.
  - bhartzer 12y ago
    Technically speaking, Google built the new Googlebot and then realized that they could release parts of Googlebot to the public in the form of a web browser.
- a-priori 12y ago
  I've heard speculation that this is a large reason why Google invests so heavily in Chrome. When developers make their sites work with Chrome, they end up making their sites work nicely for the Googlebot without even realizing it. If the Googlebot used a web engine that wasn't found in a common browser, then they would have to pay the costs to make it compatible with all web pages out there. Instead, they push that cost onto developers targeting Chrome. All work to make websites work right in Chrome also make them work for the Googlebot, effectively standardizing the web onto something that the Googlebot knows how to process.
chimeracoder 12y ago
> It's always a good idea to have your site degrade gracefully. This will help users enjoy your content even if their browser doesn't have compatible JavaScript implementations. It will also help visitors with JavaScript disabled or off, as well as search engines that can't execute JavaScript yet. I'm glad that they included this. I get that Javascript is required to make certain sites work the way they do, but I'm appalled by the number of sites that require Javascript just to display static text. Google themselves are guilty of this. Google Groups is (for the most part) just an archive of email mailing lists, but try reading a thread on Google Groups with Javascript disabled![0] There are very few sites that cannot gracefully downgrade to at least some degree, and there are very good reasons for doing so. A major one is that AJAX-heavy sites tend not to perform well on slow connections[1] (again, assuming essentially static content here). If you want your users to be able to access your site on-the-go, graceful degradation is your friend. [0] It's especially ironic now that Google Groups is the only place to read many old Usenet archives going back as far as the early 1980s. [1] Try browsing Twitter on a slow (ie, tethered, or "Amtrak wifi" level connection). For a website that originally originated as a way to send messages over SMS, and is still used that way in other parts of the world, it degrades amazingly poorly over slow connections.
- ludwigvan 12y ago
  The problem is, it is not easy to gracefully degrade. Users have become so accustomed to pages enhanced with JS that it is not possible to not use it even for most simple sites. If you want to go gracefully degrade, it is going to take a lot of effort nowadays. Of course, some new tech like isomorphic rendering (Airbnb's render, React's renderToString) helps this, but again, this is not as easy as it sounds.
- icehawk219 12y ago
  This is a worrying trend I've been noticing as well. The last couple of years especially I've noticed a very large increase in people just not caring about graceful downgrading as well as people just not even testing in other browsers. I've had many conversations with people that are huge fans of Angular and Backbone and similar frameworks and when I mention people without Javascript I just get the canned "but everyone has Javascript turned on anyway and if they don't, too bad" response. Interestingly enough every one of them were also people who developed and tested only against Chrome and never even bothered to acknowledge other browsers. I know someone who is a developer who actually likes IE and uses it as their main browser and for years they've dealt with breaking bugs on sites like Github and other popular tech sites because people just don't even test in other browsers any more. The cynic in me wants to say that this mentality is pushed for by companies like Google because no Javascript means no spying. But honestly I think it really just comes down to laziness. So few people truly care about their craft.
deleted 12y ago
[deleted]
- wnevets 12y ago
  I'm sure they're using chrome/v8
hosay123 12y ago
It's pretty obvious that they need to do this, whether reported or not, in order to handle some very easy spam attacks. E.g., replacing keyword-baiting content with, say, an advert for something totally irrelevant. So this isn't really new or surprising
rcthompson 12y ago
Reminds me of "Googlebot Is Chrome": http://ipullrank.com/googlebot-is-chrome/ http://ipullrank.com/googlebot-is-chrome/ Obviously the claim probably isn't literally true, but it they could certainly share a JS engine (V8) at the very least, and the idea that the motivation behind developing that JS engine may have been spidering JS-dependent pages doesn't seem too far-fetched.
- bhartzer 12y ago
  Actually, they built Googlebot first--and then took parts of it to build Chrome. Googlebot is Chrome and Chrome is Googlebot (minus a few features like crawling).
sergiotapia 12y ago
Makes total sense. When I was researching Angular, something that didn't make any sense was Google not being able to crawl Angular websites. Google makes Angular you see, hence my confusion. This instantly puts me back on the Angular hunt, as now I don't have to pay for a service as ridiculous as 'static page SEO'.
- untog 12y ago
  There might be something as ridiculous as 'static page SEO' but there is nothing ridiculous about 'static pages'.
- ludwigvan 12y ago
  > Google makes Angular you see, hence my confusion. The thing is, Google is not a single entity, like any other big corp. So there will be teams doing things differently, even in conflicting ways. Google is not using Angular for most of its sites, I think Closure is more popular there. You can do static page SEO with JS using libraries that allow isomorphic processing, see rendr, react etc.
- dragonwriter 12y ago
  Google makes many things with different purposes, and it doesn't usually hold back on creating them to wait till every possible integration that might be useful with other Google efforts is done. If it did, it would never ship anything.
- Isofarro 12y ago
  If a "static page SEO" service emits something remotely useful to Googlebot, that's an indication you should have used progressive enhancement from the start.
thrillscience 12y ago
So I wonder if we can trick Googlebot into doing computations for us? Maybye mine bitcoins! :-)
- frik 12y ago
  This could work, especially on high-ranked news sites that are crawled every few minutes for new content. I just noticed a comment on HN that I wrote a few minutes ago, was already in Google search results - was shocked a bit.
  - bhartzer 12y ago
    They're indexing blog posts and content within seconds, not minutes. I've seen my blog posts indexed within seconds and they've been doing that for a few years now.
    - Raphael 12y ago
      The best way to achieve this is with PubSubHubbub to avoid the delay of polling your feed or pages. (Not that it would necessarily stop.)
    - 0x0 12y ago
      That's probably because a default wordpress install has pingomatic.com listed in the callbacks for on-post ("update services").
  - marcosdumay 12y ago
    I had a not-important site with many pages, for a while Google was crawing it continuously, with 10 threads in paralell.
- tantalor 12y ago
  An old trick is to use abuse the crawler to generate a huge index for reverse lookups. For example, reverse "72b302bf297a228a75730123efef7c41" to md5("banana").
- ihsw 12y ago
valarauca1 12y ago
I wonder how Google handles traffic to its crawlers. A webpage I viewed recently loaded an entire database query into my web browser, then I made queries locally to sort it, which kinda sucked since 20MB of information is a lot. I figure Google has to have some form of safeguard against this. Either CPU or network bandwidth limited, likely a time limit too. In my head I'm picturing a crawler locked in a loop of forever querying random google search results and adding them to a page.
- andrenotgiant 12y ago
  Across many requests, Google has a very prevalent "bandwidth/time" calculation for each domain/subdomain that factors in domain importance, # of pages, frequency of page updates, and even input from webmasters via Webmaster Tools. The bot will just stop requesting pages after that ratio is reached. (This is why new domains that publish LOTS of pages at once may take a while to get crawled.) Across individual requests, they have distanced themselves from setting a concrete limit to request sizes[1] (they used to only cache the first 100Kb, then people saw them caching up to 400Kb, now they definitely index things like PDFs that are much larger.) [1] http://www.mattcutts.com/blog/how-many-links-per-page/ http://www.mattcutts.com/blog/how-many-links-per-page/
nateabele 12y ago
I was at ng-conf in January and asked the Angular team a question about improving SEO. Without going into any detail, they hinted at the idea that very shortly it would no longer matter. Honestly I'm kind of surprised it took this long.
- saddestcatever 12y ago
  I couldn't be more excited for this change. The current process of making Angular sites compatible for Google SEO is atrocious
callmeed 12y ago
SEO has been a major factor in my reluctance to implement client-side JS frameworks in many projects (we work with wedding photographers and other small businesses who live and breath on being found in Google). It actually seems harder to optimize a JS-heavy site than a Flash site (which we used to sell a lot of). If Google can actually index and rank a businesses website that is, say, pure BackboneJS that would be awesome. But I'd like to see it in the wild before trying to sell something like that. For example, AirBnB appears to be using Backbone here: https://www.airbnb.com/s/San-Francisco--CA--United-States https://www.airbnb.com/s/San-Francisco--CA--United-States Is Google able to crawl their listings by executing all the JS? Or is AirBnB implementing other tricks to get indexed?
- frik 12y ago
  Google licensed the Adobe Flash text extraction library. Using the library it was easy to extract text and links. * Google Now Crawling And Indexing Flash Content (2008): http://searchengineland.com/google-now-crawling-and-indexing-flash-content-14299 http://searchengineland.com/google-now-crawling-and-indexing... * Adobe page: https://web.archive.org/web/20080702135702/http://www.adobe.com/devnet/flashplayer/articles/swf_searchability.html https://web.archive.org/web/20080702135702/http://www.adobe.... Adobe is working with Google and Yahoo! to enable one of the largest fundamental improvements in web search results by making the Flash file format (SWF) a first- class citizen in searchable web content. Google uses the Adobe Flash Player technology to run SWF content for their search engines to crawl and provide the logic that chooses how to walk through a SWF. Edit: parent commenter edited/changed his text quite a bit, originally it was about Flash content
  - callmeed 12y ago
    Yeah, but this didn't work for dynamic content inside of Flash (i.e. the swf loading image or text content from a server-side API).
  - KMag 12y ago
    Google engineer Ran Adler (sometimes he spells his first name Ron to avoid confusing English speakers) deserves most of the credit here. He went to Adobe with a proposal for the hooks into the Flash player that he needed and worked with their engineers to get those hooks working. I don't doubt there was a fair amount of work on Adobe's side, too, but it wasn't like Adobe had the technology in place before Ran started working with them. Ran came up with an API for the hooks he needed that didn't give away too much of the most clever parts of what he was doing. The belief was that Google would get the hooks it wanted and in return, Adobe could share the special Flash player with other major search engines and everyone would be indexing Flash content. The hope was that Google would just be doing it a bit more cleverly than the competition. I'm not sure if any of the other major search engines ever used the hooks Ran designed. Source: I worked on Google's rich content indexing team from 2006 to 2010. Ran worked mostly on Flash indexing and I worked mostly on JavaScript indexing.
dallen33 12y ago
Does this mean Googlebot can crawl pjax pages? https://github.com/defunkt/jquery-pjax https://github.com/defunkt/jquery-pjax
- andybak 12y ago
  I thought the whole point of pjax was that it would also return the content for any URL via a normal GET. At least - that's how I've always implemented it (using django-pjax).
- jordanlev 12y ago
  My understanding of pjax is that this should be irrelevant. I thought the point of pjax is that you're not generating content with javascript but rather pulling it in from an existing page and inserting it into the current DOM (but the other page that the new content came from also exists at a specific URL, so the site still "works" even without javascript -- it's just a speed boost if you do have js enabled).
andrenotgiant 12y ago
I think the important thing to keep in mind is HOW Google ensures that their understanding of Javascript helps them improve the Search Experience. If they crawl your page with javascript enabled, and find that after a hover event a button appears and after a click on that button a modal appears, and that modal has content about BLUE WIDGETS, they are still NEVER going to rank that URL for "BLUE WIDGETS" Google wants to send users searching for "BLUE WIDGETS" to a page where content about "BLUE WIDGETS" is instantly visible and apparent.
chestnut-tree 12y ago
"It's always a good idea to have your site degrade gracefully. This will help users enjoy your content even if their browser doesn't have compatible JavaScript implementations. It will also help visitors with JavaScript disabled or off, as well as search engines that can't execute JavaScript yet." Are Google going to follow their own advice here? Try visiting the official Android blog with Javascript disabled http://officialandroid.blogspot.co.uk/ http://officialandroid.blogspot.co.uk/ In fact, try visiting a whole bunch of *blogspot.co.uk sites with Javascript disabled and see how "gracefully" they degrade. Remember, these are blog sites with mostly text content. And yet Google won't serve them up without Javsacipt enabled.
- thinxer 12y ago
  yet they provide a rendered version for crawlers. try append "?_escaped_fragment_=" to the pages, like this one: http://officialandroid.blogspot.com/2014/04/new-mobile-apps-for-docs-sheets-and.html?_escaped_fragment_= http://officialandroid.blogspot.com/2014/04/new-mobile-apps-... And you will see the text content. It sucks, anyway.
sashagim 12y ago
Interesting. I wonder if some changes will need to take place in order to make sure the client side tracking services (e.g MixPanel, Kissmetrics, Google's own GA) ignore Google bot.
snake_plissken 12y ago
Just out of curiosity, how is this possible? If the web server can't handle being crawled...how can it handle serving web pages? "If your web server is unable to handle the volume of crawl requests for resources, it may have a negative impact on our capability to render your pages. If you’d like to ensure that your pages can be rendered by Google, make sure your servers are able to handle crawl requests for resources."
- devNoise 12y ago
  Try to equate being crawled with the /. effect. The site can server web pages, but not at scale. So if the googlebot comes along to index your site, the webserver may fail under load. Perhaps you're on co-hosting plan and your provider suspends your site because the crawling has put you over one of the limits on your plan.
gwbas1c 12y ago
Honestly, I find sites that completely rely on Javascript to be somewhat unreliable. The web is fundamentally a document retrieval system. Content needs to work without Javascript.
- grkvlt 12y ago
  I believe you're thinking of FTP, not HTTP. Content is rather more than documents, and spans from animations to interactive presentations, visualisations and demonstrations, games and ... you get the idea ;)
znowi 12y ago
Guidelines on how to conform to the rules of the Matrix, otherwise you may be expelled from the system into obscurity of Zion.
known 12y ago
Googlebot = Google's "Customized" Chrome

236 points by dmnd 12y ago

This is great news. However, will you be able to feed the Googlebot a script to simulate user interaction and download all of the ajax content that normally needs clicks or mouseovers?

cleverjake 12y ago
no. however it already clicks things.
- Touche 12y ago
  But how does it know when to take the snapshot? A clicked link can do all sorts of asynchronous things and Google is unaware when the new rendering is "finished".
  - kclay 12y ago
    Giving that chrome can debug ajax request with full call stack ,I wouldn't think it would be to hard to adapt to know these things.
  - ryanpetrich 12y ago
    When there are no outstanding HTTP requests, DOM events, CSS transitions or setTimeouts the page can be assumed to be rendered. Not all pages will enter this state, so some heuristics are likely used.
    - Touche 12y ago
      What you are describing is not possible with, for example, Phantom. HTTP request observation is, but AFAIK the others are not. I know it's likely that Google has something far more advanced / customized, but just wanted to point this out for anyone thinking about doing this themselves; it's a really hard problem to solve.
  - Theodores 12y ago
    "Snapshot" also implies that Google search results are likely to get a lot more visual. The tone of the article made me feel that. Does a 'snapshot' imply imagery in U.S. English?
    - ForHackernews 12y ago
      > Does a 'snapshot' imply imagery in U.S. English? Not necessarily. For instance, I would talk about a "database snapshot", meaning a "backup of a database at a particular moment in time". Obviously the word "snapshot" originates from photography, but to me, the connotation is much more "capture an exact copy at a moment" than "make an image".
      - stan_rogers 12y ago
        (Unnecessary pedantry ahead.) Actually, it originates in hunting (fowling) and meant an opportunity shot taken without time to set up or consciously aim, relying on reaction time and trained instinct. In photography, it was a pejorative term applied to amateur photographs taken with essentially the same amount of thought and planning by people who owned those new Kodak gizmos that Mr. Eastman made, and it's pretty much retained that meaning.
        ForHackernews 12y ago
        Interesting: http://www.etymonline.com/index.php?term=snapshot http://www.etymonline.com/index.php?term=snapshot http://www.youtube.com/watch?v=Mv7atRAoU9s http://www.youtube.com/watch?v=Mv7atRAoU9s

mbrubeck 12y ago

Googlebot has done various amounts of JS parsing/execution for a while now. They've also issued similar webmaster guidelines in the past (e.g., don't use robots.txt to block crawling of scripts and styles). 2008: http://moz.com/ugc/new-reality-google-follows-links-in-javascript-4930 http://moz.com/ugc/new-reality-google-follows-links-in-javas... 2009: http://www.labnol.org/internet/search/googlebot-executes-javascript-on-web-pages/8040/ http://www.labnol.org/internet/search/googlebot-executes-jav... 2011: https://twitter.com/mattcutts/status/131425949597179904 https://twitter.com/mattcutts/status/131425949597179904 2012: http://www.thegooglecache.com/white-hat-seo/googlebots-javascript-interpreter-a-diagnostic/ http://www.thegooglecache.com/white-hat-seo/googlebots-javas... From the 2012 article: "Google is actually interpreting the Javascript it spiders. It is not merely trying to extract strings of text and it does appear to be nuanced enough to know what text is and is not added to the Document Object Model."

ChuckMcM 12y ago
I was surprised, its kind of old news. There is useful JS to look at when you're crawling a web page. Its also a useful way to detect malware on a page.
- thatthatis 12y ago
  My understanding is that the state before was: Content augmented with JS -- OK to index. Blank page with all content rendered by JS -- not indexed And what they're saying today is: "we are now going to be able to index single page apps that don't have any server rendered content" Perhaps I've missed something, but this is my interpretation: SPAs are now full citizens in the SEO world.
  - KMag 12y ago
    Google has understood pages that have had only client-side rendered content for years. I worked mostly on JavaScript execution in the indexing system from 2006 until 2010, as well as other "rich content" indexing. Certainly somewhere in the 2008 to 2009 timeframe we saw that the Chinese language version of the Wall Street Journal had a lot of pages in their archive where all of the non-boilerplate content was rendered via JavaScript. Since they didn't do this with their English content, it didn't seem to be an attempt to hide content from search engines, but much more likely a workaround for an older browser that wouldn't properly render Unicode, but with a JavaScript engine that would properly render Unicode. Sometime in the 2008 to 2009 timeframe, Google's indexing system started understanding text that was written into documents from JavaScript body onload handlers, and the Chinese Wall Street Journal archive content was exhibit A in my argument that my changes should be turned on in production. I'm sure they've increased the accuracy of the analysis since, but they've certainly been able to index content written by JavaScript for something like 5 years now. Edit: without giving away any Google secrets, here's a pretty good analysis of my work from 2008: http://moz.com/ugc/new-reality-google-follows-links-in-javascript-4930 http://moz.com/ugc/new-reality-google-follows-links-in-javas... Edit 2: Since the 2008-2009 timeframe, Google also notices when you use JavaScript to change a page's title. I caused a crash in Google's indexing system when I made a bad assumption about Google's HTML parser's handling of XHTML-style empty title tags <title/> and tried to construct negative-length std::strings from them. When your code runs on every single webpage that Google can find, you're certain to hit corner cases you didn't anticipate. I did test for empty <title></title>, but not <title/>, and made incorrect assumptions about the two pointers I'd get to the beginning and end of the title.

superduper33 12y ago

careface

frik 12y ago

I was under the assumption that Googlebot already used a headless Chrome to index websites for some time. Google used Chrome to generated page preview pictures (at index-time) to show the search terms (mouse over, but this feature is no more, as it seems). Some websites that shows you the user agent displayed the Chrome user-agent in the preview picture, back then (~2 years ago). it was called Instant Preview: http://googlesystem.blogspot.co.at/2010/11/google-instant-previews.html http://googlesystem.blogspot.co.at/2010/11/google-instant-pr... details: https://sites.google.com/site/webmasterhelpforum/en/faq-instant-previews https://sites.google.com/site/webmasterhelpforum/en/faq-inst... Google removed this useful feature in 04/2013 :( As we’ve streamlined the results page, we’ve had to remove certain features, such as Instant Previews. -- https://productforums.google.com/forum/#!topic/websearch/AomrTzXMWDI%5B1-25-false%5D https://productforums.google.com/forum/#!topic/websearch/Aom...

sp332 12y ago
Especially for their own Blogger platform, which doesn't show anything with JS disabled.
27182818284 12y ago
>Chrome to index websites for some time I always thought this was the secret reason Chrome was built. Build a better Googlebot and then, wait a minute, why not just release an awesome browser to get more people using our product at the same time? Forked.
- bhartzer 12y ago
  Technically speaking, Google built the new Googlebot and then realized that they could release parts of Googlebot to the public in the form of a web browser.
a-priori 12y ago
I've heard speculation that this is a large reason why Google invests so heavily in Chrome. When developers make their sites work with Chrome, they end up making their sites work nicely for the Googlebot without even realizing it. If the Googlebot used a web engine that wasn't found in a common browser, then they would have to pay the costs to make it compatible with all web pages out there. Instead, they push that cost onto developers targeting Chrome. All work to make websites work right in Chrome also make them work for the Googlebot, effectively standardizing the web onto something that the Googlebot knows how to process.

chimeracoder 12y ago

> It's always a good idea to have your site degrade gracefully. This will help users enjoy your content even if their browser doesn't have compatible JavaScript implementations. It will also help visitors with JavaScript disabled or off, as well as search engines that can't execute JavaScript yet. I'm glad that they included this. I get that Javascript is required to make certain sites work the way they do, but I'm appalled by the number of sites that require Javascript just to display static text. Google themselves are guilty of this. Google Groups is (for the most part) just an archive of email mailing lists, but try reading a thread on Google Groups with Javascript disabled![0] There are very few sites that cannot gracefully downgrade to at least some degree, and there are very good reasons for doing so. A major one is that AJAX-heavy sites tend not to perform well on slow connections[1] (again, assuming essentially static content here). If you want your users to be able to access your site on-the-go, graceful degradation is your friend. [0] It's especially ironic now that Google Groups is the only place to read many old Usenet archives going back as far as the early 1980s. [1] Try browsing Twitter on a slow (ie, tethered, or "Amtrak wifi" level connection). For a website that originally originated as a way to send messages over SMS, and is still used that way in other parts of the world, it degrades amazingly poorly over slow connections.

ludwigvan 12y ago
The problem is, it is not easy to gracefully degrade. Users have become so accustomed to pages enhanced with JS that it is not possible to not use it even for most simple sites. If you want to go gracefully degrade, it is going to take a lot of effort nowadays. Of course, some new tech like isomorphic rendering (Airbnb's render, React's renderToString) helps this, but again, this is not as easy as it sounds.
icehawk219 12y ago
This is a worrying trend I've been noticing as well. The last couple of years especially I've noticed a very large increase in people just not caring about graceful downgrading as well as people just not even testing in other browsers. I've had many conversations with people that are huge fans of Angular and Backbone and similar frameworks and when I mention people without Javascript I just get the canned "but everyone has Javascript turned on anyway and if they don't, too bad" response. Interestingly enough every one of them were also people who developed and tested only against Chrome and never even bothered to acknowledge other browsers. I know someone who is a developer who actually likes IE and uses it as their main browser and for years they've dealt with breaking bugs on sites like Github and other popular tech sites because people just don't even test in other browsers any more. The cynic in me wants to say that this mentality is pushed for by companies like Google because no Javascript means no spying. But honestly I think it really just comes down to laziness. So few people truly care about their craft.

deleted 12y ago

[deleted]

wnevets 12y ago
I'm sure they're using chrome/v8

hosay123 12y ago

It's pretty obvious that they need to do this, whether reported or not, in order to handle some very easy spam attacks. E.g., replacing keyword-baiting content with, say, an advert for something totally irrelevant. So this isn't really new or surprising

rcthompson 12y ago

Reminds me of "Googlebot Is Chrome": http://ipullrank.com/googlebot-is-chrome/ http://ipullrank.com/googlebot-is-chrome/ Obviously the claim probably isn't literally true, but it they could certainly share a JS engine (V8) at the very least, and the idea that the motivation behind developing that JS engine may have been spidering JS-dependent pages doesn't seem too far-fetched.

bhartzer 12y ago
Actually, they built Googlebot first--and then took parts of it to build Chrome. Googlebot is Chrome and Chrome is Googlebot (minus a few features like crawling).

sergiotapia 12y ago

Makes total sense. When I was researching Angular, something that didn't make any sense was Google not being able to crawl Angular websites. Google makes Angular you see, hence my confusion. This instantly puts me back on the Angular hunt, as now I don't have to pay for a service as ridiculous as 'static page SEO'.

untog 12y ago
There might be something as ridiculous as 'static page SEO' but there is nothing ridiculous about 'static pages'.
ludwigvan 12y ago
> Google makes Angular you see, hence my confusion. The thing is, Google is not a single entity, like any other big corp. So there will be teams doing things differently, even in conflicting ways. Google is not using Angular for most of its sites, I think Closure is more popular there. You can do static page SEO with JS using libraries that allow isomorphic processing, see rendr, react etc.
dragonwriter 12y ago
Google makes many things with different purposes, and it doesn't usually hold back on creating them to wait till every possible integration that might be useful with other Google efforts is done. If it did, it would never ship anything.
Isofarro 12y ago
If a "static page SEO" service emits something remotely useful to Googlebot, that's an indication you should have used progressive enhancement from the start.

thrillscience 12y ago

So I wonder if we can trick Googlebot into doing computations for us? Maybye mine bitcoins! :-)

frik 12y ago
This could work, especially on high-ranked news sites that are crawled every few minutes for new content. I just noticed a comment on HN that I wrote a few minutes ago, was already in Google search results - was shocked a bit.
- bhartzer 12y ago
  They're indexing blog posts and content within seconds, not minutes. I've seen my blog posts indexed within seconds and they've been doing that for a few years now.
  - Raphael 12y ago
    The best way to achieve this is with PubSubHubbub to avoid the delay of polling your feed or pages. (Not that it would necessarily stop.)
  - 0x0 12y ago
    That's probably because a default wordpress install has pingomatic.com listed in the callbacks for on-post ("update services").
- marcosdumay 12y ago
  I had a not-important site with many pages, for a while Google was crawing it continuously, with 10 threads in paralell.
tantalor 12y ago
An old trick is to use abuse the crawler to generate a huge index for reverse lookups. For example, reverse "72b302bf297a228a75730123efef7c41" to md5("banana").
ihsw 12y ago

valarauca1 12y ago

I wonder how Google handles traffic to its crawlers. A webpage I viewed recently loaded an entire database query into my web browser, then I made queries locally to sort it, which kinda sucked since 20MB of information is a lot. I figure Google has to have some form of safeguard against this. Either CPU or network bandwidth limited, likely a time limit too. In my head I'm picturing a crawler locked in a loop of forever querying random google search results and adding them to a page.

andrenotgiant 12y ago
Across many requests, Google has a very prevalent "bandwidth/time" calculation for each domain/subdomain that factors in domain importance, # of pages, frequency of page updates, and even input from webmasters via Webmaster Tools. The bot will just stop requesting pages after that ratio is reached. (This is why new domains that publish LOTS of pages at once may take a while to get crawled.) Across individual requests, they have distanced themselves from setting a concrete limit to request sizes[1] (they used to only cache the first 100Kb, then people saw them caching up to 400Kb, now they definitely index things like PDFs that are much larger.) [1] http://www.mattcutts.com/blog/how-many-links-per-page/ http://www.mattcutts.com/blog/how-many-links-per-page/

nateabele 12y ago

I was at ng-conf in January and asked the Angular team a question about improving SEO. Without going into any detail, they hinted at the idea that very shortly it would no longer matter. Honestly I'm kind of surprised it took this long.

saddestcatever 12y ago
I couldn't be more excited for this change. The current process of making Angular sites compatible for Google SEO is atrocious

callmeed 12y ago

SEO has been a major factor in my reluctance to implement client-side JS frameworks in many projects (we work with wedding photographers and other small businesses who live and breath on being found in Google). It actually seems harder to optimize a JS-heavy site than a Flash site (which we used to sell a lot of). If Google can actually index and rank a businesses website that is, say, pure BackboneJS that would be awesome. But I'd like to see it in the wild before trying to sell something like that. For example, AirBnB appears to be using Backbone here: https://www.airbnb.com/s/San-Francisco--CA--United-States https://www.airbnb.com/s/San-Francisco--CA--United-States Is Google able to crawl their listings by executing all the JS? Or is AirBnB implementing other tricks to get indexed?

frik 12y ago
Google licensed the Adobe Flash text extraction library. Using the library it was easy to extract text and links. * Google Now Crawling And Indexing Flash Content (2008): http://searchengineland.com/google-now-crawling-and-indexing-flash-content-14299 http://searchengineland.com/google-now-crawling-and-indexing... * Adobe page: https://web.archive.org/web/20080702135702/http://www.adobe.com/devnet/flashplayer/articles/swf_searchability.html https://web.archive.org/web/20080702135702/http://www.adobe.... Adobe is working with Google and Yahoo! to enable one of the largest fundamental improvements in web search results by making the Flash file format (SWF) a first- class citizen in searchable web content. Google uses the Adobe Flash Player technology to run SWF content for their search engines to crawl and provide the logic that chooses how to walk through a SWF. Edit: parent commenter edited/changed his text quite a bit, originally it was about Flash content
- callmeed 12y ago
  Yeah, but this didn't work for dynamic content inside of Flash (i.e. the swf loading image or text content from a server-side API).
- KMag 12y ago
  Google engineer Ran Adler (sometimes he spells his first name Ron to avoid confusing English speakers) deserves most of the credit here. He went to Adobe with a proposal for the hooks into the Flash player that he needed and worked with their engineers to get those hooks working. I don't doubt there was a fair amount of work on Adobe's side, too, but it wasn't like Adobe had the technology in place before Ran started working with them. Ran came up with an API for the hooks he needed that didn't give away too much of the most clever parts of what he was doing. The belief was that Google would get the hooks it wanted and in return, Adobe could share the special Flash player with other major search engines and everyone would be indexing Flash content. The hope was that Google would just be doing it a bit more cleverly than the competition. I'm not sure if any of the other major search engines ever used the hooks Ran designed. Source: I worked on Google's rich content indexing team from 2006 to 2010. Ran worked mostly on Flash indexing and I worked mostly on JavaScript indexing.

dallen33 12y ago

Does this mean Googlebot can crawl pjax pages? https://github.com/defunkt/jquery-pjax https://github.com/defunkt/jquery-pjax

andybak 12y ago
I thought the whole point of pjax was that it would also return the content for any URL via a normal GET. At least - that's how I've always implemented it (using django-pjax).
jordanlev 12y ago
My understanding of pjax is that this should be irrelevant. I thought the point of pjax is that you're not generating content with javascript but rather pulling it in from an existing page and inserting it into the current DOM (but the other page that the new content came from also exists at a specific URL, so the site still "works" even without javascript -- it's just a speed boost if you do have js enabled).

andrenotgiant 12y ago

I think the important thing to keep in mind is HOW Google ensures that their understanding of Javascript helps them improve the Search Experience. If they crawl your page with javascript enabled, and find that after a hover event a button appears and after a click on that button a modal appears, and that modal has content about BLUE WIDGETS, they are still NEVER going to rank that URL for "BLUE WIDGETS" Google wants to send users searching for "BLUE WIDGETS" to a page where content about "BLUE WIDGETS" is instantly visible and apparent.

chestnut-tree 12y ago

"It's always a good idea to have your site degrade gracefully. This will help users enjoy your content even if their browser doesn't have compatible JavaScript implementations. It will also help visitors with JavaScript disabled or off, as well as search engines that can't execute JavaScript yet." Are Google going to follow their own advice here? Try visiting the official Android blog with Javascript disabled http://officialandroid.blogspot.co.uk/ http://officialandroid.blogspot.co.uk/ In fact, try visiting a whole bunch of *blogspot.co.uk sites with Javascript disabled and see how "gracefully" they degrade. Remember, these are blog sites with mostly text content. And yet Google won't serve them up without Javsacipt enabled.

thinxer 12y ago
yet they provide a rendered version for crawlers. try append "?_escaped_fragment_=" to the pages, like this one: http://officialandroid.blogspot.com/2014/04/new-mobile-apps-for-docs-sheets-and.html?_escaped_fragment_= http://officialandroid.blogspot.com/2014/04/new-mobile-apps-... And you will see the text content. It sucks, anyway.

sashagim 12y ago

Interesting. I wonder if some changes will need to take place in order to make sure the client side tracking services (e.g MixPanel, Kissmetrics, Google's own GA) ignore Google bot.

snake_plissken 12y ago

Just out of curiosity, how is this possible? If the web server can't handle being crawled...how can it handle serving web pages? "If your web server is unable to handle the volume of crawl requests for resources, it may have a negative impact on our capability to render your pages. If you’d like to ensure that your pages can be rendered by Google, make sure your servers are able to handle crawl requests for resources."

devNoise 12y ago
Try to equate being crawled with the /. effect. The site can server web pages, but not at scale. So if the googlebot comes along to index your site, the webserver may fail under load. Perhaps you're on co-hosting plan and your provider suspends your site because the crawling has put you over one of the limits on your plan.

gwbas1c 12y ago

Honestly, I find sites that completely rely on Javascript to be somewhat unreliable. The web is fundamentally a document retrieval system. Content needs to work without Javascript.

grkvlt 12y ago
I believe you're thinking of FTP, not HTTP. Content is rather more than documents, and spans from animations to interactive presentations, visualisations and demonstrations, games and ... you get the idea ;)

znowi 12y ago

Guidelines on how to conform to the rules of the Matrix, otherwise you may be expelled from the system into obscurity of Zion.

known 12y ago

Googlebot = Google's "Customized" Chrome