6 ms·

Javascript-dependent content _can_ present problems for search engine accessibility

mmastrac 13y ago
There have been techniques for dealing with this for a very long time: https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot https://developers.google.com/webmasters/ajax-crawling/docs/... And this is only an issue for pages where you actually want to expose your content to search engines.
- deleted 13y ago
  [deleted]
- pornel 13y ago
  Sigh, Google-only websites. _escaped_fragment_ is a stupid hack that requires hardest bit of work needed for progressive enhancement, but provides none of the benefit.
  - slig 13y ago
    It's not like other search engines can't step up their game and build the same functionality.
    - repsilat 13y ago
      The problem is that now everyone who wants to build a decent crawler/scraper has to do it as well. Used to be any kid in his parents' basement could put wget and sed together and explore the internet. Now Timmy has to bolt a whole damn browser runtime onto his script to make sure he's getting the content text from blog posts and twitter feeds. Asking Timmy to "step up his game" is heartless, and it'd be unnecessary if we were better citizens.
      - ylesaout 13y ago
        Maybe Timmy should just have a look at the Google specification! And he would see that he could continue to crawl the web with his favorite tool! He has just to replace #! with an _escaped_fragment_ query parameter, that's not so heartless ;)
  - huhtenberg 13y ago
    Google has been crawling AJAX'd websites bypassing the _escaped_fragment_ mechanism for at least a year now.
    - chaddeshon 13y ago
      Has it really? Do you have an example? I've never seen an example where Google crawled a page and included content that was loaded via ajax or an external javascript file. I have seen Google index content that is loaded from a script tag included in the html page.
      - huhtenberg 13y ago
        https://news.ycombinator.com/item?id=3985981 https://news.ycombinator.com/item?id=3985981
        chaddeshon 13y ago
        Certainly cool that Google is working on this. I suspect someday it will all work perfectly. However, from working with customers of BromBone, I know this. People have javascript powered webpages that were not getting indexed by Google. Google would crawl the page, and see no content. The pages were not in Google. After creating prerendered snapshots and serving them when the _escaped_fragment_ parameter was present, the pages showed up in Google.
  - ylesaout 13y ago
    Not only Google! Bing (and so Yahoo), Yandex and Facebook (only #! for this last) support this protocol too. And if you use html5 pushstate you could even serve the HTML captures to any bots. Eventually you also could easily add graceful degradation to your site serving snapshots for non js users. As of PoC for SEO4Ajax (http://www.seo4ajax.com http://www.seo4ajax.com), I built this application (http://www.appscharts.me http://www.appscharts.me) illustrating this purpose.
  - chaddeshon 13y ago
    _escaped_fragment_ is a bit of a hack, but it also provide enormous benefit. It allows you to gain all the benefits this article is talking about (which I won't repeat here) and still make you site crawlable without duplicating code. You are right about one thing, it is a pain to setup. Much harder than it sounds at first. After doing it a few times, I finally got sick of it and built http://www.BromBone.com http://www.BromBone.com. It's a service that takes generates the html snapshots for site owners and keeps them up to date. I hope it will let site owner keep all the positives and eliminate most of the new negatives.
mwsherman 13y ago
I have to assume, in Google’s case anyway, that they are indexing according to ‘what the user sees’. Not to do so would means lower-quality search results. Users don’t care how the page ‘happens’. I make this assumption because most of us don’t actually know how Google works, acting as a user’s proxy is in Google’s interest, and it is certainly within their technological capability. (Let me emphasize assume, again.) This would require that the crawler actually renders sites, indexes the resulting DOM, and clicks what can be clicked. A headless Chrome, perhaps.
- kintamanimatt 13y ago
  From Google's technical guidlines: "Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site." https://support.google.com/webmasters/answer/35769?hl=en https://support.google.com/webmasters/answer/35769?hl=en
  - jasonlotito 13y ago
    Not sure what you are trying to say by merely copying and pasting text about random search engine spiders that may have trouble crawling you. Maybe you could explain? With regard to Google, however, they are explicit in their support for JavaScript rendered sites: https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot https://developers.google.com/webmasters/ajax-crawling/docs/... Amongst many other links.

There have been techniques for dealing with this for a very long time: https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot https://developers.google.com/webmasters/ajax-crawling/docs/... And this is only an issue for pages where you actually want to expose your content to search engines.

deleted 13y ago
[deleted]
pornel 13y ago
Sigh, Google-only websites. _escaped_fragment_ is a stupid hack that requires hardest bit of work needed for progressive enhancement, but provides none of the benefit.
- slig 13y ago
  It's not like other search engines can't step up their game and build the same functionality.
  - repsilat 13y ago
    The problem is that now everyone who wants to build a decent crawler/scraper has to do it as well. Used to be any kid in his parents' basement could put wget and sed together and explore the internet. Now Timmy has to bolt a whole damn browser runtime onto his script to make sure he's getting the content text from blog posts and twitter feeds. Asking Timmy to "step up his game" is heartless, and it'd be unnecessary if we were better citizens.
    - ylesaout 13y ago
      Maybe Timmy should just have a look at the Google specification! And he would see that he could continue to crawl the web with his favorite tool! He has just to replace #! with an _escaped_fragment_ query parameter, that's not so heartless ;)
- huhtenberg 13y ago
  Google has been crawling AJAX'd websites bypassing the _escaped_fragment_ mechanism for at least a year now.
  - chaddeshon 13y ago
    Has it really? Do you have an example? I've never seen an example where Google crawled a page and included content that was loaded via ajax or an external javascript file. I have seen Google index content that is loaded from a script tag included in the html page.
    - huhtenberg 13y ago
      https://news.ycombinator.com/item?id=3985981 https://news.ycombinator.com/item?id=3985981
      - chaddeshon 13y ago
        Certainly cool that Google is working on this. I suspect someday it will all work perfectly. However, from working with customers of BromBone, I know this. People have javascript powered webpages that were not getting indexed by Google. Google would crawl the page, and see no content. The pages were not in Google. After creating prerendered snapshots and serving them when the _escaped_fragment_ parameter was present, the pages showed up in Google.
- ylesaout 13y ago
  Not only Google! Bing (and so Yahoo), Yandex and Facebook (only #! for this last) support this protocol too. And if you use html5 pushstate you could even serve the HTML captures to any bots. Eventually you also could easily add graceful degradation to your site serving snapshots for non js users. As of PoC for SEO4Ajax (http://www.seo4ajax.com http://www.seo4ajax.com), I built this application (http://www.appscharts.me http://www.appscharts.me) illustrating this purpose.
- chaddeshon 13y ago
  _escaped_fragment_ is a bit of a hack, but it also provide enormous benefit. It allows you to gain all the benefits this article is talking about (which I won't repeat here) and still make you site crawlable without duplicating code. You are right about one thing, it is a pain to setup. Much harder than it sounds at first. After doing it a few times, I finally got sick of it and built http://www.BromBone.com http://www.BromBone.com. It's a service that takes generates the html snapshots for site owners and keeps them up to date. I hope it will let site owner keep all the positives and eliminate most of the new negatives.

mwsherman 13y ago

I have to assume, in Google’s case anyway, that they are indexing according to ‘what the user sees’. Not to do so would means lower-quality search results. Users don’t care how the page ‘happens’. I make this assumption because most of us don’t actually know how Google works, acting as a user’s proxy is in Google’s interest, and it is certainly within their technological capability. (Let me emphasize assume, again.) This would require that the crawler actually renders sites, indexes the resulting DOM, and clicks what can be clicked. A headless Chrome, perhaps.

kintamanimatt 13y ago
From Google's technical guidlines: "Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site." https://support.google.com/webmasters/answer/35769?hl=en https://support.google.com/webmasters/answer/35769?hl=en
- jasonlotito 13y ago
  Not sure what you are trying to say by merely copying and pasting text about random search engine spiders that may have trouble crawling you. Maybe you could explain? With regard to Google, however, they are explicit in their support for JavaScript rendered sites: https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot https://developers.google.com/webmasters/ajax-crawling/docs/... Amongst many other links.