5 ms·

Making AJAX Applications Crawlable

32 points by romland 16y ago

asnyder 16y ago
Interestingly enough NOLOH (http://www.noloh.com http://www.noloh.com) made AJAX applications crawlable years ago. If you're interested in the specifics, see http://dev.noloh.com/#/articles/Search-Engine-Friendly/ http://dev.noloh.com/#/articles/Search-Engine-Friendly/ Disclaimer: I'm a co-founder of NOLOH.
- gruseom 16y ago
  I took a look. Your approach is to detect requests from spiders and respond with plain HTML content rather than the content wrapped in Javascript, etc., that a normal user would get. You address the obvious question, "But isn't that cloaking?" by saying no, the content itself is the same, so nobody should object. Fair enough, I happen to agree with you, but our opinion is irrelevant; what matters is whether Google consider this practice legitimate. Can you (or any HNer) tell me definitively whether they do or not? And has their policy changed recently with the introduction of this new crawlability spec?
mmastrac 16y ago
Facebook started using this style of URL a little while ago but I can't get it to serve the static equivalent up yet. My bet is that they'll be the first big name to ship this in production.
endergen 16y ago
Has google actually started doing this? I thought this was still in an exploratory stage.
prodigal_erik 16y ago
I don't get it. They're requiring authors to provide HTML content that lives behind a URL. How is this different than just requiring the application gracefully degrade to a plain HTML mode that's usable without JavaScript, and crawling that?
- gojomo 16y ago
  It's similar in effort required, but this more clearly tells Google to display as target URLs (and send searchers directly to) a site's #!fragment-ed pages, rather than their degraded/simple pages. (Of course, the site could accept visits to degraded pages, detect AJAX-capable browsers, then redirect the users to the preferred AJAXy URLs... but perhaps they don't even want the non-AJAX URLs to appear in normal use.)
  - stan_rogers 16y ago
    It shouldn't require a redirect to "grow" the context around the fragment, though -- which is kind of the point of progressive enhancement.
    - gojomo 16y ago
      You can see a problem that the Google convention handles better than 'degrade to simple pages with distinct URLs' with the 'Noloh' site touted elsewhere in this thread. Consider one of their AJAXy-#fragment pages: http://www.noloh.com/#/features/ http://www.noloh.com/#/features/ Search for a phrase on that page: ["NOLOH generates only the absolutely necessary concise"] Google finds and sends you to their 'simplified' version of the same page: http://www.noloh.com/?features/ http://www.noloh.com/?features/ ...which upon visit "grows" itself with a fragment to... http://www.noloh.com/?features/#/features/ http://www.noloh.com/?features/#/features/ Ugh! Does the site really want people on that URL, possibly bookmarking and sharing it? Probably not; they could be using a redirect on first Google-visit. Try clicking to another page from the double-feature page, like FAQs. You wind up at: http://www.noloh.com/?features/#/features/&faqs%2F= http://www.noloh.com/?features/#/features/&faqs%2F= Ugh, it just keeps getting worse. Using the Google #!fragment convention, the initial URL appearing in the index could be the simple/direct: http://www.noloh.com/#/features/ http://www.noloh.com/#/features/ Some sites will want that. One canonical #fragment-filled URL collects all the inbound traffic/linkjuice.
      - stan_rogers 16y ago
        That's the bad behaviour of a specific implementation (more than the context of the fragment is "grown") and the resulting URL cruft is entirely unnecessary. No redirect, at least at the browser end, should be necessary to serve the fragment in context.
korch 16y ago
Google has to do this, it's an existential threat to their money maker PageRank. Imagine a few years out, after most of the web is no longer organized as static documents linked together(nice for crawling!), but transforms into a real-time evolving mish-mash of web API's, re-re-aggregators, and interconnected web services. i.e. the top 10 social networking sites will account for 90%+ of user activity, and it's their APIs & data we'll all be using. You can't crawl that. Anyone else worry that Google's inevitable grand "evil" act will ironically be them holding back the web from transforming into this? Microsoft could have totally killed Google by hitting the fast-forward button on Ajax in 2004, by leveraging IE to make the whole web into the "Deep Web." You can't sell ads on what you can't crawl—just see what Apple is doing with iAd to carve out mobile.
patio11 16y ago
This kind of rubs me wrong, in that it is nakedly making Google's engineering problems into the Internets' engineering problems, there isn't even a scintilla of the usual "You're just making it better for your end users and our crawler just happens to benefit" fig leaf, and compliance will be about as optional as complying with HTTP because Google is navigation on the Internet.
- noibl 16y ago
  "making Google's engineering problems into the Internets' engineering problems" -- indeed, and more specifically, saying that because Google can't/won't embed a JS engine in their crawler, publishers should embed one in their webserver. For which they suggest you use Java or, alternatively, Java.
  - retube 16y ago
    Is that right? I have been wondering to what extent the Google crawler renders js. Given that it's Google, I was imagining that they probably do pretty much full DOM rendering of pages, as so many pages are now rendered dynamically. Of course, it's difficult to simulate human-interaction with the page, so I guess this is their solution to that - the web designer leaving a signpost.
gcb 16y ago
thats a cloackers wet dream.
robryan 16y ago
Wouldn't it be a lot easier for Google to build a JS parser, and just examine the requests and follow it through as a normal browser would?

32 points by romland 16y ago

Interestingly enough NOLOH (http://www.noloh.com http://www.noloh.com) made AJAX applications crawlable years ago. If you're interested in the specifics, see http://dev.noloh.com/#/articles/Search-Engine-Friendly/ http://dev.noloh.com/#/articles/Search-Engine-Friendly/ Disclaimer: I'm a co-founder of NOLOH.

gruseom 16y ago
I took a look. Your approach is to detect requests from spiders and respond with plain HTML content rather than the content wrapped in Javascript, etc., that a normal user would get. You address the obvious question, "But isn't that cloaking?" by saying no, the content itself is the same, so nobody should object. Fair enough, I happen to agree with you, but our opinion is irrelevant; what matters is whether Google consider this practice legitimate. Can you (or any HNer) tell me definitively whether they do or not? And has their policy changed recently with the introduction of this new crawlability spec?

mmastrac 16y ago

Facebook started using this style of URL a little while ago but I can't get it to serve the static equivalent up yet. My bet is that they'll be the first big name to ship this in production.

endergen 16y ago

Has google actually started doing this? I thought this was still in an exploratory stage.

prodigal_erik 16y ago

I don't get it. They're requiring authors to provide HTML content that lives behind a URL. How is this different than just requiring the application gracefully degrade to a plain HTML mode that's usable without JavaScript, and crawling that?

gojomo 16y ago
It's similar in effort required, but this more clearly tells Google to display as target URLs (and send searchers directly to) a site's #!fragment-ed pages, rather than their degraded/simple pages. (Of course, the site could accept visits to degraded pages, detect AJAX-capable browsers, then redirect the users to the preferred AJAXy URLs... but perhaps they don't even want the non-AJAX URLs to appear in normal use.)
- stan_rogers 16y ago
  It shouldn't require a redirect to "grow" the context around the fragment, though -- which is kind of the point of progressive enhancement.
  - gojomo 16y ago
    You can see a problem that the Google convention handles better than 'degrade to simple pages with distinct URLs' with the 'Noloh' site touted elsewhere in this thread. Consider one of their AJAXy-#fragment pages: http://www.noloh.com/#/features/ http://www.noloh.com/#/features/ Search for a phrase on that page: ["NOLOH generates only the absolutely necessary concise"] Google finds and sends you to their 'simplified' version of the same page: http://www.noloh.com/?features/ http://www.noloh.com/?features/ ...which upon visit "grows" itself with a fragment to... http://www.noloh.com/?features/#/features/ http://www.noloh.com/?features/#/features/ Ugh! Does the site really want people on that URL, possibly bookmarking and sharing it? Probably not; they could be using a redirect on first Google-visit. Try clicking to another page from the double-feature page, like FAQs. You wind up at: http://www.noloh.com/?features/#/features/&faqs%2F= http://www.noloh.com/?features/#/features/&faqs%2F= Ugh, it just keeps getting worse. Using the Google #!fragment convention, the initial URL appearing in the index could be the simple/direct: http://www.noloh.com/#/features/ http://www.noloh.com/#/features/ Some sites will want that. One canonical #fragment-filled URL collects all the inbound traffic/linkjuice.
    - stan_rogers 16y ago
      That's the bad behaviour of a specific implementation (more than the context of the fragment is "grown") and the resulting URL cruft is entirely unnecessary. No redirect, at least at the browser end, should be necessary to serve the fragment in context.

korch 16y ago

Google has to do this, it's an existential threat to their money maker PageRank. Imagine a few years out, after most of the web is no longer organized as static documents linked together(nice for crawling!), but transforms into a real-time evolving mish-mash of web API's, re-re-aggregators, and interconnected web services. i.e. the top 10 social networking sites will account for 90%+ of user activity, and it's their APIs & data we'll all be using. You can't crawl that. Anyone else worry that Google's inevitable grand "evil" act will ironically be them holding back the web from transforming into this? Microsoft could have totally killed Google by hitting the fast-forward button on Ajax in 2004, by leveraging IE to make the whole web into the "Deep Web." You can't sell ads on what you can't crawl—just see what Apple is doing with iAd to carve out mobile.

patio11 16y ago

This kind of rubs me wrong, in that it is nakedly making Google's engineering problems into the Internets' engineering problems, there isn't even a scintilla of the usual "You're just making it better for your end users and our crawler just happens to benefit" fig leaf, and compliance will be about as optional as complying with HTTP because Google is navigation on the Internet.

noibl 16y ago
"making Google's engineering problems into the Internets' engineering problems" -- indeed, and more specifically, saying that because Google can't/won't embed a JS engine in their crawler, publishers should embed one in their webserver. For which they suggest you use Java or, alternatively, Java.
- retube 16y ago
  Is that right? I have been wondering to what extent the Google crawler renders js. Given that it's Google, I was imagining that they probably do pretty much full DOM rendering of pages, as so many pages are now rendered dynamically. Of course, it's difficult to simulate human-interaction with the page, so I guess this is their solution to that - the web designer leaving a signpost.

gcb 16y ago

thats a cloackers wet dream.

robryan 16y ago

Wouldn't it be a lot easier for Google to build a JS parser, and just examine the requests and follow it through as a normal browser would?