11 ms·

Does Google execute JavaScript?

339 points by rrradical 10y ago

linkregister 10y ago
Good, uncomplicated article. If you can get Google servers to execute Javascript, that sounds like a possible attack vector. It's likely that Google runs these in a proprietary feature-sparse interpreter. The lack of AJAX would make it difficult to leak information about the black-box interpreter.
jwatte 10y ago
My pet theory is that part of the anonymous usage data Chrome sends back, is digested page contents that go into pagerank. And such browser level digesting would be on rendered pages (after JavaScript execution.) I have no other reason to believe it is true other than it's what I would do to distribute the job of crawling the web to my users if i were Google :-)
- Fogest 10y ago
  Yeah, it would make sense. There has to be more reason to giving people a free web browser other than just that it uses Google search.
  - neurostimulant 10y ago
    Chrome does send domains/urls entered in the omnibar to google (I think someone did the experiment to test it several years ago), but sending out page content to google? If it were true it'll cause a huge legal and privacy problem, especially in places with tight privacy law like Europe.
    - samsonradu 10y ago
      Exactly, imagine if Google sends the page content after you log into your bank account.
      - 4ad 10y ago
        And yet this is exactly what happens with automatic translation.
    - stuckagain 10y ago
      Microsoft did exactly that with their Bing toolbar. https://www.google.com/search?q=hiybbprqag https://www.google.com/search?q=hiybbprqag
  - matt4077 10y ago
    The reason is Google's interest in advancing the web platform. They're afraid of content migrating into either native apps or Facebook, and Chrome is their (somewhat successful) attempt to make the open web competitive.
- codedokode 10y ago
  I think that is unlikely because then SEO specialists would generate and send such data to improve the positions of their websites. But Google could use this to discover new URLs that are not linked anywhere.
  - a3_nm 10y ago
    Some URLs are private by design, so I think it would be an awfully bad idea to discover new URLs in this way.
    - codedokode 10y ago
      There were cases when such private URLs got into search engines. You should protect such URLs with authorization or at least hide them behind a POST form. Or block them with robots.txt. Also if a user follows a link from such page, the URL would be leaked via Referer header so it is not secure anyway.
      - puddintane 10y ago
        I feel robots.txt is the most effective to prevent a private site from being crawled by Google followed up by an inline <meta name="robots" content="noindex"> tag.
    - puddintane 10y ago
      Chrome by default searches Google with whatever you type in the top bar - so by not disabling that private URLs are sent to them and then are scanned (however if the correct robots.txt file is setup on those URLs then Google Crawlbot would stop at that point). I believe it was this setting (I have it disabled as well as other options - however I believe it's only this option that sends data back) [1]. [1] https://i.snag.gy/cmkGzp.jpg https://i.snag.gy/cmkGzp.jpg
    - deleted 10y ago
      [deleted]
nkkollaw 10y ago
I'm loading i18n before showing any content, and was afraid Google wouldn't index the content, but it didn't have any problem doing that.
pul 10y ago
IT has now also executed ajax: https://www.google.nl/search?q=site%3Adoesgoogleexecutejavascript.com+yes https://www.google.nl/search?q=site%3Adoesgoogleexecutejavas...
latenightcoding 10y ago
Nicely done! I have been writing crawlers for a while now and executing Javascript is very expensive and slow, even for Google. When I crawl the web I usually run javascript from a headless browser only on top priority sites.
- elorant 10y ago
  What I find very useful is to run Firefox through Selenium with a plugin installed to disable pictures. Then it's blazing fast.
nrjdhsbsid 10y ago
My pet theory is that Google actually developed chrome as a web crawler and that consumer release was to ensure that Google would always be able to crawl pages (since sites would always want to work properly with chrome) It also explains why they effectively killed flash and Java applets. They were competing technologies that weren't owned by Google and not crawlable. If they would have taken off, Google's position as top search engine could have been in danger
- deleted 10y ago
  [deleted]
- Gigablah 10y ago
  And here I was thinking that it was Apple and the iPhone that effectively killed Flash. Silly me.
  - nrjdhsbsid 10y ago
    They disliked flash for different reasons. Google for search and apple because flash games were too powerful, jobs wanted those games turned into iPhone apps. The offical reasons were speed and security, which was obviously a farce since all the major browser vendors wrote their own PDF viewers which is both slow and known for tons of security issues as well. They could have just as easily wrote their own fast and secure implementations of flash and Java instead of disabling them. Especially for Java given Google's deep experience with it and that they already have their own runtime. Apple may have fired first on flash but google recently disabled all NPAPI plugins in chrome as well. The reason... "Security". I guess code not made by Google isn't secure enough to be used on the internet?
    - morganvachon 10y ago
      I modded you up, not because I agree with everything you wrote, but because I want people to post counterpoints instead of brigading you. I think you're mistaken about Google not being able to index Flash; they've had that capability for about eight years to my knowledge. I do agree with you that Google's security model is broken in many ways, both on the web and in Android.
chinathrow 10y ago
I've checked a site I know which is using nothing but Angular/JS on the frontend towards PageSpeed Insights [1] and it fully failed that test - no results visible. Also, the whole page is not indexed but the root URL itself. No page snippet preview, nothing. [1] https://developers.google.com/speed/pagespeed/insights/ https://developers.google.com/speed/pagespeed/insights/
amorphid 10y ago
One random thought... Google goes to SomeWebsite.com. The site has only enough HTML to load a big ol' JavaScript app, which Google slowly crawls. Well, that JS app makes a bunch of AJAX calls. There's no reason I can think of that would prevent Google from remembering which AJAX calls were made, and then just crawling the URLs for those calls on subsequent visits. Why load SomeWebsite.com's JavaScript.com every time you want to index the site, when you can just remember that the JS calls SomeWebsite.com/some-endpoint.json? Sucking the JSON out of an endpoint might even be faster that indexing regular HTML. Haven't written a lot of crawlers, so I'm mostly guessing here.
- Shanea93 10y ago
  Because then the crawler would be making the assumption that the main page with the JS loader on it would never change. If it never loads that page and always assumes it should be requesting the XMLhttpRequest endpoints, then changes to the site as a whole might never be indexed.
- icebraining 10y ago
  You still have to run the JS on the original page to know which text loaded from the JSON actually gets shown. You might have N pages loading the same file but showing different things.
- codedokode 10y ago
  Crawling AJAX data alone makes no sense because it could be just a piece of JSON and Google needs a rendered HTML page with an URL it can show in the results. If you have some data that are not available at a separate URL (e.g. they are loaded when user presses a button), they will not be indexed.
franze 10y ago
For a more in depth look at how Google treats JS best watch this talk https://youtu.be/JlP5rBynK3E https://youtu.be/JlP5rBynK3E by Googles John Müller at an Angular conference.
codedokode 10y ago
Google executes JS but maybe not on every website. If you have a JS error reporting tool on such site then you can get reports from Google IP addresses. I saw them first maybe 4 or 5 years ago. Executing JS everywhere would require a lot of CPU time and I think Google prefers not to do that when possible. And indexing a JS app is a very complicated task anyway (it is difficult for a robot to even find navigation elements if they are implemented as div's with onclick handler instead of links) so you better use sitemaps to make sure the bot can find content. And I don't think it is necessary to index rich apps. It makes no sense to index a ticket search app (the data become outdated too fast) or an online spreadsheet editor. Just make indexable pages as server-rendered HTML pages and put their URLs into a sitemap. Also Google looks for strings in JS code that look like URLS (e.g var url = '/some/page') and crawls them later.
mootothemax 10y ago
I'm convinced that Google has several Googlebots that are run depending on how popular a site is. That is, new and low traffic sites are crawled by less intelligent bots, and as the site gets more visitors or better rankings, more complicated and resource intensive bots are deployed. How this might work with the most popular sites out there, the Amazons and Wikipedias of this world - I'm not so sure about that. If I were in charge, I'd be tempted to have customised bots and ranking weights for each of these exceptional sites. Sadly the chances of getting a real answer on this in my lifetime are close to zero.
- manojlds 10y ago
  You could join Google :)
  - dx034 10y ago
    The chances of becoming a part of the search team are still very low. Even most Googlers won't know the exact details.
    - tr1ck5t3r 10y ago
      Parse your content, then to finalise your page ranking, listen to what former Defence Secretary Donald Rumsfeld had to say. I am a Robot. https://en.wikipedia.org/wiki/There_are_known_knowns https://en.wikipedia.org/wiki/There_are_known_knowns
    - lern_too_spel 10y ago
      But within Google, you could ask somebody and get the right answer.
      - user5994461 10y ago
        And outside Google, you could look for Google employees on linkedin who may be in position to have the answer and ask nicely ;)
    - 10y ago
avitzurel 10y ago
Beyond the theory, the talks and the articles. I have multiple 100% JS rendered pages (blank page with no javascript). Google is crawling and indexing them with zero issues.
GoToRO 10y ago
My experience is that content hidden behind js will get indexed later, if at all, and it will be updated not as often. Also they will run js for bigger sites first, and not so much for smaller sites.
binaryanomaly 10y ago
While google is certainly the main search engine most people use, isn't it to some point also very important what other engines such as bing, yandex, baidu etc. do. If you have a professional website you want to be found also by these other engines. Until also these support javascript you may end up with a hybrid SEO architecture anyway which means nothing was gained?
- coldtea 10y ago
  >If you have a professional website you want to be found also by these other engines. Do you really? Except if you are interested in the Chinese market. In fact the inverse is probably more true: if you run one of those other search engines, then you want to be as good at indexing any particular site as Google is.
  - binaryanomaly 10y ago
    Well at first, Google is the major player but not at 100% market share, some figures say 80%. Different by countries, continents... Second supporting other platforms is also important, think of FB, Twitter, etc. So either this really becomes the new standard that everyone supports or you may end up with a hybrid or traditional approach if you care about other platforms as well - imho you should.
tigras2 10y ago
We got website, react+babel+ajax, we monitor those ajax requests because of bad scrappers:) And we constantly see googlebot. At least 1k request per day, google bot agent from google ip range. So yes google does ajax and does understand packed react also.
zitterbewegung 10y ago
I tried this on bing and duckduckgo and they only have in the description the body text.
lprubin 10y ago
Another, faster way to see what JavaScript google can crawl on your website is the google search console (previously known as google webmaster tools). They have a fetch as google button that allows you to enter a url on a site you own and see a visual rendering of how google crawlers see your page. It even gives you a side by side comparison of what the crawler sees vs what a user sees.
- m0th87 10y ago
  My company has seen major discrepancies between what this tool gives back vs how the page ultimately ends up getting indexed.
tyingq 10y ago
Google's announcement when they started parsing javascript: https://webmasters.googleblog.com/2014/05/understanding-web-pages-better.html https://webmasters.googleblog.com/2014/05/understanding-web-...
- cbr 10y ago
  They had already been doing it for a while by 2014. For example, I saw them doing it on my site in January 2012: https://www.jefftk.com/p/googlebot-running-javascript https://www.jefftk.com/p/googlebot-running-javascript (Disclosure: I work for Google, on unrelated stuff, though I didn't at the time I wrote that blog post.)
tyingq 10y ago
I wonder what Google does to avoid indexing too many pages. There are a fair number of SPAs and software like shopping carts that have a large number of checkboxes, pulldowns, knobs, dials, etc...that both change the content and the current url query params.
- appleiigs 10y ago
  I think it avoids indexing too many pages by not triggering the checkboxes, pulldowns, etc. Through Google Console/Webmaster Tools, you can tell Google which query params your website uses. But for my modest website, google only uses the page query (?page=3) and I don't notice it using the other queries.
obvio 10y ago
here's an interesting experiment from a while ago http://searchengineland.com/tested-googlebot-crawls-javascript-heres-learned-220157 http://searchengineland.com/tested-googlebot-crawls-javascri... tldr; google indexes js generated content.
crispytx 10y ago
Best post I've seen on Hacker News. I've always played it safe and never assumed Google would index content displayed dynamically with JavaScript, but now I know!
faragon 10y ago
Is that safe? E.g. exploits, privilege scalation, etc.
andrewstuart2 10y ago
This is a subject that really irks the engineering side of me. It's utterly ridiculous that engineering and efficiency decisions are so deeply affected by whether or not the largest search engine will properly index your content. Why is it that Google doesn't get flak for not discovering content that's engineered to send the absolute minimum over the wire, cache intelligently in localStorage and IndexedDB, and scale well by distributing the appropriate amount of rendering work to the client agent? Why can't I expose a (JSON/)REST-API-to-deep-link mapping and have Google just crawl my JSON data and understand (perhaps verifying programmatically some percent of the time) that the links they show in search will deep link appropriately to the structured JSON content they crawled? It's such a waste of talent and resources to force server-side rendering. There's obviously the resource cost of transmitting more repetitive content over the wire, and requiring servers to do more work that the client could do. (Yes, even with compression this will still be a higher cost, because more repeated sequences reduces the value of variable-length encoding). But more than that, what bothers me is that there's this false truth that server-side rendering is a requirement for modern architectures, which must result in hundreds of thousands of wasted engineering hours trying to enable the idea of server-side and client-side rendering with the same code. This is not about time-to-first-byte either. Yes, the user-perceived latency matters, but the idea that server rendering even solves this problem is again utterly false. Sure, the time to very first byte ever may be faster, but that's not a winning long-term strategy unless you never expect your client to request the same content twice (or come back to your site at all). When properly cached and synchronized, the client-side-only app has many orders of magnitude faster TTFB, because it's coming from disk or even memory, and can be shown immediately. The only thing left to do is ask the server "what's new since my last timestamp?" All of these benefits seem to be completely disregarded 99% of the time because the golden "SEO" handcuffs are already on. I really hope we can get away from this mindset as a community and rather let the better-engineered and sites with the best and fastest UX over time will start driving search engine technology, instead of the other way around.
- nolok 10y ago
  Uhh, the answer to all your "why"s is pretty much" because that would be an open door to abuse"? For better or worse, Google has figured that the best way for them to have the most accurate indexing is to get things the exact same way browsers do, and figure it out from there.
tgtweak 10y ago
This is a great way to test a hypothesis, and a good experiment. I'll mention that there is a ule that was added a few years back, in backbone.js days, that urls with /#!route anchors will enable (read: force) ajax requests and JavaScript from the spider. Still remains a helpful way to force caching/indexing of JavaScript-only pages in Google.
jotto 10y ago
If you don't want the deal with the ambiguity of whether your AJAX will run or not, I'll shamelessly suggest https://www.prerender.cloud/ https://www.prerender.cloud/ which is helping a few sites who couldn't get Google to execute their AJAX.
jorblumesea 10y ago
I think one of the most frustrating things about indexing from Google is the complete lack of transparency. I understand that it helps Google slow down the arms race of search engines, but it also means that devs doing 100% banal work need to sift through mountains of rumors and spin up sites to test assumptions. I have literally heard every combination of practices with regards to SEO and have no idea what is truly correct. Every source contradicts each other, Google employee statements contradict those, etc.
korzun 10y ago
They have been for almost three years...
user5994461 10y ago
> Does Google execute JavaScript? Yes. There are sites that can't be loaded without javascript that are indexed fine by Google. The only explanation is that they run some javascript.
xg15 10y ago
If I click the link from the article that leads to the webcache version, I get "yes, but embedded only". If I click the link within that page that leads to the exact same webcache url, I get "yes, embedded and external but no ajax". If I google the site, the preview text is the non-changing portion of the text only ("This is an experiment to...") - not even a "No". I think Google is just trolling us.

339 points by rrradical 10y ago

Good, uncomplicated article. If you can get Google servers to execute Javascript, that sounds like a possible attack vector. It's likely that Google runs these in a proprietary feature-sparse interpreter. The lack of AJAX would make it difficult to leak information about the black-box interpreter.

jwatte 10y ago

My pet theory is that part of the anonymous usage data Chrome sends back, is digested page contents that go into pagerank. And such browser level digesting would be on rendered pages (after JavaScript execution.) I have no other reason to believe it is true other than it's what I would do to distribute the job of crawling the web to my users if i were Google :-)

Fogest 10y ago
Yeah, it would make sense. There has to be more reason to giving people a free web browser other than just that it uses Google search.
- neurostimulant 10y ago
  Chrome does send domains/urls entered in the omnibar to google (I think someone did the experiment to test it several years ago), but sending out page content to google? If it were true it'll cause a huge legal and privacy problem, especially in places with tight privacy law like Europe.
  - samsonradu 10y ago
    Exactly, imagine if Google sends the page content after you log into your bank account.
    - 4ad 10y ago
      And yet this is exactly what happens with automatic translation.
  - stuckagain 10y ago
    Microsoft did exactly that with their Bing toolbar. https://www.google.com/search?q=hiybbprqag https://www.google.com/search?q=hiybbprqag
- matt4077 10y ago
  The reason is Google's interest in advancing the web platform. They're afraid of content migrating into either native apps or Facebook, and Chrome is their (somewhat successful) attempt to make the open web competitive.
codedokode 10y ago
I think that is unlikely because then SEO specialists would generate and send such data to improve the positions of their websites. But Google could use this to discover new URLs that are not linked anywhere.
- a3_nm 10y ago
  Some URLs are private by design, so I think it would be an awfully bad idea to discover new URLs in this way.
  - codedokode 10y ago
    There were cases when such private URLs got into search engines. You should protect such URLs with authorization or at least hide them behind a POST form. Or block them with robots.txt. Also if a user follows a link from such page, the URL would be leaked via Referer header so it is not secure anyway.
    - puddintane 10y ago
      I feel robots.txt is the most effective to prevent a private site from being crawled by Google followed up by an inline <meta name="robots" content="noindex"> tag.
  - puddintane 10y ago
    Chrome by default searches Google with whatever you type in the top bar - so by not disabling that private URLs are sent to them and then are scanned (however if the correct robots.txt file is setup on those URLs then Google Crawlbot would stop at that point). I believe it was this setting (I have it disabled as well as other options - however I believe it's only this option that sends data back) [1]. [1] https://i.snag.gy/cmkGzp.jpg https://i.snag.gy/cmkGzp.jpg
  - deleted 10y ago
    [deleted]

nkkollaw 10y ago

I'm loading i18n before showing any content, and was afraid Google wouldn't index the content, but it didn't have any problem doing that.

pul 10y ago

IT has now also executed ajax: https://www.google.nl/search?q=site%3Adoesgoogleexecutejavascript.com+yes https://www.google.nl/search?q=site%3Adoesgoogleexecutejavas...

latenightcoding 10y ago

Nicely done! I have been writing crawlers for a while now and executing Javascript is very expensive and slow, even for Google. When I crawl the web I usually run javascript from a headless browser only on top priority sites.

elorant 10y ago
What I find very useful is to run Firefox through Selenium with a plugin installed to disable pictures. Then it's blazing fast.

nrjdhsbsid 10y ago

My pet theory is that Google actually developed chrome as a web crawler and that consumer release was to ensure that Google would always be able to crawl pages (since sites would always want to work properly with chrome) It also explains why they effectively killed flash and Java applets. They were competing technologies that weren't owned by Google and not crawlable. If they would have taken off, Google's position as top search engine could have been in danger

deleted 10y ago
[deleted]
Gigablah 10y ago
And here I was thinking that it was Apple and the iPhone that effectively killed Flash. Silly me.
- nrjdhsbsid 10y ago
  They disliked flash for different reasons. Google for search and apple because flash games were too powerful, jobs wanted those games turned into iPhone apps. The offical reasons were speed and security, which was obviously a farce since all the major browser vendors wrote their own PDF viewers which is both slow and known for tons of security issues as well. They could have just as easily wrote their own fast and secure implementations of flash and Java instead of disabling them. Especially for Java given Google's deep experience with it and that they already have their own runtime. Apple may have fired first on flash but google recently disabled all NPAPI plugins in chrome as well. The reason... "Security". I guess code not made by Google isn't secure enough to be used on the internet?
  - morganvachon 10y ago
    I modded you up, not because I agree with everything you wrote, but because I want people to post counterpoints instead of brigading you. I think you're mistaken about Google not being able to index Flash; they've had that capability for about eight years to my knowledge. I do agree with you that Google's security model is broken in many ways, both on the web and in Android.

chinathrow 10y ago

I've checked a site I know which is using nothing but Angular/JS on the frontend towards PageSpeed Insights [1] and it fully failed that test - no results visible. Also, the whole page is not indexed but the root URL itself. No page snippet preview, nothing. [1] https://developers.google.com/speed/pagespeed/insights/ https://developers.google.com/speed/pagespeed/insights/

amorphid 10y ago

One random thought... Google goes to SomeWebsite.com. The site has only enough HTML to load a big ol' JavaScript app, which Google slowly crawls. Well, that JS app makes a bunch of AJAX calls. There's no reason I can think of that would prevent Google from remembering which AJAX calls were made, and then just crawling the URLs for those calls on subsequent visits. Why load SomeWebsite.com's JavaScript.com every time you want to index the site, when you can just remember that the JS calls SomeWebsite.com/some-endpoint.json? Sucking the JSON out of an endpoint might even be faster that indexing regular HTML. Haven't written a lot of crawlers, so I'm mostly guessing here.

Shanea93 10y ago
Because then the crawler would be making the assumption that the main page with the JS loader on it would never change. If it never loads that page and always assumes it should be requesting the XMLhttpRequest endpoints, then changes to the site as a whole might never be indexed.
icebraining 10y ago
You still have to run the JS on the original page to know which text loaded from the JSON actually gets shown. You might have N pages loading the same file but showing different things.
codedokode 10y ago
Crawling AJAX data alone makes no sense because it could be just a piece of JSON and Google needs a rendered HTML page with an URL it can show in the results. If you have some data that are not available at a separate URL (e.g. they are loaded when user presses a button), they will not be indexed.

franze 10y ago

For a more in depth look at how Google treats JS best watch this talk https://youtu.be/JlP5rBynK3E https://youtu.be/JlP5rBynK3E by Googles John Müller at an Angular conference.

codedokode 10y ago

Google executes JS but maybe not on every website. If you have a JS error reporting tool on such site then you can get reports from Google IP addresses. I saw them first maybe 4 or 5 years ago. Executing JS everywhere would require a lot of CPU time and I think Google prefers not to do that when possible. And indexing a JS app is a very complicated task anyway (it is difficult for a robot to even find navigation elements if they are implemented as div's with onclick handler instead of links) so you better use sitemaps to make sure the bot can find content. And I don't think it is necessary to index rich apps. It makes no sense to index a ticket search app (the data become outdated too fast) or an online spreadsheet editor. Just make indexable pages as server-rendered HTML pages and put their URLs into a sitemap. Also Google looks for strings in JS code that look like URLS (e.g var url = '/some/page') and crawls them later.

mootothemax 10y ago

I'm convinced that Google has several Googlebots that are run depending on how popular a site is. That is, new and low traffic sites are crawled by less intelligent bots, and as the site gets more visitors or better rankings, more complicated and resource intensive bots are deployed. How this might work with the most popular sites out there, the Amazons and Wikipedias of this world - I'm not so sure about that. If I were in charge, I'd be tempted to have customised bots and ranking weights for each of these exceptional sites. Sadly the chances of getting a real answer on this in my lifetime are close to zero.

manojlds 10y ago
You could join Google :)
- dx034 10y ago
  The chances of becoming a part of the search team are still very low. Even most Googlers won't know the exact details.
  - tr1ck5t3r 10y ago
    Parse your content, then to finalise your page ranking, listen to what former Defence Secretary Donald Rumsfeld had to say. I am a Robot. https://en.wikipedia.org/wiki/There_are_known_knowns https://en.wikipedia.org/wiki/There_are_known_knowns
  - lern_too_spel 10y ago
    But within Google, you could ask somebody and get the right answer.
    - user5994461 10y ago
      And outside Google, you could look for Google employees on linkedin who may be in position to have the answer and ask nicely ;)
  - 10y ago

avitzurel 10y ago

Beyond the theory, the talks and the articles. I have multiple 100% JS rendered pages (blank page with no javascript). Google is crawling and indexing them with zero issues.

GoToRO 10y ago

My experience is that content hidden behind js will get indexed later, if at all, and it will be updated not as often. Also they will run js for bigger sites first, and not so much for smaller sites.

binaryanomaly 10y ago

While google is certainly the main search engine most people use, isn't it to some point also very important what other engines such as bing, yandex, baidu etc. do. If you have a professional website you want to be found also by these other engines. Until also these support javascript you may end up with a hybrid SEO architecture anyway which means nothing was gained?

coldtea 10y ago
>If you have a professional website you want to be found also by these other engines. Do you really? Except if you are interested in the Chinese market. In fact the inverse is probably more true: if you run one of those other search engines, then you want to be as good at indexing any particular site as Google is.
- binaryanomaly 10y ago
  Well at first, Google is the major player but not at 100% market share, some figures say 80%. Different by countries, continents... Second supporting other platforms is also important, think of FB, Twitter, etc. So either this really becomes the new standard that everyone supports or you may end up with a hybrid or traditional approach if you care about other platforms as well - imho you should.

tigras2 10y ago

We got website, react+babel+ajax, we monitor those ajax requests because of bad scrappers:) And we constantly see googlebot. At least 1k request per day, google bot agent from google ip range. So yes google does ajax and does understand packed react also.

zitterbewegung 10y ago

I tried this on bing and duckduckgo and they only have in the description the body text.

lprubin 10y ago

Another, faster way to see what JavaScript google can crawl on your website is the google search console (previously known as google webmaster tools). They have a fetch as google button that allows you to enter a url on a site you own and see a visual rendering of how google crawlers see your page. It even gives you a side by side comparison of what the crawler sees vs what a user sees.

m0th87 10y ago
My company has seen major discrepancies between what this tool gives back vs how the page ultimately ends up getting indexed.

tyingq 10y ago

Google's announcement when they started parsing javascript: https://webmasters.googleblog.com/2014/05/understanding-web-pages-better.html https://webmasters.googleblog.com/2014/05/understanding-web-...

cbr 10y ago
They had already been doing it for a while by 2014. For example, I saw them doing it on my site in January 2012: https://www.jefftk.com/p/googlebot-running-javascript https://www.jefftk.com/p/googlebot-running-javascript (Disclosure: I work for Google, on unrelated stuff, though I didn't at the time I wrote that blog post.)

tyingq 10y ago

I wonder what Google does to avoid indexing too many pages. There are a fair number of SPAs and software like shopping carts that have a large number of checkboxes, pulldowns, knobs, dials, etc...that both change the content and the current url query params.

appleiigs 10y ago
I think it avoids indexing too many pages by not triggering the checkboxes, pulldowns, etc. Through Google Console/Webmaster Tools, you can tell Google which query params your website uses. But for my modest website, google only uses the page query (?page=3) and I don't notice it using the other queries.

obvio 10y ago

here's an interesting experiment from a while ago http://searchengineland.com/tested-googlebot-crawls-javascript-heres-learned-220157 http://searchengineland.com/tested-googlebot-crawls-javascri... tldr; google indexes js generated content.

crispytx 10y ago

Best post I've seen on Hacker News. I've always played it safe and never assumed Google would index content displayed dynamically with JavaScript, but now I know!

faragon 10y ago

Is that safe? E.g. exploits, privilege scalation, etc.

andrewstuart2 10y ago

This is a subject that really irks the engineering side of me. It's utterly ridiculous that engineering and efficiency decisions are so deeply affected by whether or not the largest search engine will properly index your content. Why is it that Google doesn't get flak for not discovering content that's engineered to send the absolute minimum over the wire, cache intelligently in localStorage and IndexedDB, and scale well by distributing the appropriate amount of rendering work to the client agent? Why can't I expose a (JSON/)REST-API-to-deep-link mapping and have Google just crawl my JSON data and understand (perhaps verifying programmatically some percent of the time) that the links they show in search will deep link appropriately to the structured JSON content they crawled? It's such a waste of talent and resources to force server-side rendering. There's obviously the resource cost of transmitting more repetitive content over the wire, and requiring servers to do more work that the client could do. (Yes, even with compression this will still be a higher cost, because more repeated sequences reduces the value of variable-length encoding). But more than that, what bothers me is that there's this false truth that server-side rendering is a requirement for modern architectures, which must result in hundreds of thousands of wasted engineering hours trying to enable the idea of server-side and client-side rendering with the same code. This is not about time-to-first-byte either. Yes, the user-perceived latency matters, but the idea that server rendering even solves this problem is again utterly false. Sure, the time to very first byte ever may be faster, but that's not a winning long-term strategy unless you never expect your client to request the same content twice (or come back to your site at all). When properly cached and synchronized, the client-side-only app has many orders of magnitude faster TTFB, because it's coming from disk or even memory, and can be shown immediately. The only thing left to do is ask the server "what's new since my last timestamp?" All of these benefits seem to be completely disregarded 99% of the time because the golden "SEO" handcuffs are already on. I really hope we can get away from this mindset as a community and rather let the better-engineered and sites with the best and fastest UX over time will start driving search engine technology, instead of the other way around.

nolok 10y ago
Uhh, the answer to all your "why"s is pretty much" because that would be an open door to abuse"? For better or worse, Google has figured that the best way for them to have the most accurate indexing is to get things the exact same way browsers do, and figure it out from there.

tgtweak 10y ago

This is a great way to test a hypothesis, and a good experiment. I'll mention that there is a ule that was added a few years back, in backbone.js days, that urls with /#!route anchors will enable (read: force) ajax requests and JavaScript from the spider. Still remains a helpful way to force caching/indexing of JavaScript-only pages in Google.

jotto 10y ago

If you don't want the deal with the ambiguity of whether your AJAX will run or not, I'll shamelessly suggest https://www.prerender.cloud/ https://www.prerender.cloud/ which is helping a few sites who couldn't get Google to execute their AJAX.

jorblumesea 10y ago

I think one of the most frustrating things about indexing from Google is the complete lack of transparency. I understand that it helps Google slow down the arms race of search engines, but it also means that devs doing 100% banal work need to sift through mountains of rumors and spin up sites to test assumptions. I have literally heard every combination of practices with regards to SEO and have no idea what is truly correct. Every source contradicts each other, Google employee statements contradict those, etc.

korzun 10y ago

They have been for almost three years...

user5994461 10y ago

> Does Google execute JavaScript? Yes. There are sites that can't be loaded without javascript that are indexed fine by Google. The only explanation is that they run some javascript.

xg15 10y ago

If I click the link from the article that leads to the webcache version, I get "yes, but embedded only". If I click the link within that page that leads to the exact same webcache url, I get "yes, embedded and external but no ajax". If I google the site, the preview text is the non-changing portion of the text only ("This is an experiment to...") - not even a "No". I think Google is just trolling us.