7 ms·

Understand the JavaScript SEO basics

228 points by jaequery 7y ago

rhacker 7y ago
Is this the first confirmation from Google that they render JS/SPAs? I've heard rumors before...
- jayflux 7y ago
  Not really, there’s been a few posts and talks about it. This was back in 2014: https://webmasters.googleblog.com/2014/05/understanding-web-pages-better.html?m=1 https://webmasters.googleblog.com/2014/05/understanding-web-...
- provolone 7y ago
  You can see the results in 'fetch as Google' and in the structured data tool. Injected JS content is there, including changes to the page title and head. As always take SEO advice from Google with a grain of salt.
  - fabiomaia 7y ago
    > As always take SEO advice from Google with a grain of salt. Why?
    - bryanrasmussen 7y ago
      Well, is SEO advice just allows you to get rid of any impediments from rising to where you should in the results then you should trust them, but if SEO advice is a thing that can help you rise to a prominence that you should not have then that advice would be detrimental to Google and they would not give it to you. So really it matters if you believe in the light or dark sides of the SEO force.
- PudgePacket 7y ago
  They've confirmed it for years. It's less often than html only crawls though. Search their docs and you should find info about it.
tobr 7y ago
Search Engine Optimization? More like Googlebot Optimization, I think.
- a13n 7y ago
  When 95%+ of search traffic comes from Google, these two things are the same thing.
  - grandphuba 7y ago
    Isn't that his point
- walshemj 7y ago
  You have to be crawled and rendered first
kazinator 7y ago
A good search engine shouldn't process JS at all, and index only the content that is reachable via links pulled from HTML (which may, of course, be dynamic). The processing of JS explains a lot of the sheer garbage that is in the Google results. I think they do it because it ties into their business model. JS is needed to unearth the kind of crap that a certain segment of the user base is looking for. That certain segment is worth catering to because it consists of users who are likely to click on ads. Otherwise, nobody in their right mind would integrate JS into a crawler. "I'm going to write a program that automatically finds random source code, much of it malicious, and runs it!" Come on; without a business agenda justifying it, it's a complete non-starter.
- jdormit 7y ago
  This is so out-of-touch with the modern web. Lots of totally legitimate websites, including major news outlets and Wikipedia, render their web pages on the client using JavaScript. Whether or not this is a good thing is a separate issue (personally I see nothing wrong with it), but it should be obvious that a useful search engine in 2019 needs to be able to index JavaScript-rendered content.
  - irq11 7y ago
    I, for one, would welcome a force with the size and influence of Google telling devs to cut the Javascript crap. To me, it’s like violating the conditional rendering rule: there are plausible reasons to want to do it, but Google is right to push back on it. It’s not as if totally legitimate websites don’t have reasonable technical alternatives here.
    - cameronbrown 7y ago
      > I, for one, would welcome a force with the size and influence of Google telling devs to cut the JavaScript crap. Wasn't this basically AMP though? The solution to the monster web developers were creating?
      - 7y ago
rayshan 7y ago
I echo what others like @sharkweek are saying - Google has been saying that they index JavaScript for quite a while now. At I/O 2019 they said the crawler should be on the latest Chromium codebase that supports all modern JavaScript features. I built some test cases [0] to see if this is true, and just couldn't get Google to index content inserted via JavaScript. For example, scroll down to the Popover button test case. Maybe I'm doing something wrong? Open to feedback. Opensourced code is here [1]. [0]https://shan.io/will-it-index/ https://shan.io/will-it-index/ [1]https://github.com/rayshan/rayshan.github.io/tree/master/will-it-index https://github.com/rayshan/rayshan.github.io/tree/master/wil...
- jotto 7y ago
  I can confirm that they do not parse low ranking JavaScript sites, but they will parse higher ranking JavaScript sites. Bots aside (Twitter and Slack still don't parse JS), it is still in your best interest to server-side render or pre-render a JavaScript site. The time-to-first-paint difference always starts small, but as you add features the JS apps become so bloated you'd be better off starting over with a server-side rendered HTML app. (I am the creator of https://www.prerender.cloud/ https://www.prerender.cloud/)
  - rayshan 7y ago
    Thank you! Good point about bots not parsing JavaScript. I find it increasingly important to build and test structured data so users sharing the content within these walled gardens will have a good experience.
davnicwil 7y ago
For those here to find out how relevant server rendering is for SPAs in light of this, I think the answer is still very much 'better if you do'. Some relevant excerpts from the article (emphasis mine): > Keep in mind that server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript. > Once Googlebot's resources allow, a headless Chromium renders the page and executes the JavaScript. So, there are still no specifics on what extra delays might be involved, but this seems to suggest what I've heard about JS-rendered content still taking much longer than static content to update in the index (i.e. days to weeks) might indeed be the case. I.e. seems they are implying that while this works, static content is still preferred by them and takes priority.
sharkweek 7y ago
SEO consultant here (yeah, yeah). Please, still in 2019, be careful with JavaScript unless you're willing to deal with a lot of uncertainty. Google routinely says how things are supposed to work, but there are plenty of examples where their crawlers act inconsistent to public statements. To be clear, probably not Googlebot's fault though; we as a species do a lot of weird things when building websites. Just finished a big project with a client who had a client-side render of their react app for the customer facing website. Googlebot was VERY sporadic with how they were indexing certain parts of the site, and in general, the site lost out to inferior sites in more competitive SERPs. Server side render fixed a lot of this and rankings/traffic jumped. It's also worth noting that the Bing/Yahoo crawlers (still a notable chunk of web traffic!) can't crawl JS. You can ignore this chunk of market share if you want but someone is going to happily take it. As a general rule, my advice is always this: Make it as easy as possible for bots to crawl and index your site's content. The more hoops the crawler is jumping through, the worse your site will perform in search engines.
- NullPrefix 7y ago
  > Server side render fixed a lot of this and rankings/traffic jumped. Good, they should just delist client side rendered web pages.
  - cameronbrown 7y ago
    While I love to rag on SPAs as much as the next person, doing that is ripe for antitrust. "They're trying to force the web to look like what they want"
    - greglindahl 7y ago
      “You didn’t go to huge extra expense to index my website so I am going to sue you for anti-trust” - good luck. Google has been picky about executing js all along.
  - hazz99 7y ago
    Why? I enjoy writing client-side SPAs, and I can pump out a better UI/UX must faster than I could otherwise. My users don't care how I build it, just that the site works and is enjoyable to use. If I need server-side rendering I'll still write it with a client-side rendered (e.g. React), but tack on server-side rendering at the end.
tuukkah 7y ago
Perhaps the biggest gotcha was that Googlebot was running Chromium that was years old and required polyfills that you wouldn't otherwise include anymore. Since May they run a recent Chromium though: "The new evergreen Googlebot" https://news.ycombinator.com/item?id=20482235 https://news.ycombinator.com/item?id=20482235
mattmanser 7y ago
This feels like such a smoke and mirrors thing. This gem: Write compatible code Browsers offer many APIs and JavaScript is a quickly-evolving language. Googlebot has some limitations regarding which APIs and JavaScript features it supports. To make sure your code is compatible with Googlebot, follow our guidelines for troubleshooting JavaScript problems. Leads to this page: https://developers.google.com/search/docs/guides/fix-search-javascript https://developers.google.com/search/docs/guides/fix-search-... Which says absolutely nothing. It details nothing about what is supported. Utterly useless. No version of HTML running, no info about what gets run, no actual technical details at all.
ashton314 7y ago
A little off-topic, but one of the most exciting things in my mind about Phoenix LiveView [1] is that you can get some dynamic web applications and not have to navigate this JS-rendering minefield. When a LiveView page is request, the server sends over the full page as just static HTML. The JS half of LiveView then upgrades to a websocket connection, and your page becomes dynamic. Not a good fit for everything, but it's great if you want to add some fancy form validation to, say, your landing page. You can render whatever you want with LiveView, get your fancy dynamic stuff, without having to worry about making your page more indexable. [1]: https://github.com/phoenixframework/phoenix_live_view https://github.com/phoenixframework/phoenix_live_view
- pmarreck 7y ago
  1) huge elixir/phoenix fan, glad to see you promoting this here 2) but I'm really here to commend your comment on this now-flagged post that I now cannot respond to: https://news.ycombinator.com/item?id=20500181 https://news.ycombinator.com/item?id=20500181 . I don't know if HN mods understand the irony in deplatforming a post arguing against deplatforming, but I am 100% in agreement with you there.
  - ashton314 7y ago
    1) :) 2) Thank you! Is it really flagged? That's too bad. And funny, in a sad way.
    - pmarreck 7y ago
      It got unflagged. ??? (Sorry for OT)
arrty88 7y ago
I use middleware on my SPA’s static server that sends bots a lite SSR version of the page that gets SEO’d up pretty well. Best of both worlds
- wusatiuk 7y ago
  Showing the bot something different than the user could be seen as cloaking. We also do that in some cases. Just be really carefull with that strategy.

228 points by jaequery 7y ago

rhacker 7y ago

Is this the first confirmation from Google that they render JS/SPAs? I've heard rumors before...

jayflux 7y ago
Not really, there’s been a few posts and talks about it. This was back in 2014: https://webmasters.googleblog.com/2014/05/understanding-web-pages-better.html?m=1 https://webmasters.googleblog.com/2014/05/understanding-web-...
provolone 7y ago
You can see the results in 'fetch as Google' and in the structured data tool. Injected JS content is there, including changes to the page title and head. As always take SEO advice from Google with a grain of salt.
- fabiomaia 7y ago
  > As always take SEO advice from Google with a grain of salt. Why?
  - bryanrasmussen 7y ago
    Well, is SEO advice just allows you to get rid of any impediments from rising to where you should in the results then you should trust them, but if SEO advice is a thing that can help you rise to a prominence that you should not have then that advice would be detrimental to Google and they would not give it to you. So really it matters if you believe in the light or dark sides of the SEO force.
PudgePacket 7y ago
They've confirmed it for years. It's less often than html only crawls though. Search their docs and you should find info about it.

tobr 7y ago

Search Engine Optimization? More like Googlebot Optimization, I think.

a13n 7y ago
When 95%+ of search traffic comes from Google, these two things are the same thing.
- grandphuba 7y ago
  Isn't that his point
walshemj 7y ago
You have to be crawled and rendered first

kazinator 7y ago

A good search engine shouldn't process JS at all, and index only the content that is reachable via links pulled from HTML (which may, of course, be dynamic). The processing of JS explains a lot of the sheer garbage that is in the Google results. I think they do it because it ties into their business model. JS is needed to unearth the kind of crap that a certain segment of the user base is looking for. That certain segment is worth catering to because it consists of users who are likely to click on ads. Otherwise, nobody in their right mind would integrate JS into a crawler. "I'm going to write a program that automatically finds random source code, much of it malicious, and runs it!" Come on; without a business agenda justifying it, it's a complete non-starter.

jdormit 7y ago
This is so out-of-touch with the modern web. Lots of totally legitimate websites, including major news outlets and Wikipedia, render their web pages on the client using JavaScript. Whether or not this is a good thing is a separate issue (personally I see nothing wrong with it), but it should be obvious that a useful search engine in 2019 needs to be able to index JavaScript-rendered content.
- irq11 7y ago
  I, for one, would welcome a force with the size and influence of Google telling devs to cut the Javascript crap. To me, it’s like violating the conditional rendering rule: there are plausible reasons to want to do it, but Google is right to push back on it. It’s not as if totally legitimate websites don’t have reasonable technical alternatives here.
  - cameronbrown 7y ago
    > I, for one, would welcome a force with the size and influence of Google telling devs to cut the JavaScript crap. Wasn't this basically AMP though? The solution to the monster web developers were creating?
    - 7y ago

rayshan 7y ago

I echo what others like @sharkweek are saying - Google has been saying that they index JavaScript for quite a while now. At I/O 2019 they said the crawler should be on the latest Chromium codebase that supports all modern JavaScript features. I built some test cases [0] to see if this is true, and just couldn't get Google to index content inserted via JavaScript. For example, scroll down to the Popover button test case. Maybe I'm doing something wrong? Open to feedback. Opensourced code is here [1]. [0]https://shan.io/will-it-index/ https://shan.io/will-it-index/ [1]https://github.com/rayshan/rayshan.github.io/tree/master/will-it-index https://github.com/rayshan/rayshan.github.io/tree/master/wil...

jotto 7y ago
I can confirm that they do not parse low ranking JavaScript sites, but they will parse higher ranking JavaScript sites. Bots aside (Twitter and Slack still don't parse JS), it is still in your best interest to server-side render or pre-render a JavaScript site. The time-to-first-paint difference always starts small, but as you add features the JS apps become so bloated you'd be better off starting over with a server-side rendered HTML app. (I am the creator of https://www.prerender.cloud/ https://www.prerender.cloud/)
- rayshan 7y ago
  Thank you! Good point about bots not parsing JavaScript. I find it increasingly important to build and test structured data so users sharing the content within these walled gardens will have a good experience.

davnicwil 7y ago

For those here to find out how relevant server rendering is for SPAs in light of this, I think the answer is still very much 'better if you do'. Some relevant excerpts from the article (emphasis mine): > Keep in mind that server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript. > Once Googlebot's resources allow, a headless Chromium renders the page and executes the JavaScript. So, there are still no specifics on what extra delays might be involved, but this seems to suggest what I've heard about JS-rendered content still taking much longer than static content to update in the index (i.e. days to weeks) might indeed be the case. I.e. seems they are implying that while this works, static content is still preferred by them and takes priority.

sharkweek 7y ago

SEO consultant here (yeah, yeah). Please, still in 2019, be careful with JavaScript unless you're willing to deal with a lot of uncertainty. Google routinely says how things are supposed to work, but there are plenty of examples where their crawlers act inconsistent to public statements. To be clear, probably not Googlebot's fault though; we as a species do a lot of weird things when building websites. Just finished a big project with a client who had a client-side render of their react app for the customer facing website. Googlebot was VERY sporadic with how they were indexing certain parts of the site, and in general, the site lost out to inferior sites in more competitive SERPs. Server side render fixed a lot of this and rankings/traffic jumped. It's also worth noting that the Bing/Yahoo crawlers (still a notable chunk of web traffic!) can't crawl JS. You can ignore this chunk of market share if you want but someone is going to happily take it. As a general rule, my advice is always this: Make it as easy as possible for bots to crawl and index your site's content. The more hoops the crawler is jumping through, the worse your site will perform in search engines.

NullPrefix 7y ago
> Server side render fixed a lot of this and rankings/traffic jumped. Good, they should just delist client side rendered web pages.
- cameronbrown 7y ago
  While I love to rag on SPAs as much as the next person, doing that is ripe for antitrust. "They're trying to force the web to look like what they want"
  - greglindahl 7y ago
    “You didn’t go to huge extra expense to index my website so I am going to sue you for anti-trust” - good luck. Google has been picky about executing js all along.
- hazz99 7y ago
  Why? I enjoy writing client-side SPAs, and I can pump out a better UI/UX must faster than I could otherwise. My users don't care how I build it, just that the site works and is enjoyable to use. If I need server-side rendering I'll still write it with a client-side rendered (e.g. React), but tack on server-side rendering at the end.

tuukkah 7y ago

Perhaps the biggest gotcha was that Googlebot was running Chromium that was years old and required polyfills that you wouldn't otherwise include anymore. Since May they run a recent Chromium though: "The new evergreen Googlebot" https://news.ycombinator.com/item?id=20482235 https://news.ycombinator.com/item?id=20482235

mattmanser 7y ago

This feels like such a smoke and mirrors thing. This gem: Write compatible code Browsers offer many APIs and JavaScript is a quickly-evolving language. Googlebot has some limitations regarding which APIs and JavaScript features it supports. To make sure your code is compatible with Googlebot, follow our guidelines for troubleshooting JavaScript problems. Leads to this page: https://developers.google.com/search/docs/guides/fix-search-javascript https://developers.google.com/search/docs/guides/fix-search-... Which says absolutely nothing. It details nothing about what is supported. Utterly useless. No version of HTML running, no info about what gets run, no actual technical details at all.

ashton314 7y ago

A little off-topic, but one of the most exciting things in my mind about Phoenix LiveView [1] is that you can get some dynamic web applications and not have to navigate this JS-rendering minefield. When a LiveView page is request, the server sends over the full page as just static HTML. The JS half of LiveView then upgrades to a websocket connection, and your page becomes dynamic. Not a good fit for everything, but it's great if you want to add some fancy form validation to, say, your landing page. You can render whatever you want with LiveView, get your fancy dynamic stuff, without having to worry about making your page more indexable. [1]: https://github.com/phoenixframework/phoenix_live_view https://github.com/phoenixframework/phoenix_live_view

pmarreck 7y ago
1) huge elixir/phoenix fan, glad to see you promoting this here 2) but I'm really here to commend your comment on this now-flagged post that I now cannot respond to: https://news.ycombinator.com/item?id=20500181 https://news.ycombinator.com/item?id=20500181 . I don't know if HN mods understand the irony in deplatforming a post arguing against deplatforming, but I am 100% in agreement with you there.
- ashton314 7y ago
  1) :) 2) Thank you! Is it really flagged? That's too bad. And funny, in a sad way.
  - pmarreck 7y ago
    It got unflagged. ??? (Sorry for OT)

arrty88 7y ago

I use middleware on my SPA’s static server that sends bots a lite SSR version of the page that gets SEO’d up pretty well. Best of both worlds

wusatiuk 7y ago
Showing the bot something different than the user could be seen as cloaking. We also do that in some cases. Just be really carefull with that strategy.