6 ms·

The whole Lyft site shows nothing without Javascript enabled. Even the title is set via script. Google indexed titles and content for all pages though. How are

by xyby 12y ago

The whole Lyft site shows nothing without Javascript enabled. Even the title is set via script. Google indexed titles and content for all pages though. How are they doing that? Are the delivering different content based on ip or useragent or something?

akavi 12y ago
Google crawlers have been able to execute javascript since at least May of this year[0]. [0] http://googlewebmastercentral.blogspot.ca/2014/05/understanding-web-pages-better.html http://googlewebmastercentral.blogspot.ca/2014/05/understand...
- xyby 12y ago
  Interesting. The content must have changed recently then. Because I see different text in the Google serps then on the page with js enabled. For lyft.com Google shows me "Lyft is your friend with a car" which is not on the page. For lyft.com/stories Google shows "Discover extraordinary stories from the Lyft community" which is also not on the page. Maybe it's from good old metatags. The have some, but somehow they apply a scripting engine (angular?) to them: <meta property="og:title" content="{{meta.title || 'A ride whenever you need one' }}{{meta.signoff || ' - Lyft'}}"> The "Discover extraordinary stories" text is not in the metatags though.
  - fnom 12y ago
    they dont necessarily give the google bot this version of the page. its perfectly ok to prerender the page with something like phantomjs and send the crawler the prerendered pages. this was , to my knowledge, quite common with angularjs apps that relied heavily on seo/google juice.
- mootothemax 12y ago
  Google crawlers have been able to execute javascript since at least May of this year[0]. As I wrote in another comment, that hasn't stopped Google requested pages via _escaped_fragment_.
patrickyeon 12y ago
Googlebot has been executing some limited amount of JS for some time. See http://googlewebmastercentral.blogspot.com/2011/11/get-post-and-safely-surfacing-more-of.html http://googlewebmastercentral.blogspot.com/2011/11/get-post-...
sveme 12y ago
Google crawlers execute Javascript [1] since this year. Before that you apparently had to deliver a pre-rendered page to the crawler. I don't know anything about the lyft stack, but there are solutions for SEO with Single-Page Apps (see e.g. [2]). [1]: http://googlewebmastercentral.blogspot.no/2014/05/understanding-web-pages-better.html http://googlewebmastercentral.blogspot.no/2014/05/understand... [2]: http://www.yearofmoo.com/2012/11/angularjs-and-seo.html http://www.yearofmoo.com/2012/11/angularjs-and-seo.html
- mootothemax 12y ago
  Google crawlers execute Javascript [1] since this year. Before that you apparently had to deliver a pre-rendered page to the crawler. For what it's worth, Google and Bing both still make _escaped_fragment_ requests to our AngularJS site. I'm not sure if this means that Google prefer prerendered views over the standard website. For now, it seems sensible to implement.
nmjohn 12y ago
Google crawlers do index javascript now as many people have already pointed out. The other common practice is to use a "prerender" server which uses a tool like phantomjs to render the HTML of the page. When googlebot or any other robot hits your http server, you check the useragent, and if it matches known robots you forward the request to prerender instead of your app server.
- xyby 12y ago
  The prerendering might be in place here. When you set your useragent to Googlebot, you get a page with tons of html. Still nothing readable though. The textcontent Google indexed is not in the body of this but in the metatags. Strange. They could have put it in the metatags independent of the useragent, right?
mootothemax 12y ago
Looks like they're following Google's guidelines[1], and passing an "_escaped_fragment_" parameter returns a pre-rendered view. Compare the source of the original page with this one: https://www.lyft.com/work?_escaped_fragment_= https://www.lyft.com/work?_escaped_fragment_= Implementing this is thankfully not too tricky. Whilst there are various SaaS providers that will take care of it for you, it's relatively simple to add a rule that checks for the parameter, loads up the requested page in PhantomJS, and returns the rendered HTML. Add a bit of caching, and the job's done. [1] https://support.google.com/webmasters/answer/174992?hl=en https://support.google.com/webmasters/answer/174992?hl=en
- xyby 12y ago
  https://www.lyft.com/work?_escaped_fragment_= Interesting. How did they make google check for that url in the first place? Via a sitemap or something?
  - mootothemax 12y ago
    Interesting. How did they make google check for that url in the first place? Via a sitemap or something? By including the following meta tag: <meta name="fragment" content="!"> Google provide their own guide for this: https://support.google.com/webmasters/answer/174992?hl=en https://support.google.com/webmasters/answer/174992?hl=en
    - xyby 12y ago
      Wow. That's nice. I wonder whats better: deliver the content via a <noscript> element or via the fragment url.

by xyby 12y ago

akavi 12y ago

Google crawlers have been able to execute javascript since at least May of this year[0]. [0] http://googlewebmastercentral.blogspot.ca/2014/05/understanding-web-pages-better.html http://googlewebmastercentral.blogspot.ca/2014/05/understand...

xyby 12y ago
Interesting. The content must have changed recently then. Because I see different text in the Google serps then on the page with js enabled. For lyft.com Google shows me "Lyft is your friend with a car" which is not on the page. For lyft.com/stories Google shows "Discover extraordinary stories from the Lyft community" which is also not on the page. Maybe it's from good old metatags. The have some, but somehow they apply a scripting engine (angular?) to them: <meta property="og:title" content="{{meta.title || 'A ride whenever you need one' }}{{meta.signoff || ' - Lyft'}}"> The "Discover extraordinary stories" text is not in the metatags though.
- fnom 12y ago
  they dont necessarily give the google bot this version of the page. its perfectly ok to prerender the page with something like phantomjs and send the crawler the prerendered pages. this was , to my knowledge, quite common with angularjs apps that relied heavily on seo/google juice.
mootothemax 12y ago
Google crawlers have been able to execute javascript since at least May of this year[0]. As I wrote in another comment, that hasn't stopped Google requested pages via _escaped_fragment_.

patrickyeon 12y ago

Googlebot has been executing some limited amount of JS for some time. See http://googlewebmastercentral.blogspot.com/2011/11/get-post-and-safely-surfacing-more-of.html http://googlewebmastercentral.blogspot.com/2011/11/get-post-...

sveme 12y ago

Google crawlers execute Javascript [1] since this year. Before that you apparently had to deliver a pre-rendered page to the crawler. I don't know anything about the lyft stack, but there are solutions for SEO with Single-Page Apps (see e.g. [2]). [1]: http://googlewebmastercentral.blogspot.no/2014/05/understanding-web-pages-better.html http://googlewebmastercentral.blogspot.no/2014/05/understand... [2]: http://www.yearofmoo.com/2012/11/angularjs-and-seo.html http://www.yearofmoo.com/2012/11/angularjs-and-seo.html

mootothemax 12y ago
Google crawlers execute Javascript [1] since this year. Before that you apparently had to deliver a pre-rendered page to the crawler. For what it's worth, Google and Bing both still make _escaped_fragment_ requests to our AngularJS site. I'm not sure if this means that Google prefer prerendered views over the standard website. For now, it seems sensible to implement.

nmjohn 12y ago

Google crawlers do index javascript now as many people have already pointed out. The other common practice is to use a "prerender" server which uses a tool like phantomjs to render the HTML of the page. When googlebot or any other robot hits your http server, you check the useragent, and if it matches known robots you forward the request to prerender instead of your app server.

xyby 12y ago
The prerendering might be in place here. When you set your useragent to Googlebot, you get a page with tons of html. Still nothing readable though. The textcontent Google indexed is not in the body of this but in the metatags. Strange. They could have put it in the metatags independent of the useragent, right?

mootothemax 12y ago

Looks like they're following Google's guidelines[1], and passing an "_escaped_fragment_" parameter returns a pre-rendered view. Compare the source of the original page with this one: https://www.lyft.com/work?_escaped_fragment_= https://www.lyft.com/work?_escaped_fragment_= Implementing this is thankfully not too tricky. Whilst there are various SaaS providers that will take care of it for you, it's relatively simple to add a rule that checks for the parameter, loads up the requested page in PhantomJS, and returns the rendered HTML. Add a bit of caching, and the job's done. [1] https://support.google.com/webmasters/answer/174992?hl=en https://support.google.com/webmasters/answer/174992?hl=en

xyby 12y ago
https://www.lyft.com/work?_escaped_fragment_= Interesting. How did they make google check for that url in the first place? Via a sitemap or something?
- mootothemax 12y ago
  Interesting. How did they make google check for that url in the first place? Via a sitemap or something? By including the following meta tag: <meta name="fragment" content="!"> Google provide their own guide for this: https://support.google.com/webmasters/answer/174992?hl=en https://support.google.com/webmasters/answer/174992?hl=en
  - xyby 12y ago
    Wow. That's nice. I wonder whats better: deliver the content via a <noscript> element or via the fragment url.