12 ms·

Show HN: SEO for JavaScript, HTML5 and Single Page Applications

50 points by CMCDragonkai 12y ago

CMCDragonkai 12y ago
Hey HN, Roger here, happy to answer any questions.
- ilaksh 12y ago
  This seems exactly like prerender.io but with more code to insert (harder to use) and more expensive..
  - CMCDragonkai 12y ago
    I calculated the average price per usage versus Prerender.io for average sites. And it comes out to be cheaper or equivalent to the standard price of Prerender.io. Basically take the amount of usages you use of Prerender.io per month, and compare it yourself. What do you mean by more code to insert? I do know of Prerender.io but more code depends on which framework/language you're using. If it's node. It's pretty much one line. The extra code is just options that allow you to customise the way the interceptor works, these options include things like blacklist, whitelist, regex, extensions ignoring, client side caching.. etc. But the basic usage is the just key + email, and you're good to go. Also note that unlike Prerender, our cache storage is free, so we don't track how many pages you have.
    - gildas 12y ago
      Same question than the one asked some months ago [1]: > Why should someone use your service instead of existing ones? What is your added-value compared for example to seo4ajax.com? It loooks like the innovation is just to use Firefox instead of PhantomJS. Maybe I am wrong. [1] https://news.ycombinator.com/item?id=6781843 https://news.ycombinator.com/item?id=6781843
      - CMCDragonkai 12y ago
        Hey gildas, I answered a similar question here https://news.ycombinator.com/item?id=7765731 https://news.ycombinator.com/item?id=7765731
    - ilaksh 12y ago
      I see three lines of code whereas prerender.io is one line. I guess that's because they don't need you to configure the client with the API key or email. That's not really a huge difference, although it is still a little bit more effort. It really comes down to price I think. I can't really tell for sure what calculation to use to compare the pricing. Looks like you might be cheaper. You might consider doing a blog or something comparing the pricing though because I think other people know about prerender also and having a simple explanation that proves your pricing is better would make it easier for people to make that decision.
      - CMCDragonkai 12y ago
        Ah, well Prerender does use a single file, which means they haven't broken down their middleware into 3 components. All SnapSearch middleware is broken down (OOP style) into Detector, Client and Interceptor. I recognised, that this OOP strictness leads to a bit more verbosity. Therefore, for PHP I build a Stack interceptor. For Ruby, there's a Rack alternative. NodeJS has the expressConnector. Python doesn't have a single thing that brings them together atm. These integration points, make the entire thing a one liner to integrate. If you're referring to node, here's the connectInterceptor. Usage is on the README: https://github.com/SnapSearch/SnapSearch-Client-Node/blob/master/src/connectInterceptor.js https://github.com/SnapSearch/SnapSearch-Client-Node/blob/ma... Also sure, I'm going to get a blog up soon. Thanks for the advice. I'll do one on comparing pricing. But do note, that the fact that we use Firefox instead of QTWebkit means that we get 6 week release cycles so our scrapers keep up with the latest development HTML5 unlike the slower QTWebkit of PhantomJS. So we do have a technical advantage :)
    - sv123 12y ago
      Is there any difference in functionality or added benefit of using your service over prerender?
      - CMCDragonkai 12y ago
        Here's a shortlist of differences and benefits: 1. SnapSearch uses load balanced Firefox, this allows us to keep up with the latest in HTML5 unlike the QtWebkit of PhantomJS 2. SnapSearch does metered billing, this comes out cheaper for average sites, or sites that have lots of pages but dont send out tons of requests. We charge on usages, not cached requests, and not on cach storage. 3. SnapSearch's middleware is more robust, with 200 robots that it checks, and regularly updated robot list. There's a lot of flexibility in the code. 4. We provide you a bit more analytics in the control panel, so you can see things like request portion from multiple domains, requests/usages from search engines over a month. A progress bar on usages bar. A cache storage list of all the snapshots you have so you can precache pages if you wish. Try it out, it's free for 1000 usages per month! Also note that our API supports custom javascript callbacks, which allows you to do all sorts of interesting things prior to the caching of the snapshot. Furthermore, we also support soft 404s and redirections (including meta tag, header and js redirections (synchronoous and asynchronous). All of it is explained on the documentation. Flash support coming soon too.
andrenotgiant 12y ago
When you scrape the HTML from a single page app (let's say the default app homepage) do you re-write the javascript links to be HTML links with unique URLs? If not, how does Google find anything more than the app homepage?
- albemuth 12y ago
  Sitemap works
  - CMCDragonkai 12y ago
    Yep a sitemap can also work. But it's not needed, since Google will follow your anchored links anyway. SnapSearch in the future can also create sitemaps dynamically.
- CMCDragonkai 12y ago
  Your links on your homepage, should already be "<a></a>" anchor links. When google hits your page, and it's served up by SnapSearch, then it will automatically find those anchor tags and navigate to the subpages. If you're talking about buttons, perhaps Google will click on those buttons. But any links should be represented by anchor links. This works for SnapSeach.io. Check on Google, search https://snapsearch.io https://snapsearch.io and all the subpages of SnapSearch are indexed. Also works for http://dreamitapp.com http://dreamitapp.com So to answer your question, there's no need to rewrite your links.
programminggeek 12y ago
Am I crazy for thinking that SEO is a huge reason to NOT do a Single Page App for a public facing website that you want Google to index? It seems like the wrong business choice from a marketing perspective that we are now looking for a tech solution to solve a problem we created ourselves. Single Page Apps feel like building Flash web apps 5 years ago. It's a valid choice, but with clear drawbacks.
- CMCDragonkai 12y ago
  Well SnapSearch allows you to make dynamic web applications without worrying about SEO problems. It's not so much a technical solution to a technical problem that we created ourselves, it's the fact that search engines and social network robots have not caught up to next generation websites (which I believe will be a fully dynamic user experience). We're just pushing the web ahead.
- integraton 12y ago
  You are not crazy. The single page app trend is solidly in the "peak of inflated expectations" phase of the hype cycle ( http://en.wikipedia.org/wiki/Hype_cycle http://en.wikipedia.org/wiki/Hype_cycle ), which is why there exists a subculture of developers who are trying to use the approach for everything, even where it isn't appropriate.
  - CMCDragonkai 12y ago
    I think the single page app trend is coinciding with the growth of javascript, HTML5. It shows that the websites of the future will be more and more like web applications, and they need dynamic UI. The thing that's stopping us the SEO problem, which I hope is no longer problem anymore with snapsearch.
    - hippee-lee 12y ago
      Why not just render the public pages of your app out to HTML or use one of the phantom JS services to do it. Then when spiders come up the waterspout have your web server tell them where to go for your sites links. If you don't ant to do it yourself there are quite a few companies that can help: http://scotch.io/tutorials/javascript/angularjs-seo-with-prerender-io http://scotch.io/tutorials/javascript/angularjs-seo-with-pre...
fidlefodl 12y ago
I feel like applications and tooling to pre-render your HTML (eg: Reactjs) are more interesting. It's definitely more work currently (when done right), but it seems like the most natural flow. Always render html, let clientside link it up, everybody wins.
timdorr 12y ago
I think the logo could use a little work. The lightning bolt doesn't come off as an S, so it looks more like "napSearch".
- CMCDragonkai 12y ago
  Thanks for the feedback :). I'll look into it. Did you know the bolt is also a tiger stripe? That's why I got the tiger theme.
mattlenz 12y ago
Will Google eventually begin javascript-enabled scraping themselves?
- CMCDragonkai 12y ago
  They may one day. But it's a big computational overhead from straight curling to running a JS VM for everything. Right now SPAs are still a minority on the web. But I guess the question is will you wait for Google to support your site and lose out on the awesomeness of single page applications?
- Encosia 12y ago
  I don't know the specifics, but they already have been for years (at least to some extent). Several of my clients have found content deep inside public-facing client-side apps indexed in situations where we didn't care if it was indexed, but assumed it wouldn't/couldn't be.
  - CMCDragonkai 12y ago
    Do you have any examples? Because I've been building single page apps for 2 years, and none of it is indexed. Also indexing for search is not the only thing, sometimes clients need google adverts, which require the google media bot check your site's content. One of my clients could not get any data out, and the site could not get approved by Google for the adverts. But with SnapSearch, the google media bot was able to extract the necessary information. Check your google webmaster tools, and you can force a scrape from google bots, and you can see that SPA sites are not discoverable.
    - Encosia 12y ago
      I can't share the specific examples, unfortunately. That's right that if you force a googlebot request from the webmaster tools, it's essentially just how the basic googlebot would wget a page on your site. We know they're doing more than that though, because it would be trivial to hide cloaking and malware from Google if that's the only tool they used for scraping. What Google actually does for crawling content seems to be a bit more nuanced. If you watch the server-side logs on a site that Google crawls often, you'll sometimes see a pattern of googlebot crawling a page and then an immediate subsequent request from a Google IP from a Chrome UA. I'm sure that's partially just to assess malware/cloaking, but it's probably also related to developments like this one: https://twitter.com/mattcutts/status/131425949597179904 https://twitter.com/mattcutts/status/131425949597179904 They're definitely doing more than just the ?_escaped_fragment_ rigamarole that almost no one uses. Fair point about the AdSense crawler though. It seems to be just about as dumb as a rock in my experience.
puppetmaster3 12y ago
Dear N00bs, SPA works great w/ SEO. Hashtags! (ex: https://github.com/puppetMaster3/ModuleBU/blob/master/latestCDN/lib/ModuleBU.js https://github.com/puppetMaster3/ModuleBU/blob/master/latest... in AppBu section, github has examples). Just because twitter or Facebook programers don't know how to do it, but then they do hire 0 experience out of school. Cheers.
- puppetmaster3 12y ago
  Down votes? It upsets you that I show you how to SEO w/ SPA? Says a bit about you I think.
  - puppetmaster3 12y ago
    Dear downvoter: http://theunboundedspirit.com/wp-content/uploads/2014/05/truth-quote-nietzsche.jpg http://theunboundedspirit.com/wp-content/uploads/2014/05/tru...
ant_sz 12y ago
I think this should be more a problem for google than for web developer. web developer have the right to choose new technique they like, it's google's responsibility to scratch your content well. But now things are reversed, It's kind of monopoly, isn't it?
- CMCDragonkai 12y ago
  But it's not just google now. It's anybody who does any kind of crawling on the web, which includes social media robots (that acquire images from the link), personal web crawlers, private internal search engine, data mining, even advertisement bots that check content. Would they all have to embed and maintain/update a JS VM?
- erichurkman 12y ago
  Developers have always been free to choose techniques. Search engines owe you, the developer, nothing. If Google et al can't scrape your content, they'll scrape a competitor who better serves parse-able content. If you want to play ball with a particular search engine, you have to serve content that they can parse at the time.
nnq 12y ago
Am I crazy thinking that we as web developers should NOT HAVE TO have to care about this? It's Google's and Bing's and Yahoo's problem to go build decent spiders that run multiple versions of the most recent web browsers inside them, and they do have the computing power that's needed. All spiders/crawlers should see webpages just as human users see them that's the point! They should do whatever it takes to get this working, regardless the effort, because this is why they get paid (indirectly, though adds, but still...). And this is still not enough: it's any respectable search engine's duty to try and develop near-human-level AIs to be able to also extract meaningful information from visual structure, from pictures, and from hard to understand prose. And longer term, it will be their duty to develop above-human-level AIs to be even better at this than humans so they can provide even better search results. Since when are we, web developers, expected to do the search engines' work for them?! I expect this is just a transition phase and things will get on the right track soon. (Note: and no, I'm not against helping search engines. Providing semantic markup, microformats and all this is great. But there is a difference between helping them and doing the work for them, on our dime!) EDIT+: @OP: I'm not critical, what you and prerender.io are doing is awesome and immensely helpful for web developers! But you do realize that you're doing what search engines should be doing themselves, right? And when they will wake up and realize they should get into the game, they will have basically put you out of business... or buy your business, which I think is what you expect :) Anyway, good luck with it!
- CMCDragonkai 12y ago
  Hey I understand where you're coming from. I had the same impression before I began on SnapSearch. But right now SPAs are still a minority of the websites in the world, but eventually everybody will start making web applications. But do note, as I said in the other comments, it's not just search engines. There'll be a day when curlling sites will require a JS VM. There's a big difference between a standard curl vs a full blown chrome/firefox level VM. Considering the amount of sites google crawls per second, their computing costs would explode compared to what they're doing now. And this doesn't even address all the social network bots, and pretty much all the other bots doing stuff other than search indexing. So I think we're still got quite some time before we're redundant. At any case, the question for you is, are you going to wait for Google to implement this before you can start using single page application technologies which can give you an edge when it comes to customer user experience?
philbo 12y ago
If you're building JavaScript-heavy/single-page applications, you should be applying the principle of progressive enhancement anyway, there is no reason not to do so. Progressive enhancement brings a number of benefits, of which friendliness to search engines is just one. User has JS turned off? Page still works. Request to CDN fails? Page still works. Silly lint error slips through to production, breaking all your JS? Page still works. Weird cross-browser issue breaks your JS in some clients? Page still works. The most frequent arguments against progressive enhancement, which has been prevalent amongst a number of back-end and product people I've worked with, is that it is a waste of time and that it adds too much extra effort. I disagree with both. It is not always necessary to reach full feature parity between your JS and non-JS implementations, for instance an auto-suggest input in your JS code can happily be a vanilla input in the non-JS. But if you do just enough to ensure that your application is functional without JavaScript enabled, my (subjective, biased) experience has been that it doesn't take too long.
- Kiro 12y ago
  Easier said than done if you're using something like Angular or Ember.
  - philbo 12y ago
    If angular or ember make it difficult to follow best practices, are they the best tool for the job? Less trollishly, although I have no experience with angular or ember, I have built a progressively enhanced application with knockout and I thought it went quite well. As with any progressively enhanced application, it requires that your back-end is set up to respond with both markup and JSON responses, but the data- attributes responsible for binding behaviour are just ignored by the browser when script is not running. Perhaps my experience is too simplistic, though. Can you give me an example of the kind of things that angular and ember make difficult to progressively enhance?
  - Isofarro 12y ago
    Yehuda Katz, the lead on Ember.js was categorical about the use of ember.js when you require progressive enhancement: Don't use ember.js. So this is just a case of using the wrong tools for the job, buying the hype without validating the requirements.

50 points by CMCDragonkai 12y ago

CMCDragonkai 12y ago

Hey HN, Roger here, happy to answer any questions.

ilaksh 12y ago
This seems exactly like prerender.io but with more code to insert (harder to use) and more expensive..
- CMCDragonkai 12y ago
  I calculated the average price per usage versus Prerender.io for average sites. And it comes out to be cheaper or equivalent to the standard price of Prerender.io. Basically take the amount of usages you use of Prerender.io per month, and compare it yourself. What do you mean by more code to insert? I do know of Prerender.io but more code depends on which framework/language you're using. If it's node. It's pretty much one line. The extra code is just options that allow you to customise the way the interceptor works, these options include things like blacklist, whitelist, regex, extensions ignoring, client side caching.. etc. But the basic usage is the just key + email, and you're good to go. Also note that unlike Prerender, our cache storage is free, so we don't track how many pages you have.
  - gildas 12y ago
    Same question than the one asked some months ago [1]: > Why should someone use your service instead of existing ones? What is your added-value compared for example to seo4ajax.com? It loooks like the innovation is just to use Firefox instead of PhantomJS. Maybe I am wrong. [1] https://news.ycombinator.com/item?id=6781843 https://news.ycombinator.com/item?id=6781843
    - CMCDragonkai 12y ago
      Hey gildas, I answered a similar question here https://news.ycombinator.com/item?id=7765731 https://news.ycombinator.com/item?id=7765731
  - ilaksh 12y ago
    I see three lines of code whereas prerender.io is one line. I guess that's because they don't need you to configure the client with the API key or email. That's not really a huge difference, although it is still a little bit more effort. It really comes down to price I think. I can't really tell for sure what calculation to use to compare the pricing. Looks like you might be cheaper. You might consider doing a blog or something comparing the pricing though because I think other people know about prerender also and having a simple explanation that proves your pricing is better would make it easier for people to make that decision.
    - CMCDragonkai 12y ago
      Ah, well Prerender does use a single file, which means they haven't broken down their middleware into 3 components. All SnapSearch middleware is broken down (OOP style) into Detector, Client and Interceptor. I recognised, that this OOP strictness leads to a bit more verbosity. Therefore, for PHP I build a Stack interceptor. For Ruby, there's a Rack alternative. NodeJS has the expressConnector. Python doesn't have a single thing that brings them together atm. These integration points, make the entire thing a one liner to integrate. If you're referring to node, here's the connectInterceptor. Usage is on the README: https://github.com/SnapSearch/SnapSearch-Client-Node/blob/master/src/connectInterceptor.js https://github.com/SnapSearch/SnapSearch-Client-Node/blob/ma... Also sure, I'm going to get a blog up soon. Thanks for the advice. I'll do one on comparing pricing. But do note, that the fact that we use Firefox instead of QTWebkit means that we get 6 week release cycles so our scrapers keep up with the latest development HTML5 unlike the slower QTWebkit of PhantomJS. So we do have a technical advantage :)
  - sv123 12y ago
    Is there any difference in functionality or added benefit of using your service over prerender?
    - CMCDragonkai 12y ago
      Here's a shortlist of differences and benefits: 1. SnapSearch uses load balanced Firefox, this allows us to keep up with the latest in HTML5 unlike the QtWebkit of PhantomJS 2. SnapSearch does metered billing, this comes out cheaper for average sites, or sites that have lots of pages but dont send out tons of requests. We charge on usages, not cached requests, and not on cach storage. 3. SnapSearch's middleware is more robust, with 200 robots that it checks, and regularly updated robot list. There's a lot of flexibility in the code. 4. We provide you a bit more analytics in the control panel, so you can see things like request portion from multiple domains, requests/usages from search engines over a month. A progress bar on usages bar. A cache storage list of all the snapshots you have so you can precache pages if you wish. Try it out, it's free for 1000 usages per month! Also note that our API supports custom javascript callbacks, which allows you to do all sorts of interesting things prior to the caching of the snapshot. Furthermore, we also support soft 404s and redirections (including meta tag, header and js redirections (synchronoous and asynchronous). All of it is explained on the documentation. Flash support coming soon too.

andrenotgiant 12y ago

When you scrape the HTML from a single page app (let's say the default app homepage) do you re-write the javascript links to be HTML links with unique URLs? If not, how does Google find anything more than the app homepage?

albemuth 12y ago
Sitemap works
- CMCDragonkai 12y ago
  Yep a sitemap can also work. But it's not needed, since Google will follow your anchored links anyway. SnapSearch in the future can also create sitemaps dynamically.
CMCDragonkai 12y ago
Your links on your homepage, should already be "<a></a>" anchor links. When google hits your page, and it's served up by SnapSearch, then it will automatically find those anchor tags and navigate to the subpages. If you're talking about buttons, perhaps Google will click on those buttons. But any links should be represented by anchor links. This works for SnapSeach.io. Check on Google, search https://snapsearch.io https://snapsearch.io and all the subpages of SnapSearch are indexed. Also works for http://dreamitapp.com http://dreamitapp.com So to answer your question, there's no need to rewrite your links.

programminggeek 12y ago

Am I crazy for thinking that SEO is a huge reason to NOT do a Single Page App for a public facing website that you want Google to index? It seems like the wrong business choice from a marketing perspective that we are now looking for a tech solution to solve a problem we created ourselves. Single Page Apps feel like building Flash web apps 5 years ago. It's a valid choice, but with clear drawbacks.

CMCDragonkai 12y ago
Well SnapSearch allows you to make dynamic web applications without worrying about SEO problems. It's not so much a technical solution to a technical problem that we created ourselves, it's the fact that search engines and social network robots have not caught up to next generation websites (which I believe will be a fully dynamic user experience). We're just pushing the web ahead.
integraton 12y ago
You are not crazy. The single page app trend is solidly in the "peak of inflated expectations" phase of the hype cycle ( http://en.wikipedia.org/wiki/Hype_cycle http://en.wikipedia.org/wiki/Hype_cycle ), which is why there exists a subculture of developers who are trying to use the approach for everything, even where it isn't appropriate.
- CMCDragonkai 12y ago
  I think the single page app trend is coinciding with the growth of javascript, HTML5. It shows that the websites of the future will be more and more like web applications, and they need dynamic UI. The thing that's stopping us the SEO problem, which I hope is no longer problem anymore with snapsearch.
  - hippee-lee 12y ago
    Why not just render the public pages of your app out to HTML or use one of the phantom JS services to do it. Then when spiders come up the waterspout have your web server tell them where to go for your sites links. If you don't ant to do it yourself there are quite a few companies that can help: http://scotch.io/tutorials/javascript/angularjs-seo-with-prerender-io http://scotch.io/tutorials/javascript/angularjs-seo-with-pre...

fidlefodl 12y ago

I feel like applications and tooling to pre-render your HTML (eg: Reactjs) are more interesting. It's definitely more work currently (when done right), but it seems like the most natural flow. Always render html, let clientside link it up, everybody wins.

timdorr 12y ago

I think the logo could use a little work. The lightning bolt doesn't come off as an S, so it looks more like "napSearch".

CMCDragonkai 12y ago
Thanks for the feedback :). I'll look into it. Did you know the bolt is also a tiger stripe? That's why I got the tiger theme.

mattlenz 12y ago

Will Google eventually begin javascript-enabled scraping themselves?

CMCDragonkai 12y ago
They may one day. But it's a big computational overhead from straight curling to running a JS VM for everything. Right now SPAs are still a minority on the web. But I guess the question is will you wait for Google to support your site and lose out on the awesomeness of single page applications?
Encosia 12y ago
I don't know the specifics, but they already have been for years (at least to some extent). Several of my clients have found content deep inside public-facing client-side apps indexed in situations where we didn't care if it was indexed, but assumed it wouldn't/couldn't be.
- CMCDragonkai 12y ago
  Do you have any examples? Because I've been building single page apps for 2 years, and none of it is indexed. Also indexing for search is not the only thing, sometimes clients need google adverts, which require the google media bot check your site's content. One of my clients could not get any data out, and the site could not get approved by Google for the adverts. But with SnapSearch, the google media bot was able to extract the necessary information. Check your google webmaster tools, and you can force a scrape from google bots, and you can see that SPA sites are not discoverable.
  - Encosia 12y ago
    I can't share the specific examples, unfortunately. That's right that if you force a googlebot request from the webmaster tools, it's essentially just how the basic googlebot would wget a page on your site. We know they're doing more than that though, because it would be trivial to hide cloaking and malware from Google if that's the only tool they used for scraping. What Google actually does for crawling content seems to be a bit more nuanced. If you watch the server-side logs on a site that Google crawls often, you'll sometimes see a pattern of googlebot crawling a page and then an immediate subsequent request from a Google IP from a Chrome UA. I'm sure that's partially just to assess malware/cloaking, but it's probably also related to developments like this one: https://twitter.com/mattcutts/status/131425949597179904 https://twitter.com/mattcutts/status/131425949597179904 They're definitely doing more than just the ?_escaped_fragment_ rigamarole that almost no one uses. Fair point about the AdSense crawler though. It seems to be just about as dumb as a rock in my experience.

puppetmaster3 12y ago

Dear N00bs, SPA works great w/ SEO. Hashtags! (ex: https://github.com/puppetMaster3/ModuleBU/blob/master/latestCDN/lib/ModuleBU.js https://github.com/puppetMaster3/ModuleBU/blob/master/latest... in AppBu section, github has examples). Just because twitter or Facebook programers don't know how to do it, but then they do hire 0 experience out of school. Cheers.

puppetmaster3 12y ago
Down votes? It upsets you that I show you how to SEO w/ SPA? Says a bit about you I think.
- puppetmaster3 12y ago
  Dear downvoter: http://theunboundedspirit.com/wp-content/uploads/2014/05/truth-quote-nietzsche.jpg http://theunboundedspirit.com/wp-content/uploads/2014/05/tru...

ant_sz 12y ago

I think this should be more a problem for google than for web developer. web developer have the right to choose new technique they like, it's google's responsibility to scratch your content well. But now things are reversed, It's kind of monopoly, isn't it?

CMCDragonkai 12y ago
But it's not just google now. It's anybody who does any kind of crawling on the web, which includes social media robots (that acquire images from the link), personal web crawlers, private internal search engine, data mining, even advertisement bots that check content. Would they all have to embed and maintain/update a JS VM?
erichurkman 12y ago
Developers have always been free to choose techniques. Search engines owe you, the developer, nothing. If Google et al can't scrape your content, they'll scrape a competitor who better serves parse-able content. If you want to play ball with a particular search engine, you have to serve content that they can parse at the time.

nnq 12y ago

Am I crazy thinking that we as web developers should NOT HAVE TO have to care about this? It's Google's and Bing's and Yahoo's problem to go build decent spiders that run multiple versions of the most recent web browsers inside them, and they do have the computing power that's needed. All spiders/crawlers should see webpages just as human users see them that's the point! They should do whatever it takes to get this working, regardless the effort, because this is why they get paid (indirectly, though adds, but still...). And this is still not enough: it's any respectable search engine's duty to try and develop near-human-level AIs to be able to also extract meaningful information from visual structure, from pictures, and from hard to understand prose. And longer term, it will be their duty to develop above-human-level AIs to be even better at this than humans so they can provide even better search results. Since when are we, web developers, expected to do the search engines' work for them?! I expect this is just a transition phase and things will get on the right track soon. (Note: and no, I'm not against helping search engines. Providing semantic markup, microformats and all this is great. But there is a difference between helping them and doing the work for them, on our dime!) EDIT+: @OP: I'm not critical, what you and prerender.io are doing is awesome and immensely helpful for web developers! But you do realize that you're doing what search engines should be doing themselves, right? And when they will wake up and realize they should get into the game, they will have basically put you out of business... or buy your business, which I think is what you expect :) Anyway, good luck with it!

CMCDragonkai 12y ago
Hey I understand where you're coming from. I had the same impression before I began on SnapSearch. But right now SPAs are still a minority of the websites in the world, but eventually everybody will start making web applications. But do note, as I said in the other comments, it's not just search engines. There'll be a day when curlling sites will require a JS VM. There's a big difference between a standard curl vs a full blown chrome/firefox level VM. Considering the amount of sites google crawls per second, their computing costs would explode compared to what they're doing now. And this doesn't even address all the social network bots, and pretty much all the other bots doing stuff other than search indexing. So I think we're still got quite some time before we're redundant. At any case, the question for you is, are you going to wait for Google to implement this before you can start using single page application technologies which can give you an edge when it comes to customer user experience?

philbo 12y ago

If you're building JavaScript-heavy/single-page applications, you should be applying the principle of progressive enhancement anyway, there is no reason not to do so. Progressive enhancement brings a number of benefits, of which friendliness to search engines is just one. User has JS turned off? Page still works. Request to CDN fails? Page still works. Silly lint error slips through to production, breaking all your JS? Page still works. Weird cross-browser issue breaks your JS in some clients? Page still works. The most frequent arguments against progressive enhancement, which has been prevalent amongst a number of back-end and product people I've worked with, is that it is a waste of time and that it adds too much extra effort. I disagree with both. It is not always necessary to reach full feature parity between your JS and non-JS implementations, for instance an auto-suggest input in your JS code can happily be a vanilla input in the non-JS. But if you do just enough to ensure that your application is functional without JavaScript enabled, my (subjective, biased) experience has been that it doesn't take too long.

Kiro 12y ago
Easier said than done if you're using something like Angular or Ember.
- philbo 12y ago
  If angular or ember make it difficult to follow best practices, are they the best tool for the job? Less trollishly, although I have no experience with angular or ember, I have built a progressively enhanced application with knockout and I thought it went quite well. As with any progressively enhanced application, it requires that your back-end is set up to respond with both markup and JSON responses, but the data- attributes responsible for binding behaviour are just ignored by the browser when script is not running. Perhaps my experience is too simplistic, though. Can you give me an example of the kind of things that angular and ember make difficult to progressively enhance?
- Isofarro 12y ago
  Yehuda Katz, the lead on Ember.js was categorical about the use of ember.js when you require progressive enhancement: Don't use ember.js. So this is just a case of using the wrong tools for the job, buying the hype without validating the requirements.