4 ms·

Am I crazy thinking that we as web developers should NOT HAVE TO have to care about this? It's Google's and Bing's and Yahoo's problem to go build decent spider

by nnq 12y ago

Am I crazy thinking that we as web developers should NOT HAVE TO have to care about this? It's Google's and Bing's and Yahoo's problem to go build decent spiders that run multiple versions of the most recent web browsers inside them, and they do have the computing power that's needed. All spiders/crawlers should see webpages just as human users see them that's the point! They should do whatever it takes to get this working, regardless the effort, because this is why they get paid (indirectly, though adds, but still...). And this is still not enough: it's any respectable search engine's duty to try and develop near-human-level AIs to be able to also extract meaningful information from visual structure, from pictures, and from hard to understand prose. And longer term, it will be their duty to develop above-human-level AIs to be even better at this than humans so they can provide even better search results. Since when are we, web developers, expected to do the search engines' work for them?! I expect this is just a transition phase and things will get on the right track soon. (Note: and no, I'm not against helping search engines. Providing semantic markup, microformats and all this is great. But there is a difference between helping them and doing the work for them, on our dime!) EDIT+: @OP: I'm not critical, what you and prerender.io are doing is awesome and immensely helpful for web developers! But you do realize that you're doing what search engines should be doing themselves, right? And when they will wake up and realize they should get into the game, they will have basically put you out of business... or buy your business, which I think is what you expect :) Anyway, good luck with it!

CMCDragonkai 12y ago
Hey I understand where you're coming from. I had the same impression before I began on SnapSearch. But right now SPAs are still a minority of the websites in the world, but eventually everybody will start making web applications. But do note, as I said in the other comments, it's not just search engines. There'll be a day when curlling sites will require a JS VM. There's a big difference between a standard curl vs a full blown chrome/firefox level VM. Considering the amount of sites google crawls per second, their computing costs would explode compared to what they're doing now. And this doesn't even address all the social network bots, and pretty much all the other bots doing stuff other than search indexing. So I think we're still got quite some time before we're redundant. At any case, the question for you is, are you going to wait for Google to implement this before you can start using single page application technologies which can give you an edge when it comes to customer user experience?
- tracker1 12y ago
  Why would curl need a full VM? Just curl the API/Service endpoints for the data you want.. if you need more that's what PhantomJS is for.
SilkRoadie 12y ago
We as web developers need to create websites that make money. Part of that is to create websites which are visible and indexable by search engines.. In the UK people have been naming their businesses AAA whatever to appear first in the yellow pages for like 20 years. I don't see how this is much different from what we are doing now.
tracker1 12y ago
The search engines (google and bing) both run js content. Hell bing's IE browsers show up in google analytics. I worked on a site where we cleaned up our url structure with proper redirects etc... and bing went crazy with it, totally threw off our analytics (tons of unique clients, no retention, high drop off). I also know that google loads content in a headless browser as well. If you use push-state or shebang urls that load properly, and have proper output, google and bing get it fine. over 35k results for photogallery.classiccars.com in google. (I didn't design it, just implemented it a couple years ago) If you want an SPA interface, be prepared to do the work for SEO. If you can't handle that, then you should deal with it... I don't expect actual applications to be SEO friendly.
Isofarro 12y ago
> Since when are we, web developers, expected to do the search engines' work for them?! The Web is a trifecta of HTTP, URLs and hyperlinks. That is the fundamental baseline of what it is to be Web. That's what makes it accessible to anyone with a basic internet connection. With that universality comes an overall improvement in education, which improves the quality of life of people on this planet. Not just your little home-town, but the planet. The Web isn't an artificial intelligence. Not even in the dark days of RDF, respite with logical reasoning engines and ontologies. That isn't going to be your saviour. So far, there's nothing in the HTTP spec document that stipulates that it requires a JavaScript capable execution. The Web is built to be extensible, which is why techniques like progressive enhancement are just as fundamental. Assume the trifecta of HTTP, URLs and hyperlinks are present and build with that, then enhance with the litany of technologies and shiny toys built on top of the Web stack. Don't just assume those technical building blocks are always going to be there. So what you are doing, by requiring JavaScript at the HTTP level, isn't Web Development. It's probably more akin to the death-throes of the last Flash developers struggling to justify their existence. If you need your site to be indexable and searchable, universal, a JavaScript-dependent solution is sub-standard. These "post-implementation" Javascript static site scrapers, they emulate progressive enhancement badly, because they are backwards. They are about as qualitatively as useful as building a separate accessible website to your main website, you know for them blind people with their talking browsers. It's a band-aid at the point you realised you did not spend the requisite amount of time understanding the environment in which your "website" runs. And, please, read the full specification of the Google Crawlable Ajax spec before you throw it at me. I have. Including the part that says you really should have built it properly in the first place.

by nnq 12y ago

CMCDragonkai 12y ago

Hey I understand where you're coming from. I had the same impression before I began on SnapSearch. But right now SPAs are still a minority of the websites in the world, but eventually everybody will start making web applications. But do note, as I said in the other comments, it's not just search engines. There'll be a day when curlling sites will require a JS VM. There's a big difference between a standard curl vs a full blown chrome/firefox level VM. Considering the amount of sites google crawls per second, their computing costs would explode compared to what they're doing now. And this doesn't even address all the social network bots, and pretty much all the other bots doing stuff other than search indexing. So I think we're still got quite some time before we're redundant. At any case, the question for you is, are you going to wait for Google to implement this before you can start using single page application technologies which can give you an edge when it comes to customer user experience?

tracker1 12y ago
Why would curl need a full VM? Just curl the API/Service endpoints for the data you want.. if you need more that's what PhantomJS is for.

SilkRoadie 12y ago

We as web developers need to create websites that make money. Part of that is to create websites which are visible and indexable by search engines.. In the UK people have been naming their businesses AAA whatever to appear first in the yellow pages for like 20 years. I don't see how this is much different from what we are doing now.

tracker1 12y ago

The search engines (google and bing) both run js content. Hell bing's IE browsers show up in google analytics. I worked on a site where we cleaned up our url structure with proper redirects etc... and bing went crazy with it, totally threw off our analytics (tons of unique clients, no retention, high drop off). I also know that google loads content in a headless browser as well. If you use push-state or shebang urls that load properly, and have proper output, google and bing get it fine. over 35k results for photogallery.classiccars.com in google. (I didn't design it, just implemented it a couple years ago) If you want an SPA interface, be prepared to do the work for SEO. If you can't handle that, then you should deal with it... I don't expect actual applications to be SEO friendly.

Isofarro 12y ago

> Since when are we, web developers, expected to do the search engines' work for them?! The Web is a trifecta of HTTP, URLs and hyperlinks. That is the fundamental baseline of what it is to be Web. That's what makes it accessible to anyone with a basic internet connection. With that universality comes an overall improvement in education, which improves the quality of life of people on this planet. Not just your little home-town, but the planet. The Web isn't an artificial intelligence. Not even in the dark days of RDF, respite with logical reasoning engines and ontologies. That isn't going to be your saviour. So far, there's nothing in the HTTP spec document that stipulates that it requires a JavaScript capable execution. The Web is built to be extensible, which is why techniques like progressive enhancement are just as fundamental. Assume the trifecta of HTTP, URLs and hyperlinks are present and build with that, then enhance with the litany of technologies and shiny toys built on top of the Web stack. Don't just assume those technical building blocks are always going to be there. So what you are doing, by requiring JavaScript at the HTTP level, isn't Web Development. It's probably more akin to the death-throes of the last Flash developers struggling to justify their existence. If you need your site to be indexable and searchable, universal, a JavaScript-dependent solution is sub-standard. These "post-implementation" Javascript static site scrapers, they emulate progressive enhancement badly, because they are backwards. They are about as qualitatively as useful as building a separate accessible website to your main website, you know for them blind people with their talking browsers. It's a band-aid at the point you realised you did not spend the requisite amount of time understanding the environment in which your "website" runs. And, please, read the full specification of the Google Crawlable Ajax spec before you throw it at me. I have. Including the part that says you really should have built it properly in the first place.