6 ms·

Leaving JSPs in the dust: moving LinkedIn to dust.js client-side templates

134 points by eneveu 15y ago

kls 15y ago
I am surprised that many of the big sites are just now getting around to moving away from the server side frameworks for UI. The benefits of doing so are just so great that I am surprised that they ignored it for so long. Flattening the web stack makes development rapid again, UI engineers can stub out JSON files to emulate the REST back end and iterate through bugs so much faster. At many of the companies that I have implemented new style web front ends out TCO dropped significantly. I guess it's telling of the times when you are more shocked to find out that people are just now doing as opposed to the fact that they are doing it. As well I enjoyed the article, I have not used dust.js yet so it was interesting to see how it works. Dojo has a good templating model and I tend to prefer it over some of the other offerings, it is good to see that others are getting into the space. Though I do prefer Dojo's widget based templating over standard tiles type templateing, I find that it makes the code more compartmentalized and reusable. More of a black box, where code and template can just be added to a page and it works.
rhizome 15y ago
JSP, I never knew. That certainly explains a lot, particularly how it meshes with my sense of anything-JSP being a couple of years behind, featurewise. I don't know why this is, if it's a language non-feature, or a benefit of open-source libs, Java coding culture, or what, but I've thought that LinkedIn was making some silly site decisions and being inexplicably behind the times for some time now.
luser001 15y ago
I did something similar for a recent project. My backend basically only provides an HTTP API. All rendering is client-side. It felt natural as a programmer to develop the backend as a "library" that was "called" by the client side. Immmmmense productivity. All pages require auth and shouldn't be indexed, so I don't worry about excluding search engines etc.
- jrockway 15y ago
  Not sure why you're being downvoted, but I like this too. Your application should just be an API and pages should be tiny programs that call into this API. That way you can easily test your application and easily test your pages. And when you want to release "an API", well, you already designed, implemented, and tested it.
  - rjurney 15y ago
    Here here. Long live the one page web app.
    - greut 15y ago
      hear hear
      - rjurney 15y ago
        heer heer
reinhardt 15y ago
"All told, we evaluated 26 different templating technologies, scoring them on a variety of factors: DRY, JS library agnostic, mature, open source, easy to learn, documentation, flexibility, performance, and so on." If anyone from Linkedin is reading - any chance of getting this evaluation posted somewhere? Would be useful even if it's just a "tech x factor" matrix without commentary.
- vybs 15y ago
  Sure, we should be able to make it available on the blog site.
  - toonketels 15y ago
    That would be great. I also would love to see the see the matrix. Any change you are going to discuss in more detail how to use the js templates on client and server side?
    - vybs 15y ago
      Sure, we'd like to soon write more about this. Node.js and jvm based rhino are the two solutions we are trying out internally. We are building some of our seo use cases with dust templates and server-side rendering. It is quite easy and efficient to reuse the same template for both non seo and seo cases
prodigal_erik 15y ago
> Once written, the same dust.js template can be rendered not only in the browser, but also on the server using node.js or Rhino. I'm relieved to see their staff know what they're doing, and even curl http://www.linkedin.com/in/example http://www.linkedin.com/in/example still yields content. An alarming fraction of devs and toolsets out there would have screwed this up so badly that their resources would be cut off from reuse by the rest of the web, trapped behind an unstable single-site API.
- johnx123-up 15y ago
  Can you please elaborate on this?
  - simonw 15y ago
    I imagine the parent means that building single-page apps that require a full DOM+JS implementation is bad for the Web, because it cuts content off from being accessed by anything other than a full modern browser running several million lines of C++. See also my answer to this question about single-page apps on Quora: http://www.quora.com/What-exactly-is-a-single-page-application-and-is-this-the-technology-strategy-I-should-get-my-web-application-developed-in/answer/Simon-Willison http://www.quora.com/What-exactly-is-a-single-page-applicati...
    - johnx123-up 15y ago
      IIRC, you're Node.js evangelist? What's your take on Node.js (server) + Backbone.js (or dust.js as mentioned here)? Not for all, but for Trello like client apps?
    - prodigal_erik 15y ago
      An enormous pig of a browser would be bad enough, but you also have to run the client js served by site X, because nobody else has code that's always going to be compatible with today's revision of the site X API. And if you want any behavior that isn't baked into the site X client js, you're just boned. That's why I liken this trend to the client/server days, when you were stuck with one terrible client you couldn't fix—which is what I thought the web had saved us from.
sandGorgon 15y ago
Doesnt this play havoc with caching - I mean if you had a single UI webapp (that in turn queries all other webservices to render a page), then you could simply heavily cache your UI webapp. But when your client side JS is directly querying individual web services, in that case you would have to cache each individual web service (not sure if this introduces inconsistencies vis-a-vis different web services). Is this how Amazon - who reputedly has everything as services - renders its webpages as well ?
- bigiain 15y ago
  Not only caching, but right now I suspect this approach could have _serious_ SEO implications. It seems to run directly against the guidelines against cloaking here: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=66355 http://support.google.com/webmasters/bin/answer.py?hl=en&#38... (Not that I suppose LinkedIn are too worried about search engine discoverability of a lot of their "dynamic, login required" content, but I'll bet they're _not_ using this technique on all those highly optimised pages full of people's names - like the 4th Google hit for my name: http://au.linkedin.com/in/iaindchalmers http://au.linkedin.com/in/iaindchalmers )
  - keltex 15y ago
    It wouldn't have SEO implications because using javascript to load content isn't considered cloaking. (Cloaking is where you present something different to googlebot vs. a live user). It's well known that googlebot does run javascript now: https://twitter.com/#!/mattcutts/status/131425949597179904 https://twitter.com/#!/mattcutts/status/131425949597179904 There's also some speculation that googlebot actually runs Chrome and can render anything that Chrome does: http://ipullrank.com/googlebot-is-chrome/ http://ipullrank.com/googlebot-is-chrome/
    - bigiain 15y ago
      Yep. I've seen that (and I've seen evidence googlebot executing javascript in my own logfiles). BUT - If I have a client relying on SEO/organic ranking for their money right now, there's no way I'd be suggesting they move to a "html template with javascript loaded content" style site - not until I've seen a few other similar/competitor sites ranking well with those techniques.
akavlie 15y ago
JSP is an archaic template language. I'm on a project with JSP on the front end; compared to Django, Jinja, or any of the other Python template languages it's really limited and verbose. It's interesting that LinkedIn chose dust.js as it doesn't get much attention compared to some of the other JS template languages (e.g. Mustache & Handlebars.js).
- strlen 15y ago
  Do keep in mind that LinkedIn was founded in 2003, before many of the web frameworks were released. LinkedIn also uses a custom JSP compiler, as at the time the official implementation lacked many feature.
  - akavlie 15y ago
    That wasn't meant as a criticism of LinkedIn's adoption of it, more my own circumstances and the fact that frameworks are still using JSP by default in 2011.
    - strlen 15y ago
      Definitely wouldn't use it. Even for server-side Java, there are better templating engine. I am not a web developer, so I am not fully qualified to speak, but in cases where I've needed templating, I've found Apache Velocity and Freemarket to be good options for Java (but most of the time, I am templating, e.g., code to be auto generated or configuration files, rather than web pages).
- eneveu 15y ago
  I wonder what's the best alternative for server-side template generation on the JVM. I used FreeMarker on a previous project, it's pretty cool ( http://freemarker.sourceforge.net/ http://freemarker.sourceforge.net/ ). There is also mustache.java ( https://github.com/spullara/mustache.java https://github.com/spullara/mustache.java ). I heard about Velocity and StringTemplate, but never tried them out... I guess Groovy Server Pages could also be considered if one's company is open to Grails... Anything else I should be aware of?
sonnym 15y ago
It should be noted that dustjs is not yet updated for node v0.6.x. I submitted a rather trivial pull request toward the end of October fixing the problem and another user has submitted one since then, but the author has not responded to either. This makes me question how much the project will be supported in the future - but I suppose having a large company like LinkedIn invested in the technology is a good sign. See: https://github.com/akdubya/dustjs/pull/36 https://github.com/akdubya/dustjs/pull/36
jsnewbie 15y ago
I apologize for the newbie question, but is dust.js a lightweight replacement for backbone.js?
bigiain 15y ago
Curiously, we tried this and backed away to using AJAX calls returning pre-rendered html fragments in a recent project. If you're building a mobile site, and want to target low-end Android devices (our "crappiest-phone-test-device" was a Huawei U8180), you really don't get a lot in the way of JSON parsing or DOM manipulation performance. I wonder what (if anything) LinkedIn are doing for mobile?
- vybs 15y ago
  http://engineering.linkedin.com/nodejs/blazing-fast-nodejs-10-performance-tips-linkedin-mobile http://engineering.linkedin.com/nodejs/blazing-fast-nodejs-1...
  - bigiain 15y ago
    Nice link, thanks. It doesn't address the problem we bumped into though. When a page needs a significant amount of content updated, sending that as structured JSON to a low powered Android device (or sometimes even an iPhone2 or a non S 3G) and then parsing that JSON and updatng the DOM was _way_ slower than sending back HTML and just replacing an elements innerHtml. For small tasks (like the personalisation {'name':'John'} example) we could make the JSON approach work, but updating a ~25row 4column list? Not on the cheap Android devices...
    - nevinera 15y ago
      What do you mean by 'significant amount of content'? Inserting html is of course faster than rendering in the client, but the javascript involved here is not slow - I haven't seen anything over 15 ms yet for rendering a fairly complicated page (I'm using a resig-style microtemplate, which is probably faster). Are you perhaps constructing the nodes on the document, instead of doing them off the tree and then inserting/replacing them all at once?
tunggad 15y ago
Does any know how is dust.js compared to Beebole PURE, an JS template engine too. They claim, PURE should be fastest JS template. ^^
- bvincent 15y ago
  As vybs mentioned we'll publish more detail on our evaluation process, for which performance was one of many features we looked at. It may be useful to start by asking what's most important for you/your organization (performance, productivity, ramp-up time, server-side rendering support, etc) since these priorities will likely be different from ours.
mikehal 15y ago
What about SEO and accessibility. If the content is sent via JSON it's relying on javascript, no?
- bvincent 15y ago
  Correct, the template can only be rendered on the "client" in cases where JavaScript is available. In other cases (including for the small percentage of users who have turned off JavaScript and mobile devices without JS support) the solution is to render the templates (turn them into HTML) on the server.
  - cuginoAle 15y ago
    bvincent, could you please elaborate on this? How do you tell javascript enabled users apart from disabled ones? Is there any website discussing this you can point me to? thanks
Tautologistics 15y ago
Unless I am missing something, the bullet point near the end, "Progressive rendering", is not correct when it states that template rendering is asynchronous and parallel. Fetching of data, sure, but unless they are using web workers to execute the templates client side, and I do not see workers mentioned anywhere, this is not accurate. When a template is rendering, all other JS and UI will stop. Having applied the same approach as LinkedIn on past projects and jobs (e.g. BN.com), there is definitely a lot to be gained if done right, and web workers, when available, ease some of the issues with rendering large templates and/or large datasets.

134 points by eneveu 15y ago

kls 15y ago

I am surprised that many of the big sites are just now getting around to moving away from the server side frameworks for UI. The benefits of doing so are just so great that I am surprised that they ignored it for so long. Flattening the web stack makes development rapid again, UI engineers can stub out JSON files to emulate the REST back end and iterate through bugs so much faster. At many of the companies that I have implemented new style web front ends out TCO dropped significantly. I guess it's telling of the times when you are more shocked to find out that people are just now doing as opposed to the fact that they are doing it. As well I enjoyed the article, I have not used dust.js yet so it was interesting to see how it works. Dojo has a good templating model and I tend to prefer it over some of the other offerings, it is good to see that others are getting into the space. Though I do prefer Dojo's widget based templating over standard tiles type templateing, I find that it makes the code more compartmentalized and reusable. More of a black box, where code and template can just be added to a page and it works.

rhizome 15y ago

JSP, I never knew. That certainly explains a lot, particularly how it meshes with my sense of anything-JSP being a couple of years behind, featurewise. I don't know why this is, if it's a language non-feature, or a benefit of open-source libs, Java coding culture, or what, but I've thought that LinkedIn was making some silly site decisions and being inexplicably behind the times for some time now.

luser001 15y ago

I did something similar for a recent project. My backend basically only provides an HTTP API. All rendering is client-side. It felt natural as a programmer to develop the backend as a "library" that was "called" by the client side. Immmmmense productivity. All pages require auth and shouldn't be indexed, so I don't worry about excluding search engines etc.

jrockway 15y ago
Not sure why you're being downvoted, but I like this too. Your application should just be an API and pages should be tiny programs that call into this API. That way you can easily test your application and easily test your pages. And when you want to release "an API", well, you already designed, implemented, and tested it.
- rjurney 15y ago
  Here here. Long live the one page web app.
  - greut 15y ago
    hear hear
    - rjurney 15y ago
      heer heer

reinhardt 15y ago

"All told, we evaluated 26 different templating technologies, scoring them on a variety of factors: DRY, JS library agnostic, mature, open source, easy to learn, documentation, flexibility, performance, and so on." If anyone from Linkedin is reading - any chance of getting this evaluation posted somewhere? Would be useful even if it's just a "tech x factor" matrix without commentary.

vybs 15y ago
Sure, we should be able to make it available on the blog site.
- toonketels 15y ago
  That would be great. I also would love to see the see the matrix. Any change you are going to discuss in more detail how to use the js templates on client and server side?
  - vybs 15y ago
    Sure, we'd like to soon write more about this. Node.js and jvm based rhino are the two solutions we are trying out internally. We are building some of our seo use cases with dust templates and server-side rendering. It is quite easy and efficient to reuse the same template for both non seo and seo cases

prodigal_erik 15y ago

> Once written, the same dust.js template can be rendered not only in the browser, but also on the server using node.js or Rhino. I'm relieved to see their staff know what they're doing, and even curl http://www.linkedin.com/in/example http://www.linkedin.com/in/example still yields content. An alarming fraction of devs and toolsets out there would have screwed this up so badly that their resources would be cut off from reuse by the rest of the web, trapped behind an unstable single-site API.

johnx123-up 15y ago
Can you please elaborate on this?
- simonw 15y ago
  I imagine the parent means that building single-page apps that require a full DOM+JS implementation is bad for the Web, because it cuts content off from being accessed by anything other than a full modern browser running several million lines of C++. See also my answer to this question about single-page apps on Quora: http://www.quora.com/What-exactly-is-a-single-page-application-and-is-this-the-technology-strategy-I-should-get-my-web-application-developed-in/answer/Simon-Willison http://www.quora.com/What-exactly-is-a-single-page-applicati...
  - johnx123-up 15y ago
    IIRC, you're Node.js evangelist? What's your take on Node.js (server) + Backbone.js (or dust.js as mentioned here)? Not for all, but for Trello like client apps?
  - prodigal_erik 15y ago
    An enormous pig of a browser would be bad enough, but you also have to run the client js served by site X, because nobody else has code that's always going to be compatible with today's revision of the site X API. And if you want any behavior that isn't baked into the site X client js, you're just boned. That's why I liken this trend to the client/server days, when you were stuck with one terrible client you couldn't fix—which is what I thought the web had saved us from.

sandGorgon 15y ago

Doesnt this play havoc with caching - I mean if you had a single UI webapp (that in turn queries all other webservices to render a page), then you could simply heavily cache your UI webapp. But when your client side JS is directly querying individual web services, in that case you would have to cache each individual web service (not sure if this introduces inconsistencies vis-a-vis different web services). Is this how Amazon - who reputedly has everything as services - renders its webpages as well ?

bigiain 15y ago
Not only caching, but right now I suspect this approach could have _serious_ SEO implications. It seems to run directly against the guidelines against cloaking here: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=66355 http://support.google.com/webmasters/bin/answer.py?hl=en&#38... (Not that I suppose LinkedIn are too worried about search engine discoverability of a lot of their "dynamic, login required" content, but I'll bet they're _not_ using this technique on all those highly optimised pages full of people's names - like the 4th Google hit for my name: http://au.linkedin.com/in/iaindchalmers http://au.linkedin.com/in/iaindchalmers )
- keltex 15y ago
  It wouldn't have SEO implications because using javascript to load content isn't considered cloaking. (Cloaking is where you present something different to googlebot vs. a live user). It's well known that googlebot does run javascript now: https://twitter.com/#!/mattcutts/status/131425949597179904 https://twitter.com/#!/mattcutts/status/131425949597179904 There's also some speculation that googlebot actually runs Chrome and can render anything that Chrome does: http://ipullrank.com/googlebot-is-chrome/ http://ipullrank.com/googlebot-is-chrome/
  - bigiain 15y ago
    Yep. I've seen that (and I've seen evidence googlebot executing javascript in my own logfiles). BUT - If I have a client relying on SEO/organic ranking for their money right now, there's no way I'd be suggesting they move to a "html template with javascript loaded content" style site - not until I've seen a few other similar/competitor sites ranking well with those techniques.

akavlie 15y ago

JSP is an archaic template language. I'm on a project with JSP on the front end; compared to Django, Jinja, or any of the other Python template languages it's really limited and verbose. It's interesting that LinkedIn chose dust.js as it doesn't get much attention compared to some of the other JS template languages (e.g. Mustache & Handlebars.js).

strlen 15y ago
Do keep in mind that LinkedIn was founded in 2003, before many of the web frameworks were released. LinkedIn also uses a custom JSP compiler, as at the time the official implementation lacked many feature.
- akavlie 15y ago
  That wasn't meant as a criticism of LinkedIn's adoption of it, more my own circumstances and the fact that frameworks are still using JSP by default in 2011.
  - strlen 15y ago
    Definitely wouldn't use it. Even for server-side Java, there are better templating engine. I am not a web developer, so I am not fully qualified to speak, but in cases where I've needed templating, I've found Apache Velocity and Freemarket to be good options for Java (but most of the time, I am templating, e.g., code to be auto generated or configuration files, rather than web pages).
eneveu 15y ago
I wonder what's the best alternative for server-side template generation on the JVM. I used FreeMarker on a previous project, it's pretty cool ( http://freemarker.sourceforge.net/ http://freemarker.sourceforge.net/ ). There is also mustache.java ( https://github.com/spullara/mustache.java https://github.com/spullara/mustache.java ). I heard about Velocity and StringTemplate, but never tried them out... I guess Groovy Server Pages could also be considered if one's company is open to Grails... Anything else I should be aware of?

sonnym 15y ago

It should be noted that dustjs is not yet updated for node v0.6.x. I submitted a rather trivial pull request toward the end of October fixing the problem and another user has submitted one since then, but the author has not responded to either. This makes me question how much the project will be supported in the future - but I suppose having a large company like LinkedIn invested in the technology is a good sign. See: https://github.com/akdubya/dustjs/pull/36 https://github.com/akdubya/dustjs/pull/36

jsnewbie 15y ago

I apologize for the newbie question, but is dust.js a lightweight replacement for backbone.js?

bigiain 15y ago

Curiously, we tried this and backed away to using AJAX calls returning pre-rendered html fragments in a recent project. If you're building a mobile site, and want to target low-end Android devices (our "crappiest-phone-test-device" was a Huawei U8180), you really don't get a lot in the way of JSON parsing or DOM manipulation performance. I wonder what (if anything) LinkedIn are doing for mobile?

vybs 15y ago
http://engineering.linkedin.com/nodejs/blazing-fast-nodejs-10-performance-tips-linkedin-mobile http://engineering.linkedin.com/nodejs/blazing-fast-nodejs-1...
- bigiain 15y ago
  Nice link, thanks. It doesn't address the problem we bumped into though. When a page needs a significant amount of content updated, sending that as structured JSON to a low powered Android device (or sometimes even an iPhone2 or a non S 3G) and then parsing that JSON and updatng the DOM was _way_ slower than sending back HTML and just replacing an elements innerHtml. For small tasks (like the personalisation {'name':'John'} example) we could make the JSON approach work, but updating a ~25row 4column list? Not on the cheap Android devices...
  - nevinera 15y ago
    What do you mean by 'significant amount of content'? Inserting html is of course faster than rendering in the client, but the javascript involved here is not slow - I haven't seen anything over 15 ms yet for rendering a fairly complicated page (I'm using a resig-style microtemplate, which is probably faster). Are you perhaps constructing the nodes on the document, instead of doing them off the tree and then inserting/replacing them all at once?

tunggad 15y ago

Does any know how is dust.js compared to Beebole PURE, an JS template engine too. They claim, PURE should be fastest JS template. ^^

bvincent 15y ago
As vybs mentioned we'll publish more detail on our evaluation process, for which performance was one of many features we looked at. It may be useful to start by asking what's most important for you/your organization (performance, productivity, ramp-up time, server-side rendering support, etc) since these priorities will likely be different from ours.

mikehal 15y ago

What about SEO and accessibility. If the content is sent via JSON it's relying on javascript, no?

bvincent 15y ago
Correct, the template can only be rendered on the "client" in cases where JavaScript is available. In other cases (including for the small percentage of users who have turned off JavaScript and mobile devices without JS support) the solution is to render the templates (turn them into HTML) on the server.
- cuginoAle 15y ago
  bvincent, could you please elaborate on this? How do you tell javascript enabled users apart from disabled ones? Is there any website discussing this you can point me to? thanks

Tautologistics 15y ago

Unless I am missing something, the bullet point near the end, "Progressive rendering", is not correct when it states that template rendering is asynchronous and parallel. Fetching of data, sure, but unless they are using web workers to execute the templates client side, and I do not see workers mentioned anywhere, this is not accurate. When a template is rendering, all other JS and UI will stop. Having applied the same approach as LinkedIn on past projects and jobs (e.g. BN.com), there is definitely a lot to be gained if done right, and web workers, when available, ease some of the issues with rendering large templates and/or large datasets.