6 ms·

What it's like building a real website in Node.js

158 points by travisglines 16y ago

peregrine 16y ago
This is a great article but again, like the last article posted here about your MVC setup, I'd personally like to see examples. For instance whats an example of good asynchronous code?
- travisglines 16y ago
  Hey peregrine, I'm working on a solid example here: https://github.com/tglines/throughexample https://github.com/tglines/throughexample I'm taking a bit of time with it to make sure its actually useful to people and not just a stub. Feel free to check out what I have up there so far.
  - peregrine 16y ago
    Thanks! Just what I was looking for!
  - TY 16y ago
    Thank you for a good article, Travis. Quick side note - you might want to add simple validation for sign up form for exipe.com - it currently accepts empty string as a valid value.
  - Charuru 16y ago
    This is cool! But missing secret_facebook_stuff. Do you think you can use http digest rather than facebook for auth?
- steilpass 16y ago
  If your using Express you might want to look at: https://github.com/1602/express-on-railway https://github.com/1602/express-on-railway
bitdrift 16y ago
Agreed. I spent the last few months building my first Node.js app (and MongoDB) and I resonate with a lot of what's being said here. I wouldn't want to discourage anyone from exploring Node or Mongo, because I think both are amazing projects and have huge potential, but I do think anyone considering either should stop and think carefully before taking the plunge. Things that stick out in my head: 1. What the author says about the Node community moving fast is absolutely true. Be prepared to spend time upgrading to the latest versions often or deal with dependency hell as you try to ensure you've got matching legacy versions of libraries. 2. Coding async is weird at first, but you may grow to love it (as I have). Over the months, I've discovered many cool tricks and patterns that help keep the nesting to a minimum. 3. Be prepared to fill library gaps, and submit patches if necessary. This means not being afraid to dig into the source code of the libraries you're using. Most libraries have limited documentation so often the only way to figure out how to use them is to get in there and see how they work. 4. Best practices may not apply. In the case of MongoDB, I had to redesign my schema in order to run certain queries now that MongoDB won't support until sometime in the future. All in all, I'm really glad for the experience and will definitely consider both Node.js and MongoDB for future projects. What will really be interesting is seeing what I miss about Node/Mongo after jumping back into Python/MySQL. :-)
- nailer 16y ago
  > 2. Coding async is weird at first, but you may grow to love it (as I have). Over the months, I've discovered many cool tricks and patterns that help keep the nesting to a minimum. Can you share a few? Either here or as a blog post?
  - mjs 16y ago
    If you have a function foo() that takes a callback as an argument i.e. something like: foo(..., function (n) { // closure over q, r, s }); Then what you can do is create a separate function, bar() that takes q, r, s as an argument, and returns a function: function bar(q, r, s) { return function(n) { // still has dependency on q, r, s } } Once you have bar(), the foo() call above can be written as: foo(..., bar(q, r, s)); and you save one level of nesting. (This also makes the dependencies explicit, which is helpful.) The main loop of https://github.com/ithinkihaveacat/node-fishback/blob/ce5f9c308db6a12f15fb0d790c53dfb2f60eaf9c/lib/fishback.js#L313 https://github.com/ithinkihaveacat/node-fishback/blob/ce5f9c... does quite a lot of this, if you want a real example.
alexbosworth 16y ago
I have been running Node.js in production for a while, it is kind of a wild-west, Apache/PHP holds your hand for you and takes care of a bunch of things you might not realize. If you come from a background of already writing javascript heavy apps, the browser is async so there shouldn't be as steep a learning curve. Node is moving super fast, with some critical breaks between point releases. I don't find the community crazy good, but Github is saving the day on node for me - being able to quickly browse through forks is a lifesaver
- gruseom 16y ago
  If you come from a background of already writing javascript heavy apps, the browser is async so there shouldn't be as steep a learning curve. People say this, but it's not really true. Yes, browsers are async in the sense that your code gets invoked by the browser as events occur. But once your client code gets control, it nearly always proceeds in a completely synchronous fashion. If all you're doing is changing a few dom elements in response to a button click, that happens quickly and all is well, but woe betide if you need to do some long-running computation in the browser... what are you going to do? Build your own event loop with setInterval and chop things up into short-running pieces? That's certainly uncommon. So client-side JS programmers typically don't have much experience with organizing their own code asynchronously, and the learning curve for doing so is still steep. (There is of course one big exception to what I'm saying, namely communicating with the server via XHR. But not so big that it teaches you how to write a complex program in a non-blocking asynchronous style.)
  - robin_reala 16y ago
    Web workers are another exception, but I guess they’re not mainstream enough for most developers to have bumped into them.
  - arethuza 16y ago
    Access to a client-side SQL database from Javascript in Safari/Chrome is also asynchronous - which takes a bit of getting used to.
  - 16y ago
i386 16y ago
I was hoping to know more about what corner cases he hit where he had to write subjectively ugly code and what part of the ecosystem hasn't matured yet (libraries for certain problems missing, etc). For me, building real applications and solving real problems in any new stack comes down to those two things not being much of an issue. I remember a friend and I hacking on a RoR app a few years ago where we spent more time trying to get Ruby Gems to play nice rather than actually writing the app. Now, those problems have been mostly shaken out and are a thing of the distant past. Is Node past this productivity immaturity hump yet?
penguinboy08 16y ago
Great article. Writing my node.js project has exposed me to all of the things listed. One thing I have to say is that NPM is good, but good be better with it's distro integration. The ubuntu install has broken permissions by default, which forces me to install stuff with sudo. I haven't had the time to try and fix it.
- chapel 16y ago
  NPM >0.3.0 changed to requiring sudo while installing. The reason being that it simplifies things logistically as well it allows NPM to install modules as nobody, since you have to be sudo to do that.
- justincormack 16y ago
  I dont think its worth using the Ubuntu install its just too old. At the moment easiest to install by hand, and script something for production.
  - nailer 16y ago
    His issue (though not too obvious) is that NPM assumes /usr/local is writable by unprivileged users, which isn't secure or common in Linux distros or OS X. It's a bug in NPM, it should be fixed someday.
dbro 16y ago
This would have been helpful to me when I was starting out building http://domainjig.com http://domainjig.com - which is a simpler site than the author's. That said, I'd offer these responses to those looking for some advice: 1) Yes, wrapping every database call with an asynchronous callback is a drag. I found a helpful library that made some of my workflow easier to express and maintain: https://github.com/caolan/async https://github.com/caolan/async And I'm pretty sure I could simplify my code quite a bit more if I understood functional program patterns better already. 2) Libraries can be challenging, but I was able to find good ones for everything I needed. The author discussed version issues. To deal with that, I punted on all those node library managers and just jumped into the library code, checking it in to my repository and freezing it in time (in contrast to the authors point #9 about NPM being awesome for him). The other issue I found with libraries was that there are TOO MANY of them for any single thing. For example, there are at least a half dozen separate libraries to help with asynchronous workflow, and trying to choose which one would be "best" was difficult, and depended on freshness, documentation, and then actually test driving them. Github's encouragement of forking made this extra challenging, with many slightly tweaked variations for each separate library. I found node.js lived up to its promises (for me): coding in javascript on both server and client was helpful in avoiding context switching inside my head, and encouraged me to invest more effort in writing better quality code everywhere. And node.js is plenty fast. Another thing I'd add is that I used redis and tokyo cabinet as data stores and found the node.js interfaces to them to be mature and easy enough. It was convenient to use python's interactive shell to create and experiment with redis, and then port the routines to node.js. To summarize, consider this a positive vote for node.js.
augustl 16y ago
I just wrote TLS client certificate authentication for an internal systems API with node, in about 20 lines of code. Node's biggest power is a great abstraction for low level HTTP and socket stuff. You get direct access to everything, with a sane and elegant API. For plain old websites, I'd still use something like Rails or Django.
yannickmahe 16y ago
Great article. In your opinion, is Node.js mature enough to use in production code ? Or should I wait a little longer ? Also, what is the sate of the community (i.e. is it easy to find answers and tutorials) ?
Kilimanjaro 16y ago
I wish we had s-node (s for synchronous) so we do: session.start() data.load() template.render() instead of the cretan labyrinth it becomes coding for a-node
- randylahey 16y ago
  That completely defeats the point of using Node.js; that is, to use an evented model so your app isn't sitting around waiting for IO. s-node wouldn't be node at all. FWIW any other runtime/framework will do what you suggest. For example, Rhino.
  - joeshaw 16y ago
    It would be nice of Node (or rather, v8 underneath it) supported generators and the "yield" keyword like Mozilla's Spidermonkey. Unfortunately, yield is a Javascript 1.7 extension for now, but there's hope for it in ES Harmony. Anyway, with generators and promises you can write asynchronous code which looks like synchronous code but which under the covers is still completely async and callback-driven. We use it in gjs (a GNOMEy JS runtime built on top of Spidermonkey). Instead of workflows like: function onServerRequest(req, res) { db.query("get filename", function(result) { fs.readContents(result, function (data) { res.write(data); }); }); } which can get pretty nasty if you're running things in serial, or want to run a couple of things in parallel but then run something serially, you can have a workflow more like this: function onServerRequest(req, res) { var queryPromise = db.queryPromise("get filename"); var filename = yield queryPromise; var data = yield db.readContentsPromise(filename); res.write(data); } The main difference between the two is that the async functions in the latter return promises instead of nothing. The db.queryPromise() function itself would probably look something like: function queryPromise(query) { var promise = new Promise(); db.query(query, function(result) { promise.putValue(result); }); return promise; } I imagine you could probably write a function to automatically generate promise wrappers for existing async functions as long as they followed a certain schema, and Node's functions tend to do well with that. Anyway, the onServerRequest() function would itself be wrapped in a function which took the generator values from it and returned its own promise. This way you could stack promises all the way up. The gjs implementation of this is here: http://git.gnome.org/browse/gjs/tree/modules/promise.js http://git.gnome.org/browse/gjs/tree/modules/promise.js This is pretty similar to Twisted's inlineCallbacks if you're familiar with them in Python.

158 points by travisglines 16y ago

This is a great article but again, like the last article posted here about your MVC setup, I'd personally like to see examples. For instance whats an example of good asynchronous code?

travisglines 16y ago
Hey peregrine, I'm working on a solid example here: https://github.com/tglines/throughexample https://github.com/tglines/throughexample I'm taking a bit of time with it to make sure its actually useful to people and not just a stub. Feel free to check out what I have up there so far.
- peregrine 16y ago
  Thanks! Just what I was looking for!
- TY 16y ago
  Thank you for a good article, Travis. Quick side note - you might want to add simple validation for sign up form for exipe.com - it currently accepts empty string as a valid value.
- Charuru 16y ago
  This is cool! But missing secret_facebook_stuff. Do you think you can use http digest rather than facebook for auth?
steilpass 16y ago
If your using Express you might want to look at: https://github.com/1602/express-on-railway https://github.com/1602/express-on-railway

bitdrift 16y ago

Agreed. I spent the last few months building my first Node.js app (and MongoDB) and I resonate with a lot of what's being said here. I wouldn't want to discourage anyone from exploring Node or Mongo, because I think both are amazing projects and have huge potential, but I do think anyone considering either should stop and think carefully before taking the plunge. Things that stick out in my head: 1. What the author says about the Node community moving fast is absolutely true. Be prepared to spend time upgrading to the latest versions often or deal with dependency hell as you try to ensure you've got matching legacy versions of libraries. 2. Coding async is weird at first, but you may grow to love it (as I have). Over the months, I've discovered many cool tricks and patterns that help keep the nesting to a minimum. 3. Be prepared to fill library gaps, and submit patches if necessary. This means not being afraid to dig into the source code of the libraries you're using. Most libraries have limited documentation so often the only way to figure out how to use them is to get in there and see how they work. 4. Best practices may not apply. In the case of MongoDB, I had to redesign my schema in order to run certain queries now that MongoDB won't support until sometime in the future. All in all, I'm really glad for the experience and will definitely consider both Node.js and MongoDB for future projects. What will really be interesting is seeing what I miss about Node/Mongo after jumping back into Python/MySQL. :-)

nailer 16y ago
> 2. Coding async is weird at first, but you may grow to love it (as I have). Over the months, I've discovered many cool tricks and patterns that help keep the nesting to a minimum. Can you share a few? Either here or as a blog post?
- mjs 16y ago
  If you have a function foo() that takes a callback as an argument i.e. something like: foo(..., function (n) { // closure over q, r, s }); Then what you can do is create a separate function, bar() that takes q, r, s as an argument, and returns a function: function bar(q, r, s) { return function(n) { // still has dependency on q, r, s } } Once you have bar(), the foo() call above can be written as: foo(..., bar(q, r, s)); and you save one level of nesting. (This also makes the dependencies explicit, which is helpful.) The main loop of https://github.com/ithinkihaveacat/node-fishback/blob/ce5f9c308db6a12f15fb0d790c53dfb2f60eaf9c/lib/fishback.js#L313 https://github.com/ithinkihaveacat/node-fishback/blob/ce5f9c... does quite a lot of this, if you want a real example.

alexbosworth 16y ago

I have been running Node.js in production for a while, it is kind of a wild-west, Apache/PHP holds your hand for you and takes care of a bunch of things you might not realize. If you come from a background of already writing javascript heavy apps, the browser is async so there shouldn't be as steep a learning curve. Node is moving super fast, with some critical breaks between point releases. I don't find the community crazy good, but Github is saving the day on node for me - being able to quickly browse through forks is a lifesaver

gruseom 16y ago
If you come from a background of already writing javascript heavy apps, the browser is async so there shouldn't be as steep a learning curve. People say this, but it's not really true. Yes, browsers are async in the sense that your code gets invoked by the browser as events occur. But once your client code gets control, it nearly always proceeds in a completely synchronous fashion. If all you're doing is changing a few dom elements in response to a button click, that happens quickly and all is well, but woe betide if you need to do some long-running computation in the browser... what are you going to do? Build your own event loop with setInterval and chop things up into short-running pieces? That's certainly uncommon. So client-side JS programmers typically don't have much experience with organizing their own code asynchronously, and the learning curve for doing so is still steep. (There is of course one big exception to what I'm saying, namely communicating with the server via XHR. But not so big that it teaches you how to write a complex program in a non-blocking asynchronous style.)
- robin_reala 16y ago
  Web workers are another exception, but I guess they’re not mainstream enough for most developers to have bumped into them.
- arethuza 16y ago
  Access to a client-side SQL database from Javascript in Safari/Chrome is also asynchronous - which takes a bit of getting used to.
- 16y ago

i386 16y ago

I was hoping to know more about what corner cases he hit where he had to write subjectively ugly code and what part of the ecosystem hasn't matured yet (libraries for certain problems missing, etc). For me, building real applications and solving real problems in any new stack comes down to those two things not being much of an issue. I remember a friend and I hacking on a RoR app a few years ago where we spent more time trying to get Ruby Gems to play nice rather than actually writing the app. Now, those problems have been mostly shaken out and are a thing of the distant past. Is Node past this productivity immaturity hump yet?

penguinboy08 16y ago

Great article. Writing my node.js project has exposed me to all of the things listed. One thing I have to say is that NPM is good, but good be better with it's distro integration. The ubuntu install has broken permissions by default, which forces me to install stuff with sudo. I haven't had the time to try and fix it.

chapel 16y ago
NPM >0.3.0 changed to requiring sudo while installing. The reason being that it simplifies things logistically as well it allows NPM to install modules as nobody, since you have to be sudo to do that.
justincormack 16y ago
I dont think its worth using the Ubuntu install its just too old. At the moment easiest to install by hand, and script something for production.
- nailer 16y ago
  His issue (though not too obvious) is that NPM assumes /usr/local is writable by unprivileged users, which isn't secure or common in Linux distros or OS X. It's a bug in NPM, it should be fixed someday.

dbro 16y ago

This would have been helpful to me when I was starting out building http://domainjig.com http://domainjig.com - which is a simpler site than the author's. That said, I'd offer these responses to those looking for some advice: 1) Yes, wrapping every database call with an asynchronous callback is a drag. I found a helpful library that made some of my workflow easier to express and maintain: https://github.com/caolan/async https://github.com/caolan/async And I'm pretty sure I could simplify my code quite a bit more if I understood functional program patterns better already. 2) Libraries can be challenging, but I was able to find good ones for everything I needed. The author discussed version issues. To deal with that, I punted on all those node library managers and just jumped into the library code, checking it in to my repository and freezing it in time (in contrast to the authors point #9 about NPM being awesome for him). The other issue I found with libraries was that there are TOO MANY of them for any single thing. For example, there are at least a half dozen separate libraries to help with asynchronous workflow, and trying to choose which one would be "best" was difficult, and depended on freshness, documentation, and then actually test driving them. Github's encouragement of forking made this extra challenging, with many slightly tweaked variations for each separate library. I found node.js lived up to its promises (for me): coding in javascript on both server and client was helpful in avoiding context switching inside my head, and encouraged me to invest more effort in writing better quality code everywhere. And node.js is plenty fast. Another thing I'd add is that I used redis and tokyo cabinet as data stores and found the node.js interfaces to them to be mature and easy enough. It was convenient to use python's interactive shell to create and experiment with redis, and then port the routines to node.js. To summarize, consider this a positive vote for node.js.

augustl 16y ago

I just wrote TLS client certificate authentication for an internal systems API with node, in about 20 lines of code. Node's biggest power is a great abstraction for low level HTTP and socket stuff. You get direct access to everything, with a sane and elegant API. For plain old websites, I'd still use something like Rails or Django.

yannickmahe 16y ago

Great article. In your opinion, is Node.js mature enough to use in production code ? Or should I wait a little longer ? Also, what is the sate of the community (i.e. is it easy to find answers and tutorials) ?

Kilimanjaro 16y ago

I wish we had s-node (s for synchronous) so we do: session.start() data.load() template.render() instead of the cretan labyrinth it becomes coding for a-node

randylahey 16y ago
That completely defeats the point of using Node.js; that is, to use an evented model so your app isn't sitting around waiting for IO. s-node wouldn't be node at all. FWIW any other runtime/framework will do what you suggest. For example, Rhino.
- joeshaw 16y ago
  It would be nice of Node (or rather, v8 underneath it) supported generators and the "yield" keyword like Mozilla's Spidermonkey. Unfortunately, yield is a Javascript 1.7 extension for now, but there's hope for it in ES Harmony. Anyway, with generators and promises you can write asynchronous code which looks like synchronous code but which under the covers is still completely async and callback-driven. We use it in gjs (a GNOMEy JS runtime built on top of Spidermonkey). Instead of workflows like: function onServerRequest(req, res) { db.query("get filename", function(result) { fs.readContents(result, function (data) { res.write(data); }); }); } which can get pretty nasty if you're running things in serial, or want to run a couple of things in parallel but then run something serially, you can have a workflow more like this: function onServerRequest(req, res) { var queryPromise = db.queryPromise("get filename"); var filename = yield queryPromise; var data = yield db.readContentsPromise(filename); res.write(data); } The main difference between the two is that the async functions in the latter return promises instead of nothing. The db.queryPromise() function itself would probably look something like: function queryPromise(query) { var promise = new Promise(); db.query(query, function(result) { promise.putValue(result); }); return promise; } I imagine you could probably write a function to automatically generate promise wrappers for existing async functions as long as they followed a certain schema, and Node's functions tend to do well with that. Anyway, the onServerRequest() function would itself be wrapped in a function which took the generator values from it and returned its own promise. This way you could stack promises all the way up. The gjs implementation of this is here: http://git.gnome.org/browse/gjs/tree/modules/promise.js http://git.gnome.org/browse/gjs/tree/modules/promise.js This is pretty similar to Twisted's inlineCallbacks if you're familiar with them in Python.