4 ms·

Is it possible to gain a thorough appreciation for the benefits and technical implementation of a project like Node.js without being knee-deep in the scalable w

by mcantor 16y ago

Is it possible to gain a thorough appreciation for the benefits and technical implementation of a project like Node.js without being knee-deep in the scalable web app universe? Nothing I have written has needed to handle more than, oh, six requests per eon yet, so I have never felt the constraints of "synchronous, I/O-bound" software development. I put "synchronous, I/O-bound" in scare quotes because I don't really grok what it means. Sure, I can talk the talk and mumble a few buzzwords under my breath, but when it really comes down to it, I have only a cursory understanding of what "event driven," "asynchronous" or, Hell, even "single threaded" means. I've read the Wikipedia pages and made it to "Hello world" with a few of these concepts, but I feel like I'm really missing a huge chunk of very important knowledge & experience. What kind of project can I work on that would go down this road? Are there any good books, physical or electronic? Or am I doomed to be disconnected from this universe of experience until one of my crappy web projects gets Digged and I learn through trial by fire? I just know there's more to this than "asynchronous method calls will not block method calls that come after them." EDIT: I noticed that someone downvoted this--I'd be extremely interested to know why. Am I missing something incredibly obvious?

jimmyjazz14 16y ago
learning a little about libev and epoll is a good starting place.
- aarongough 16y ago
  Wouldn't going so low-level be a bit counter-productive at this point in his learning process? I would think that playing with existing async systems like Node, EventMachine and Twisted would be more productive in the short-term and give him a base on which to build deeper knowledge...
  - mcantor 16y ago
    It's not that it would be counterproductive; it's simply impenetrable. I read libev's awful homepage and perused the epoll manpage, but they don't mean anything to me--I don't have enough context for them to sink in. (Also... and I say this understanding that I am a man filled with ignorance and hubris... how relevant can libev really be, if its canonical source repo is still maintained with cvs?) But, manpages are rarely where one goes to begin learning about something... on the very first section, I am already inundated with things that mean nothing to me: what are "edge-triggered" and "level-triggered?" What does it mean to "watch" a file descriptor? I know just enough about GNU/Linux I/O to be dangerous, so I can start to build an inkling of understanding based on this information: "watching" a file descriptor is probably how we register event listeners, and "edge vs. level" is most likely a decision about when handlers receive their data. But that's about as far as I can get before I have to start descending into the terms I don't completely understand. When I find something that explains those, it will probably rely on a bunch of other terms that I must descend into, and so forth. By the time I finally recurse all the way back up to my original quest for knowledge ("What does event-driven mean?"), I will be so inundated with new information that nothing sticks! Thanks for the suggestion about Twisted, EM and Node.js... I'll have to make some time to play with them soon.
    - aarongough 16y ago
      Exactly... That is why I recommend sticking to high-level libraries and tackling a simple real-world problem to start off. Even just playing with writing some meaningful client-side JavaScript will be enough to give you more of a 'feel' for event-driven programming... It's not going to give you a huge amount of knowledge straight out but it will definitely give you enough to get you started if you decide to go further.
  - jimmyjazz14 16y ago
    I don't believe so, unless you know exactly what asynchronous IO is and why you might need to use it then it might just seem like a more complicated method of programming. The "why" of asynchronous IO is very well explained by the C10K problem <http://www.kegel.com/c10k.html> http://www.kegel.com/c10k.html>.
    - mcantor 16y ago
      That link may be exactly what I need. Thank you!
aarongough 16y ago
Try making something that depends on an I/O resource that is really slow, that will give you more of a feeling for it quickly. I did this in one of my projects (Ajax-Gist.com) where I was directly querying a 3rd party API (GitHub) for each request. That meant that each request was client->server roundtrip time plus server->api roundtrip time, which worked out to be around 200ms, or a total throughput of 5 requests per second. This is obviously unacceptable and a complete waste of resources so I re-wrote the whole thing to be asynchronous. The new application, running in a single thread on the same hardware can now handle around 70 requests per second with 30 concurrent users. That's pretty damn fast for a Rails application... Now I didn't use Node to solve my problem (though I considered it!) but the principles and the message I took away was the same. I think it's a handy tool to have in your toolkit, but as you've already realized it's simply not necessary for a lot of projects, particularly smaller ones.
- mcantor 16y ago
  Just FYI, http://ajaxgist.com http://ajaxgist.com results in a 404. What library/technique did you end up using to make your app asynchronous?
  - aarongough 16y ago
    Haha, I got my own domain name wrong! Clearly it's been a long day... Link: http://ajax-gist.com/ http://ajax-gist.com/ In the end I built what was essentially a proof-of-concept asynchronous Rails 3 application. The libraries I used under the hood to get everything async were: * Rack/Fiber-Pool: http://github.com/mperham/rack-fiber_pool http://github.com/mperham/rack-fiber_pool * EM Synchrony: http://github.com/igrigorik/em-synchrony http://github.com/igrigorik/em-synchrony * Eventmachine: http://rubyeventmachine.com/ http://rubyeventmachine.com/ Using Rails for what is essentially a thin proxy is overkill, and I could have gotten better performance with Node or by writing a Rack application directly. But I wanted to take the opportunity to get a handle on what would be involved with doing this in Rails, as not too many people have done it before that I am aware of (and at the time no-one had done it with Rails 3).
    - 16y ago

by mcantor 16y ago

jimmyjazz14 16y ago

learning a little about libev and epoll is a good starting place.

aarongough 16y ago
Wouldn't going so low-level be a bit counter-productive at this point in his learning process? I would think that playing with existing async systems like Node, EventMachine and Twisted would be more productive in the short-term and give him a base on which to build deeper knowledge...
- mcantor 16y ago
  It's not that it would be counterproductive; it's simply impenetrable. I read libev's awful homepage and perused the epoll manpage, but they don't mean anything to me--I don't have enough context for them to sink in. (Also... and I say this understanding that I am a man filled with ignorance and hubris... how relevant can libev really be, if its canonical source repo is still maintained with cvs?) But, manpages are rarely where one goes to begin learning about something... on the very first section, I am already inundated with things that mean nothing to me: what are "edge-triggered" and "level-triggered?" What does it mean to "watch" a file descriptor? I know just enough about GNU/Linux I/O to be dangerous, so I can start to build an inkling of understanding based on this information: "watching" a file descriptor is probably how we register event listeners, and "edge vs. level" is most likely a decision about when handlers receive their data. But that's about as far as I can get before I have to start descending into the terms I don't completely understand. When I find something that explains those, it will probably rely on a bunch of other terms that I must descend into, and so forth. By the time I finally recurse all the way back up to my original quest for knowledge ("What does event-driven mean?"), I will be so inundated with new information that nothing sticks! Thanks for the suggestion about Twisted, EM and Node.js... I'll have to make some time to play with them soon.
  - aarongough 16y ago
    Exactly... That is why I recommend sticking to high-level libraries and tackling a simple real-world problem to start off. Even just playing with writing some meaningful client-side JavaScript will be enough to give you more of a 'feel' for event-driven programming... It's not going to give you a huge amount of knowledge straight out but it will definitely give you enough to get you started if you decide to go further.
- jimmyjazz14 16y ago
  I don't believe so, unless you know exactly what asynchronous IO is and why you might need to use it then it might just seem like a more complicated method of programming. The "why" of asynchronous IO is very well explained by the C10K problem <http://www.kegel.com/c10k.html> http://www.kegel.com/c10k.html>.
  - mcantor 16y ago
    That link may be exactly what I need. Thank you!

aarongough 16y ago

Try making something that depends on an I/O resource that is really slow, that will give you more of a feeling for it quickly. I did this in one of my projects (Ajax-Gist.com) where I was directly querying a 3rd party API (GitHub) for each request. That meant that each request was client->server roundtrip time plus server->api roundtrip time, which worked out to be around 200ms, or a total throughput of 5 requests per second. This is obviously unacceptable and a complete waste of resources so I re-wrote the whole thing to be asynchronous. The new application, running in a single thread on the same hardware can now handle around 70 requests per second with 30 concurrent users. That's pretty damn fast for a Rails application... Now I didn't use Node to solve my problem (though I considered it!) but the principles and the message I took away was the same. I think it's a handy tool to have in your toolkit, but as you've already realized it's simply not necessary for a lot of projects, particularly smaller ones.

mcantor 16y ago
Just FYI, http://ajaxgist.com http://ajaxgist.com results in a 404. What library/technique did you end up using to make your app asynchronous?
- aarongough 16y ago
  Haha, I got my own domain name wrong! Clearly it's been a long day... Link: http://ajax-gist.com/ http://ajax-gist.com/ In the end I built what was essentially a proof-of-concept asynchronous Rails 3 application. The libraries I used under the hood to get everything async were: * Rack/Fiber-Pool: http://github.com/mperham/rack-fiber_pool http://github.com/mperham/rack-fiber_pool * EM Synchrony: http://github.com/igrigorik/em-synchrony http://github.com/igrigorik/em-synchrony * Eventmachine: http://rubyeventmachine.com/ http://rubyeventmachine.com/ Using Rails for what is essentially a thin proxy is overkill, and I could have gotten better performance with Node or by writing a Rack application directly. But I wanted to take the opportunity to get a handle on what would be involved with doing this in Rails, as not too many people have done it before that I am aware of (and at the time no-one had done it with Rails 3).
  - 16y ago