6 ms·

Oddly enough, I just started a project that uses nodejs to access riak which is built on erlang. In addition to accessing data natively via erlang, riak speaks

Oddly enough, I just started a project that uses nodejs to access riak which is built on erlang. In addition to accessing data natively via erlang, riak speaks http and now protobuffers. Currently I have node.js speak to riak over http and I may change this in the future to use pb instead. The one issue I have with nodejs at the moment is that since it is nonblocking, when you call riak and ask for data nodejs will move on and continue execution when what you really wanted is that the next execution use values returned by riak. The main way to do this is to chain your calls so you basically nest your next call inside your current call. This can get sloppy. Apparently there is no built in method to wait on results. There were promisses but they were removed. Now there are some add-on libs like Step which I'm still trying to grok cause my js/node-fu is not up to snuff. Other than that, nodejs makes it ridiculously easy to start hacking away at all kinds of interesting stuff. anything that has to do with sockets or at a higher level, http interaction. Take a look at a post from yesterday to see just how easy, http://news.ycombinator.com/item?id=1301272 http://news.ycombinator.com/item?id=1301272.

jerf 17y ago
"Apparently there is no built in method to wait on results. There were promisses but they were removed." Because if you do that, the whole value proposition of Node.js completely falls apart. Now you've blocked the entire system waiting for your result. You might as well be programming in any other blocking language (all of which have Node.js-like libraries, by the way). This is what you get when you try to build concurrency on top of a language fundamentally built on non-concurrent primitives. Consider going pure Erlang, which does work exactly the way you are looking for it to work. You call a function, which passes around messages and gets the result for you, and the result comes back along the function call, and all the not-blocking is handled by the Erlang VM, totally transparently. (Oh, and you get OTP, which if your server is non-trivial you really need, even if you don't know it yet. And transparent multi-proc support, easy clustering support, couple of other things that Node.js may someday have with great, great, great, great effort and even then they'll still be hacks unless they fundamentally change Javascript to basically become Erlang.) Node.js is probably going to end up being a huge argument in favor of Erlang, because I suspect you will not be the last to encounter this problem in Node.js. It is absolutely inevitable that any significant application trying to be built in Node.js must collapse into a mess of callbacks and closures that will be very difficult to trace through. That's how it's designed. Any attempt to hack around this will have its own issues with gross hacks or difficulties with branching. I really don't see Node.js being a success in the long term. It is built on a foundation that is not up to the task. Building a really big Node.js app would be a nightmare, and the reasons why are not going to come out of the foundation easily, if at all. Here, let me give you an example with my work system I'm building. I have a single function which fans out an RPC request to multiple systems, collates the results with a timeout, and returns the results to the caller (or optionally calls some other code at the end, in the handler process). The initiator does not set up a callback, it just makes a call. The fanout mechanism does not set up a weird time-sliced mechanism to send out the calls, it just does it. There's some callbacks for handling whether it gets an error or a result coming back, which is pretty normal, and note, I don't have to contort around whether those callbacks may themselves want to do something over the network, they're just simple, one-layer callbacks that I would have implemented as callbacks in any language. (Imagine how you'd do that in an event-based system, deal with a response callback that may itself want to do a callback chain. It's perfectly feasible, I can easily see how to do it, but I sure as hell don't want to actually try to code it.) In Erlang, this is very natural; a function call that behind the scenes spawns a process that does what it says and says what it does, and works quite well. In an event-based system, this simple bit of functionality would be a callback explosion. I count a dozen, easily, and it would probably double if I really tried to do it. And that's not my whole system, either, it's just one bit. The multiplicative complexity explosion that occurs with Node.js's approach just isn't feasible. (Note Erlang handles "events" quite well, indeed, it's a native paradigm for many parts of the system; what it doesn't have is the multiplicative explosion of manually-created event handlers caused by offloading the scheduling onto the programmer.)
- Rauchg 17y ago
  re: promises/chaining Check out: http://github.com/rsms/node-promise http://github.com/rsms/node-promise http://github.com/creationix/step http://github.com/creationix/step http://github.com/creationix/do http://github.com/creationix/do Did the value proposition of node fall apart? . re: multi-proc support From www.nodejs.org architecture design statement: "But what about multiple-processor concurrency? Threads are necessary to scale programs to multi-core computers. Processes are necessary to scale to multi-core computers, not memory-sharing threads. The fundamentals of scalable systems are fast networking and non-blocking design—the rest is message passing. In future versions, Node will be able to fork new processes (using the Web Workers API ), but this is something that fits well into the current design."
  - jerf 17y ago
    "re: promises/chaining" Yup, exactly what I was expecting to see. Not even close to matching Erlang. You're still chopping up your code pretty grotesquely, and you're going to rapidly find out that while that made a few things you were used to doing work well, when you want to take the next step of logical flow that stuff is going to stick in your craw. It's a fairly normal abstraction, in that for the price of making one thing easy it's going to make other things harder. I implemented that for myself in browser-side javascript years ago, so I actually do know what I'm talking about. (All 3 are really the same basic approach, and it's the only basic approach there is.) It's a little more convenient than the alternative, but it only lightly brushes over a couple of the pointy bits of this approach, and leaves quite a lot of the rest of them still in your face. " Threads are necessary to scale programs to multi-core computers. Processes are necessary to scale to multi-core computers, not memory-sharing threads. The fundamentals of scalable systems are fast networking and non-blocking design—the rest is message passing. In future versions, Node will be able to fork new processes (using the Web Workers API ), but this is something that fits well into the current design." I don't have to do anything in Erlang to get multi-processor support. I just run the VM. Rationalizations about why multi-process is hard and it's OK that we don't have it, we don't really need it are irrelevant, I have it in hand. And if you're not familiar with Erlang, you may need to know what it's definition of "process" is: An "Erlang process" is basically a thread, but has process-like isolation from other threads, which is where the word "process" comes in. Erlang essentially automatically runs things in the most-optimized thread-pooling system you've ever seen, and I don't have to think about it. Are you going to be able to spawn off a few ten thousand Web Workers without worrying about the resource usage? (Or hundred thousands? ... or millions?) I'd be surprised. It might be possible, but I'd be surprised. Bear in mind I'm coming at this as a near-decade user of Javascript and a multi-year professional user of Erlang. I am not unfamiliar with the languages. You can not match Erlang in Javascript, not even with web workers. You can smooth things around, but all you'll get is a yet-more beguiling slide into complexity if you try to do anything at a large scale. (You'd honestly be better off if you hit the limits of Node.js sooner, rather than later.)
BrianHammond 17y ago
"This can get sloppy." Indeed it can. I am a fan of flow-js however. It has really helped me to avoid getting sloppy in Node.js. http://github.com/willconant/flow-js http://github.com/willconant/flow-js

jerf 17y ago

"Apparently there is no built in method to wait on results. There were promisses but they were removed." Because if you do that, the whole value proposition of Node.js completely falls apart. Now you've blocked the entire system waiting for your result. You might as well be programming in any other blocking language (all of which have Node.js-like libraries, by the way). This is what you get when you try to build concurrency on top of a language fundamentally built on non-concurrent primitives. Consider going pure Erlang, which does work exactly the way you are looking for it to work. You call a function, which passes around messages and gets the result for you, and the result comes back along the function call, and all the not-blocking is handled by the Erlang VM, totally transparently. (Oh, and you get OTP, which if your server is non-trivial you really need, even if you don't know it yet. And transparent multi-proc support, easy clustering support, couple of other things that Node.js may someday have with great, great, great, great effort and even then they'll still be hacks unless they fundamentally change Javascript to basically become Erlang.) Node.js is probably going to end up being a huge argument in favor of Erlang, because I suspect you will not be the last to encounter this problem in Node.js. It is absolutely inevitable that any significant application trying to be built in Node.js must collapse into a mess of callbacks and closures that will be very difficult to trace through. That's how it's designed. Any attempt to hack around this will have its own issues with gross hacks or difficulties with branching. I really don't see Node.js being a success in the long term. It is built on a foundation that is not up to the task. Building a really big Node.js app would be a nightmare, and the reasons why are not going to come out of the foundation easily, if at all. Here, let me give you an example with my work system I'm building. I have a single function which fans out an RPC request to multiple systems, collates the results with a timeout, and returns the results to the caller (or optionally calls some other code at the end, in the handler process). The initiator does not set up a callback, it just makes a call. The fanout mechanism does not set up a weird time-sliced mechanism to send out the calls, it just does it. There's some callbacks for handling whether it gets an error or a result coming back, which is pretty normal, and note, I don't have to contort around whether those callbacks may themselves want to do something over the network, they're just simple, one-layer callbacks that I would have implemented as callbacks in any language. (Imagine how you'd do that in an event-based system, deal with a response callback that may itself want to do a callback chain. It's perfectly feasible, I can easily see how to do it, but I sure as hell don't want to actually try to code it.) In Erlang, this is very natural; a function call that behind the scenes spawns a process that does what it says and says what it does, and works quite well. In an event-based system, this simple bit of functionality would be a callback explosion. I count a dozen, easily, and it would probably double if I really tried to do it. And that's not my whole system, either, it's just one bit. The multiplicative complexity explosion that occurs with Node.js's approach just isn't feasible. (Note Erlang handles "events" quite well, indeed, it's a native paradigm for many parts of the system; what it doesn't have is the multiplicative explosion of manually-created event handlers caused by offloading the scheduling onto the programmer.)

Rauchg 17y ago
re: promises/chaining Check out: http://github.com/rsms/node-promise http://github.com/rsms/node-promise http://github.com/creationix/step http://github.com/creationix/step http://github.com/creationix/do http://github.com/creationix/do Did the value proposition of node fall apart? . re: multi-proc support From www.nodejs.org architecture design statement: "But what about multiple-processor concurrency? Threads are necessary to scale programs to multi-core computers. Processes are necessary to scale to multi-core computers, not memory-sharing threads. The fundamentals of scalable systems are fast networking and non-blocking design—the rest is message passing. In future versions, Node will be able to fork new processes (using the Web Workers API ), but this is something that fits well into the current design."
- jerf 17y ago
  "re: promises/chaining" Yup, exactly what I was expecting to see. Not even close to matching Erlang. You're still chopping up your code pretty grotesquely, and you're going to rapidly find out that while that made a few things you were used to doing work well, when you want to take the next step of logical flow that stuff is going to stick in your craw. It's a fairly normal abstraction, in that for the price of making one thing easy it's going to make other things harder. I implemented that for myself in browser-side javascript years ago, so I actually do know what I'm talking about. (All 3 are really the same basic approach, and it's the only basic approach there is.) It's a little more convenient than the alternative, but it only lightly brushes over a couple of the pointy bits of this approach, and leaves quite a lot of the rest of them still in your face. " Threads are necessary to scale programs to multi-core computers. Processes are necessary to scale to multi-core computers, not memory-sharing threads. The fundamentals of scalable systems are fast networking and non-blocking design—the rest is message passing. In future versions, Node will be able to fork new processes (using the Web Workers API ), but this is something that fits well into the current design." I don't have to do anything in Erlang to get multi-processor support. I just run the VM. Rationalizations about why multi-process is hard and it's OK that we don't have it, we don't really need it are irrelevant, I have it in hand. And if you're not familiar with Erlang, you may need to know what it's definition of "process" is: An "Erlang process" is basically a thread, but has process-like isolation from other threads, which is where the word "process" comes in. Erlang essentially automatically runs things in the most-optimized thread-pooling system you've ever seen, and I don't have to think about it. Are you going to be able to spawn off a few ten thousand Web Workers without worrying about the resource usage? (Or hundred thousands? ... or millions?) I'd be surprised. It might be possible, but I'd be surprised. Bear in mind I'm coming at this as a near-decade user of Javascript and a multi-year professional user of Erlang. I am not unfamiliar with the languages. You can not match Erlang in Javascript, not even with web workers. You can smooth things around, but all you'll get is a yet-more beguiling slide into complexity if you try to do anything at a large scale. (You'd honestly be better off if you hit the limits of Node.js sooner, rather than later.)

BrianHammond 17y ago

"This can get sloppy." Indeed it can. I am a fan of flow-js however. It has really helped me to avoid getting sloppy in Node.js. http://github.com/willconant/flow-js http://github.com/willconant/flow-js