5 ms·

The big reason callback-based systems are hard (despite the fact that built-in flow control constructs need to be re-implemented) is that functions are no longe

The big reason callback-based systems are hard (despite the fact that built-in flow control constructs need to be re-implemented) is that functions are no longer composable. That is, if I write a bunch of code that doesn't need to do any I/O, I'll just return a value to the caller. If at any point in the future the spec changes and any function in this call stack needs to do any I/O, all the code that ends up calling this function needs to be refactored to use callbacks. There really should be language support for this sort of thing (like coroutines) so these sort of cascading changes don't need to happen.

bascule 15y ago
There are ways to compose functions, like the deferrable pattern. They just all kind of suck and are poor replacements for having a stack. I'm all for using coroutines to solve this problem. That's the approach taken by my Celluloid::IO library: https://github.com/tarcieri/celluloid-io https://github.com/tarcieri/celluloid-io Unfortunately Ryan Dahl is adamantly opposed to coroutines so that's not going to happen in Node any time soon.
- peterhunt 15y ago
  Deferreds make your code nicer, but they still don't magically make your code composable. What would make Node more attractive is if it supported copy-on-write multithreading and gave me a way to cheat and use asynchronous I/O (like a wait(myFunctionThatTakesACallbackOrDeferred) function)
  - bascule 15y ago
    There are many ways to compose deferrables, primarily by grouping several of them together into something which is also a deferrable. See the em-scenario gem for examples: https://github.com/athoune/em-scenario https://github.com/athoune/em-scenario Note: I still think this approach sucks. V8 provides a really awesome shared-nothing multithreading scheme via web workers. It's just nobody uses them.
    - peterhunt 15y ago
      Oh right that's definitely true and is much more elegant. I was talking about when a function written in synchronous style (in a long stack of synchronous calls) needs to call something asynchronous.
  - ufo 15y ago
    I wouldn't say the problem with Deferreds is composability - the big problem, as you already mentioned, is fragility towards future changes
- guan 15y ago
  I like Kris Zyp's promises and use them a lot: https://github.com/kriszyp/promised-io https://github.com/kriszyp/promised-io
- no_more_death 15y ago
  Node supports no language extensions that V8 doesn't. Ryan Dahl says: "Node does not modify the JavaScript runtime. This is for the ECMA committee and V8 to decide. If they add coroutines (which they won't) then we will include it. If they add generators we will include them." See Tim's thread, My Humble Coroutine Proposal (https://groups.google.com/forum/#!topic/nodejs/HJOyNMKLgB8 https://groups.google.com/forum/#!topic/nodejs/HJOyNMKLgB8). Warning: long. So, you don't like javascript? Don't use it. But you won't beat the control over program state offered by javascript -- not until computers understand their programmers well enough to reason about program state. We need better computers, better computer science, and better programmers -- that's all. Then we can replace NodeJS with something better. Until then, NodeJS is almost certainly the most tasteful solution to the most common problems. I hope its replacement meets so high a standard.
jvoorhis 15y ago
Callback-centric code can be composable if you adhere to the continuation passing style (CPS) discipline. While it can be difficult to write CPS code directly, it's a well-trodden path in computer science.
- bascule 15y ago
  CPS is also generally used as an IR in compilers, not something you write by hand
  - nas 15y ago
    Yes. In Scheme, call/cc is generally not used directly. Instead, you build higher level abstractions, for example full co-routines or perhaps more limited "generators" like Python has. Co-routines have been around for more than 30 years. It's quite said to see the Node.js developers ignore decades of language research. Implementing concurrency using callback functions is definitely a "worse is better" approach. Maybe it will win because of that. Edit: I guess a hopeful idea is that there should be no reason something like call/cc could not be added to Node.js. In that case, the extensive library of non-blocking functionally will be very handy since you could build a sane continuation based interface on top of it and escape from callback purgatory.
- peterhunt 15y ago
  Yeah that's true, but using CPS for everything means you can't really use normal flow control anywhere which IMO makes it inappropriate to use for day-to-day engineering.
  - robdor 15y ago
    I have been using node.js for my day-to-day engineering for the past year. It certainly takes a couple of weeks to get a good feel for solving problems without being able to use normal flow control. But once you get it becomes easy. So I wouldn't call in inappropriate, just different. It's much like the feeling of learning your first functional programming language. It's not wrong, just different until you adjust the way you reason about your programs.
freshhawk 15y ago
I'm guessing monads will be a potential solution implemented pretty soon. I'd be interested in seeing how that works out.
- ufo 15y ago
  There is already a Monad based solution - its Deferreds / Promises. However, monads don't solve this problem - they cause it, since their primary concern is correctness and not converting between monadic and non-monadic code. If you have a pure function and need to convert it to a monadic action there will be lots of collateral damage as functions that interacted with the old function have to be converted to monadic style.
  - freshhawk 15y ago
    Yeah, just after I wrote that I realized that Deferreds/Promises are monads. You don't have to convert them, but you do need a way to write code using those pure functions "inside" the monad which I don't think is easy with this model. You should be able to just chain your actions together, with a wrapper function to turn pure functions into actions. I don't see why you would need to convert the actual pure functions to monadic style. I guess what I had in my head when I made the comment was the convenience functions or the syntactic sugar around a monadic solution. Seems like the node guys are playing with things like this but all the solutions I've seen seem so ugly.
    - ufo 15y ago
      Pure code can be used in the monad just fine, (ie.: you can return values up to the monad and lift functions) //turn a function into a monadic action function lift(f){ return function(/**/){ var d = new Deferred(); //Im using the Dojo API d.resolve( f.call(this, arguments) ); return d.promise; };}; var f = function(x){ return x+1} lift(f)(0) .then(function(y){ ... }) The problem is the opposite direction: pure code using monadic code and pure code turning into monadic code If you have something like var x = f1( f2( f3() ) ) And f2 becomes a monadic action you have to rewrite this bit as var xPromise = f2( f3() ) .then(function(f2result){ return f1(f2result); });
masklinn 15y ago
> The big reason callback-based systems are hard (despite the fact that built-in flow control constructs need to be re-implemented) is that functions are no longer composable Absolutely, I've had it happen in "big" (client-side) JS projects. Only way I've found so far to handle this is to make anything which might ever have any reason to become asynchronous (so anything but helper functions) take callbacks or return deferred objects. Always. But then an other issue arises: for various reasons, callbacks-based code which works synchronously may fail asynchronously, and the other way around. And then it starts getting real fun as you still have to go through all your (supposedly) carefully constructed callbacks-based code to find out what reason it would have to fail (alternatively, you create both sync and async tests for all pieces of callbacks-based code)
SomeCallMeTim 15y ago
Coroutines on a server...like OpenResty [1] (a package containing Nginx w/Lua integration)? The example "asynchronous get" becomes something like: local myObject = query{ id=3244 }; The query function can be written to handle a cache lookup, a database query that gets stored in the cache, and any other logging you want, because Lua has coroutines. All I/O that gets sent through the "ngx" query object (which can connect to local or remote ports) yields control to the main loop. "query" is an example of a function you could create; its implementation (with a cache lookup) could look something like (yes, I use CouchDB...): function query(t) local result = ngx.location.capture( "/cacheserver/id:"..t.id ); if #result == 0 then result = ngx.location.capture( "/couchdb/usertable/"..t.id ); end return result end I've heard reports of 50k+ connections/second on a VPS running Nginx, LuaJit, and the LuaNginxModule, and on my low-end VPS it easily handles 2000+ connections per second (with CouchDB queries) with no more than 250ms latency. Actually, that was as many connections I could send at it, so it may be able to handle a lot more. [1] http://openresty.org/ http://openresty.org/
olegp 15y ago
Node has coroutine support via the fibers package: https://github.com/laverdet/node-fibers https://github.com/laverdet/node-fibers. For some benchmarks, check out my Common Node project's README: https://github.com/olegp/common-node https://github.com/olegp/common-node

bascule 15y ago

There are ways to compose functions, like the deferrable pattern. They just all kind of suck and are poor replacements for having a stack. I'm all for using coroutines to solve this problem. That's the approach taken by my Celluloid::IO library: https://github.com/tarcieri/celluloid-io https://github.com/tarcieri/celluloid-io Unfortunately Ryan Dahl is adamantly opposed to coroutines so that's not going to happen in Node any time soon.

peterhunt 15y ago
Deferreds make your code nicer, but they still don't magically make your code composable. What would make Node more attractive is if it supported copy-on-write multithreading and gave me a way to cheat and use asynchronous I/O (like a wait(myFunctionThatTakesACallbackOrDeferred) function)
- bascule 15y ago
  There are many ways to compose deferrables, primarily by grouping several of them together into something which is also a deferrable. See the em-scenario gem for examples: https://github.com/athoune/em-scenario https://github.com/athoune/em-scenario Note: I still think this approach sucks. V8 provides a really awesome shared-nothing multithreading scheme via web workers. It's just nobody uses them.
  - peterhunt 15y ago
    Oh right that's definitely true and is much more elegant. I was talking about when a function written in synchronous style (in a long stack of synchronous calls) needs to call something asynchronous.
- ufo 15y ago
  I wouldn't say the problem with Deferreds is composability - the big problem, as you already mentioned, is fragility towards future changes
guan 15y ago
I like Kris Zyp's promises and use them a lot: https://github.com/kriszyp/promised-io https://github.com/kriszyp/promised-io
no_more_death 15y ago
Node supports no language extensions that V8 doesn't. Ryan Dahl says: "Node does not modify the JavaScript runtime. This is for the ECMA committee and V8 to decide. If they add coroutines (which they won't) then we will include it. If they add generators we will include them." See Tim's thread, My Humble Coroutine Proposal (https://groups.google.com/forum/#!topic/nodejs/HJOyNMKLgB8 https://groups.google.com/forum/#!topic/nodejs/HJOyNMKLgB8). Warning: long. So, you don't like javascript? Don't use it. But you won't beat the control over program state offered by javascript -- not until computers understand their programmers well enough to reason about program state. We need better computers, better computer science, and better programmers -- that's all. Then we can replace NodeJS with something better. Until then, NodeJS is almost certainly the most tasteful solution to the most common problems. I hope its replacement meets so high a standard.

jvoorhis 15y ago

Callback-centric code can be composable if you adhere to the continuation passing style (CPS) discipline. While it can be difficult to write CPS code directly, it's a well-trodden path in computer science.

bascule 15y ago
CPS is also generally used as an IR in compilers, not something you write by hand
- nas 15y ago
  Yes. In Scheme, call/cc is generally not used directly. Instead, you build higher level abstractions, for example full co-routines or perhaps more limited "generators" like Python has. Co-routines have been around for more than 30 years. It's quite said to see the Node.js developers ignore decades of language research. Implementing concurrency using callback functions is definitely a "worse is better" approach. Maybe it will win because of that. Edit: I guess a hopeful idea is that there should be no reason something like call/cc could not be added to Node.js. In that case, the extensive library of non-blocking functionally will be very handy since you could build a sane continuation based interface on top of it and escape from callback purgatory.
peterhunt 15y ago
Yeah that's true, but using CPS for everything means you can't really use normal flow control anywhere which IMO makes it inappropriate to use for day-to-day engineering.
- robdor 15y ago
  I have been using node.js for my day-to-day engineering for the past year. It certainly takes a couple of weeks to get a good feel for solving problems without being able to use normal flow control. But once you get it becomes easy. So I wouldn't call in inappropriate, just different. It's much like the feeling of learning your first functional programming language. It's not wrong, just different until you adjust the way you reason about your programs.

freshhawk 15y ago

I'm guessing monads will be a potential solution implemented pretty soon. I'd be interested in seeing how that works out.

ufo 15y ago
There is already a Monad based solution - its Deferreds / Promises. However, monads don't solve this problem - they cause it, since their primary concern is correctness and not converting between monadic and non-monadic code. If you have a pure function and need to convert it to a monadic action there will be lots of collateral damage as functions that interacted with the old function have to be converted to monadic style.
- freshhawk 15y ago
  Yeah, just after I wrote that I realized that Deferreds/Promises are monads. You don't have to convert them, but you do need a way to write code using those pure functions "inside" the monad which I don't think is easy with this model. You should be able to just chain your actions together, with a wrapper function to turn pure functions into actions. I don't see why you would need to convert the actual pure functions to monadic style. I guess what I had in my head when I made the comment was the convenience functions or the syntactic sugar around a monadic solution. Seems like the node guys are playing with things like this but all the solutions I've seen seem so ugly.
  - ufo 15y ago
    Pure code can be used in the monad just fine, (ie.: you can return values up to the monad and lift functions) //turn a function into a monadic action function lift(f){ return function(/**/){ var d = new Deferred(); //Im using the Dojo API d.resolve( f.call(this, arguments) ); return d.promise; };}; var f = function(x){ return x+1} lift(f)(0) .then(function(y){ ... }) The problem is the opposite direction: pure code using monadic code and pure code turning into monadic code If you have something like var x = f1( f2( f3() ) ) And f2 becomes a monadic action you have to rewrite this bit as var xPromise = f2( f3() ) .then(function(f2result){ return f1(f2result); });

masklinn 15y ago

> The big reason callback-based systems are hard (despite the fact that built-in flow control constructs need to be re-implemented) is that functions are no longer composable Absolutely, I've had it happen in "big" (client-side) JS projects. Only way I've found so far to handle this is to make anything which might ever have any reason to become asynchronous (so anything but helper functions) take callbacks or return deferred objects. Always. But then an other issue arises: for various reasons, callbacks-based code which works synchronously may fail asynchronously, and the other way around. And then it starts getting real fun as you still have to go through all your (supposedly) carefully constructed callbacks-based code to find out what reason it would have to fail (alternatively, you create both sync and async tests for all pieces of callbacks-based code)

SomeCallMeTim 15y ago

Coroutines on a server...like OpenResty [1] (a package containing Nginx w/Lua integration)? The example "asynchronous get" becomes something like: local myObject = query{ id=3244 }; The query function can be written to handle a cache lookup, a database query that gets stored in the cache, and any other logging you want, because Lua has coroutines. All I/O that gets sent through the "ngx" query object (which can connect to local or remote ports) yields control to the main loop. "query" is an example of a function you could create; its implementation (with a cache lookup) could look something like (yes, I use CouchDB...): function query(t) local result = ngx.location.capture( "/cacheserver/id:"..t.id ); if #result == 0 then result = ngx.location.capture( "/couchdb/usertable/"..t.id ); end return result end I've heard reports of 50k+ connections/second on a VPS running Nginx, LuaJit, and the LuaNginxModule, and on my low-end VPS it easily handles 2000+ connections per second (with CouchDB queries) with no more than 250ms latency. Actually, that was as many connections I could send at it, so it may be able to handle a lot more. [1] http://openresty.org/ http://openresty.org/

olegp 15y ago

Node has coroutine support via the fibers package: https://github.com/laverdet/node-fibers https://github.com/laverdet/node-fibers. For some benchmarks, check out my Common Node project's README: https://github.com/olegp/common-node https://github.com/olegp/common-node