5 ms·

> Underscore.js for Iterations: each implicitly blocks ???? Blocking refers to IO system calls. underscore is not an IO library. Does a 15 year programmer with

> Underscore.js for Iterations: each implicitly blocks ???? Blocking refers to IO system calls. underscore is not an IO library. Does a 15 year programmer with C++ experience not know what an O_NONBLOCK system call is? Blocking or non blocking is about how you are performing IO with the external world. It has nothing to do with the world internal to your program: callbacks are merely correlated with nonblocking io, continuation passing style does not constitute nonblocking in an of itself. You do not have to use continuation passing style and lambdas for code that does not perform IO for it to be "nonblocking".

rpwilcox 15y ago
The way I understood it is that if I have a section of Javascript that takes, say, 1/2 a second to run, and does not yield control back to the event loop, by definition blocks the event loop from firing. In a preemptive multithreaded environment it's different, but here you have one single thread and - at least to my understanding - any code that takes time to process blocks the event loop from being able to respond to incoming requests. Unless I'm missing something and node can (say) preempt your process at certain defined conditions. ("Call a function? Ok, let's check the event loop first and come right back"). I didn't see that anywhere (but I also didn't want to dig into libenv. I understand that IO is expensive, but so is performing string interpolation on a string in memory 50 times (for whatever reason).
- bigiain 15y ago
  This is exactly what Zed was pointing out with his intentionally pessimized fibonacci generator. Sure node can make waiting on disk or network asyncronous - but it's no silver bullet, a poorly though out node.js server might scale just as badly as any other poorly thought out server in any other language. You still need to understand what's going on and not code yourself into the low performance corners.
  - stock_toaster 15y ago
    > This is exactly what Zed was pointing out with his intentionally pessimized fibonacci generator. You mean Ted Dziuba [1]? [1]: http://teddziuba.com/2011/10/node-js-is-cancer.html http://teddziuba.com/2011/10/node-js-is-cancer.html
  - rpwilcox 15y ago
    Exactly. The fibonacci generator (and subsequent async implementations) are one of the things I looked at when writing the article.
- davidhollander 15y ago
  >I understand that IO is expensive, but so is performing string interpolation on a string in memory 50 times (for whatever reason). I think you are conflating waiting and working. Both cost time, but they are not the same. Waiting can be avoided (nonblocking), working cannot. Iterating over an array in most languages probably takes .000001 seconds. Reading from a bad connection could take 30+ seconds, or whatever your servers timeout is. Since the 30 seconds is spent _waiting_, it can be avoided within a single thread of execution. On the other hand, if you are spending 30 seconds _working_ to compute prime numbers, this cannot be avoided in a single thread of execution. Blocking vs nonblocking refers to dealing with waits. Does each IO operation do the waiting, or does each IO operation return immediately and we pool waits into a single poll() (etc.) call? Parallelization refers to dealing with work. In the rare instance one needs to generate huge fibonacci sequences, one can do so in a seperate fork\process\thread\cpu core\machine, and report results back to the event loop via pipe\socket\shared memory. [1] Multiplexing waits in a single thread makes sense, multiplexing work in a single thread such as array iteration does not: on the contrary it sounds like a good way to increase the overhead of iteration and generate more work rather than reduce it. It's confusing because some tools such as threads can be used to address both the problem of parallel work and the problem of parallel waits. [1] edit: quick example in lua: -- In fork, write to the blocking end of a pipe local r = formfork(function(w) local work = fibonacci(10^10) w:write(work) w:close() end) -- In event loop, read from the nonblocking end of a pipe readline(r , 1024, function(r, line) print('got work', line) close(r) end)
glenjamin 15y ago
The main issue preventing widespread use of underscore in nodejs land is that it isn't set up to accept node-style callbacks, hence you can't mix it with asynchronous code. Eg. Mapping an async function on each member of the array, the async library provides similar functions to underscore while providing callbacks for completion. The same applies to using normal JS constructors, it's tricky to get a completion callback - I've been doing stuff like this: Klass.create = function(args, callback) { var instance = new Klass(args); instance.initialise(callback); }
- davidhollander 15y ago
  > I've been doing stuff like this Why use callbacks over 'return' for anything other than IO and waiting? If there's not two out of sync parties, there's no need to tell another person to call you back, and no advantage to performing an extra function call. re: yummyfajitas: that does not use a callback, it returns a list. And if retrieveURL is nonblocking, then iteration is nonblocking.
  - yummyfajitas 15y ago
    I believe the use case is stuff like this: _.map(listOfURLs, retrieveURL)
  - rpwilcox 15y ago
    > And if retrieveURL is nonblocking, then iteration is nonblocking. Something I mention in the article/paper/whatever is using operations that are technically blocking to call functions that happen asynchronously. So there are certainly uses for sync iteration tools (like _.each()) I think it depends on where you put the async assumptions. Do you know/assume that the function itself takes care of the async aspects, or do you know/assume that the caller must take care to set up that structure? If the answer is the former, then use whatever structure you want, because the async is taken care of for you. If the answer is the latter, perhaps you need a iteration function that works asynchronously.
    - davidhollander 15y ago

rpwilcox 15y ago

The way I understood it is that if I have a section of Javascript that takes, say, 1/2 a second to run, and does not yield control back to the event loop, by definition blocks the event loop from firing. In a preemptive multithreaded environment it's different, but here you have one single thread and - at least to my understanding - any code that takes time to process blocks the event loop from being able to respond to incoming requests. Unless I'm missing something and node can (say) preempt your process at certain defined conditions. ("Call a function? Ok, let's check the event loop first and come right back"). I didn't see that anywhere (but I also didn't want to dig into libenv. I understand that IO is expensive, but so is performing string interpolation on a string in memory 50 times (for whatever reason).

bigiain 15y ago
This is exactly what Zed was pointing out with his intentionally pessimized fibonacci generator. Sure node can make waiting on disk or network asyncronous - but it's no silver bullet, a poorly though out node.js server might scale just as badly as any other poorly thought out server in any other language. You still need to understand what's going on and not code yourself into the low performance corners.
- stock_toaster 15y ago
  > This is exactly what Zed was pointing out with his intentionally pessimized fibonacci generator. You mean Ted Dziuba [1]? [1]: http://teddziuba.com/2011/10/node-js-is-cancer.html http://teddziuba.com/2011/10/node-js-is-cancer.html
- rpwilcox 15y ago
  Exactly. The fibonacci generator (and subsequent async implementations) are one of the things I looked at when writing the article.
davidhollander 15y ago
>I understand that IO is expensive, but so is performing string interpolation on a string in memory 50 times (for whatever reason). I think you are conflating waiting and working. Both cost time, but they are not the same. Waiting can be avoided (nonblocking), working cannot. Iterating over an array in most languages probably takes .000001 seconds. Reading from a bad connection could take 30+ seconds, or whatever your servers timeout is. Since the 30 seconds is spent _waiting_, it can be avoided within a single thread of execution. On the other hand, if you are spending 30 seconds _working_ to compute prime numbers, this cannot be avoided in a single thread of execution. Blocking vs nonblocking refers to dealing with waits. Does each IO operation do the waiting, or does each IO operation return immediately and we pool waits into a single poll() (etc.) call? Parallelization refers to dealing with work. In the rare instance one needs to generate huge fibonacci sequences, one can do so in a seperate fork\process\thread\cpu core\machine, and report results back to the event loop via pipe\socket\shared memory. [1] Multiplexing waits in a single thread makes sense, multiplexing work in a single thread such as array iteration does not: on the contrary it sounds like a good way to increase the overhead of iteration and generate more work rather than reduce it. It's confusing because some tools such as threads can be used to address both the problem of parallel work and the problem of parallel waits. [1] edit: quick example in lua: -- In fork, write to the blocking end of a pipe local r = formfork(function(w) local work = fibonacci(10^10) w:write(work) w:close() end) -- In event loop, read from the nonblocking end of a pipe readline(r , 1024, function(r, line) print('got work', line) close(r) end)

glenjamin 15y ago

The main issue preventing widespread use of underscore in nodejs land is that it isn't set up to accept node-style callbacks, hence you can't mix it with asynchronous code. Eg. Mapping an async function on each member of the array, the async library provides similar functions to underscore while providing callbacks for completion. The same applies to using normal JS constructors, it's tricky to get a completion callback - I've been doing stuff like this: Klass.create = function(args, callback) { var instance = new Klass(args); instance.initialise(callback); }

davidhollander 15y ago
> I've been doing stuff like this Why use callbacks over 'return' for anything other than IO and waiting? If there's not two out of sync parties, there's no need to tell another person to call you back, and no advantage to performing an extra function call. re: yummyfajitas: that does not use a callback, it returns a list. And if retrieveURL is nonblocking, then iteration is nonblocking.
- yummyfajitas 15y ago
  I believe the use case is stuff like this: _.map(listOfURLs, retrieveURL)
- rpwilcox 15y ago
  > And if retrieveURL is nonblocking, then iteration is nonblocking. Something I mention in the article/paper/whatever is using operations that are technically blocking to call functions that happen asynchronously. So there are certainly uses for sync iteration tools (like _.each()) I think it depends on where you put the async assumptions. Do you know/assume that the function itself takes care of the async aspects, or do you know/assume that the caller must take care to set up that structure? If the answer is the former, then use whatever structure you want, because the async is taken care of for you. If the answer is the latter, perhaps you need a iteration function that works asynchronously.
  - davidhollander 15y ago