i do node all day every day, but i've actually found it sub-optimal for shell scripts / one-off jobs. For me these tend to take the form of import scripts or ba

i do node all day every day, but i've actually found it sub-optimal for shell scripts / one-off jobs. For me these tend to take the form of import scripts or batch files that have to process various different files and stick it in a database. There's a lot of conceptual overhead that comes with asynchronous IO for these kinds of tasks. However, some of the things that node allows you to do is kind of spectacular too. On a recent project I had to parse data from a set of csv files, while pre-processing the data and pushing it into a Couch database. At the same time I needed to generate a sqlite database with spatially aware geographic indexes using parts of the data, so we could build maps from the data. I ended up having a situation where the data was streaming into the couch database waaaay faster than the sqlite db could write the records, which resulted in the process growing in memory bogging the entire application down (the csv files could be uploaded via a form as well). The solution ended up being that because my csv parser was building on the stream classes in node, i could literally pause/resume the stream based on the state of the buffer(s). This is exactly the same technique you would use when building an http proxy or limiting the transfer rate of file uploads. This is some pretty low level stuff to have to get into for a simple import script, and if I were to do it in something other than JS I would probably have used temporary files or separate processes or whatever. Debugging and testing this kind of problem is not really straightforward either. As always YMMV, but I have become more appreciative of plain bash scripting as of late. edit: formatting