6 ms·

Parsing file uploads at 500 mb/s with node.js

74 points by felixge 16y ago

rythie 16y ago
It's a pity that browsers couldn't just tell you the length of the part in the header of each part.
- felixge2 16y ago
  I think the reason multipart works that way, is so you can stream data that you don't know the full length of beforehand. But afaik, that's pretty much never the case with file uploads, unless you are uploading a file that is still growing in size - so yeah, it's annoying : ). ('felixge2' because my other account is in noprocrast mode: )
  - stingraycharles 16y ago
    Think of it the other way too: it allows the HTTP server to start writing the file to disk without having to completely load the file into memory.
    - felixge2 16y ago
      You could still do that if the length was pre-announced, or am I missing something?
      - stingraycharles 16y ago
        Ehrm, yes, you're correct, I wasn't fully awake yet. I apologize.
      - irrelative 16y ago
        Yeah, you could -- I wonder what would happen if the client gets it wrong or is deliberately dishonest? Not trusting the client is a big part of writing an open server and this seems like you would have to trust the client in a big way.
        felixge2 16y ago
        Well, it's not a big problem - you should have a timeout on incoming connections, and node is pretty well-suited for having lots of "hanging" connections (I ran some test with 56k active connections). If a connection is closed, either by a timeout, or EOF, you simply check if the promised content length matches the count of received bytes - if not you should probably discard the whole thing (unless you're specifically supporting broken clients).
jules 16y ago
Writing everything in callback style is not nice. Why aren't they using a language with coroutines?
- felixge2 16y ago
  This presentation has a tiny bit on coroutines: http://nodejs.org/jsconf2010.pdf http://nodejs.org/jsconf2010.pdf > Coroutines complicate the mental model while adding only cheap syntactic pleasures.
  - kevingadd 16y ago
    The statements made about co-routines/cooperative threading and the stated reasoning for removing them makes me wonder whether the decision to remove them was made based on benchmarking and experimentation or just based on taste. In my experience they simplify many forms of asynchronous logic tremendously and can have performance benefits as well, as long as you're not forced to use them for everything. I suppose someone who wants them in node.js can just reimplement them as an extension.
    - felixge2 16y ago
      Well, just as there is the problem of thread-safety, there is also the problem of co-routine safety. Basically callbacks and coroutines don't play well together. But there were a bunch of reasons for the removal, performance and quality of the initial implementation also played into it AFAIK.
      - kevingadd 16y ago
        If you implement coroutines as a state machine they integrate just fine with callback-oriented APIs. But perhaps that was the problem since the PDF mentions stack swapping.
      - jules 16y ago
        > Basically callbacks and coroutines don't play well together. Can you elaborate on this? I thought the opposite: coroutines play very well with callbacks. Callback libraries force you to write programs in continuation passing style. Coroutines let you write asynchronous programs in normal style. Lets assume we have an asynchronous readAsync(callback) function that reads a number and calls the callback with the input when it's ready. Now you're writing your program like this: readAsync(function(x){ readAsync(function(y){ write(x+y) }) }) What you want to write is this: x = read() // may block this coroutine, but *not* the entire program y = read() write(x+y) This can be achieved with something like: function read(){ coro = getCurrentCoroutine() var result callback = function(x){ result = x; coro.resume() } readAsync(callback) yield // stops the current coroutine until it is resumed return result } It seems to me that this is a general way to adapt any callback style library to coroutines.
marketer 16y ago
When parsing boundaries you know the first character is going to be a hyphen(-), and the last character is going to be a newline. Wouldn't it be easier to search for hyphens, and then read until you see a newline, and then compare to the boundary? Typically boundary characters are random printable characters, so you might be doing more work than you need.
- inimino 16y ago
  The rub is "search for hyphens": you have to look at every character until you find a hyphen; the point of Boyer-Moore is that you don't have to even look at most of the characters at all.
  - marketer 16y ago
    Ah, that makes sense, thanks.
pjscott 16y ago
This is nitpicking, but it always bothers me: "mb/s" is an abbreviation for "millibits-per-second". You meant "MB/s", although this wasn't obvious; if you hadn't mentioned maxing out a GigE connection, it could have been interpreted as "Mb/s", which is short for "megabits-per-second". </pedantry>
GrandMasterBirt 16y ago
If I can get a revenue stream of approximately $1000 a month my entire website will be literally a call to a few different web services and some fancy-shmancy CSS files. God I love hacker news! Ok fine there will be some of my code in there, hopefully not for long because soon there will be a tool to solve any other problem as well :P As long as these don't get too pricey.

74 points by felixge 16y ago

rythie 16y ago

It's a pity that browsers couldn't just tell you the length of the part in the header of each part.

felixge2 16y ago
I think the reason multipart works that way, is so you can stream data that you don't know the full length of beforehand. But afaik, that's pretty much never the case with file uploads, unless you are uploading a file that is still growing in size - so yeah, it's annoying : ). ('felixge2' because my other account is in noprocrast mode: )
- stingraycharles 16y ago
  Think of it the other way too: it allows the HTTP server to start writing the file to disk without having to completely load the file into memory.
  - felixge2 16y ago
    You could still do that if the length was pre-announced, or am I missing something?
    - stingraycharles 16y ago
      Ehrm, yes, you're correct, I wasn't fully awake yet. I apologize.
    - irrelative 16y ago
      Yeah, you could -- I wonder what would happen if the client gets it wrong or is deliberately dishonest? Not trusting the client is a big part of writing an open server and this seems like you would have to trust the client in a big way.
      - felixge2 16y ago
        Well, it's not a big problem - you should have a timeout on incoming connections, and node is pretty well-suited for having lots of "hanging" connections (I ran some test with 56k active connections). If a connection is closed, either by a timeout, or EOF, you simply check if the promised content length matches the count of received bytes - if not you should probably discard the whole thing (unless you're specifically supporting broken clients).

jules 16y ago

Writing everything in callback style is not nice. Why aren't they using a language with coroutines?

felixge2 16y ago
This presentation has a tiny bit on coroutines: http://nodejs.org/jsconf2010.pdf http://nodejs.org/jsconf2010.pdf > Coroutines complicate the mental model while adding only cheap syntactic pleasures.
- kevingadd 16y ago
  The statements made about co-routines/cooperative threading and the stated reasoning for removing them makes me wonder whether the decision to remove them was made based on benchmarking and experimentation or just based on taste. In my experience they simplify many forms of asynchronous logic tremendously and can have performance benefits as well, as long as you're not forced to use them for everything. I suppose someone who wants them in node.js can just reimplement them as an extension.
  - felixge2 16y ago
    Well, just as there is the problem of thread-safety, there is also the problem of co-routine safety. Basically callbacks and coroutines don't play well together. But there were a bunch of reasons for the removal, performance and quality of the initial implementation also played into it AFAIK.
    - kevingadd 16y ago
      If you implement coroutines as a state machine they integrate just fine with callback-oriented APIs. But perhaps that was the problem since the PDF mentions stack swapping.
    - jules 16y ago
      > Basically callbacks and coroutines don't play well together. Can you elaborate on this? I thought the opposite: coroutines play very well with callbacks. Callback libraries force you to write programs in continuation passing style. Coroutines let you write asynchronous programs in normal style. Lets assume we have an asynchronous readAsync(callback) function that reads a number and calls the callback with the input when it's ready. Now you're writing your program like this: readAsync(function(x){ readAsync(function(y){ write(x+y) }) }) What you want to write is this: x = read() // may block this coroutine, but *not* the entire program y = read() write(x+y) This can be achieved with something like: function read(){ coro = getCurrentCoroutine() var result callback = function(x){ result = x; coro.resume() } readAsync(callback) yield // stops the current coroutine until it is resumed return result } It seems to me that this is a general way to adapt any callback style library to coroutines.

marketer 16y ago

When parsing boundaries you know the first character is going to be a hyphen(-), and the last character is going to be a newline. Wouldn't it be easier to search for hyphens, and then read until you see a newline, and then compare to the boundary? Typically boundary characters are random printable characters, so you might be doing more work than you need.

inimino 16y ago
The rub is "search for hyphens": you have to look at every character until you find a hyphen; the point of Boyer-Moore is that you don't have to even look at most of the characters at all.
- marketer 16y ago
  Ah, that makes sense, thanks.

pjscott 16y ago

This is nitpicking, but it always bothers me: "mb/s" is an abbreviation for "millibits-per-second". You meant "MB/s", although this wasn't obvious; if you hadn't mentioned maxing out a GigE connection, it could have been interpreted as "Mb/s", which is short for "megabits-per-second". </pedantry>

GrandMasterBirt 16y ago

If I can get a revenue stream of approximately $1000 a month my entire website will be literally a call to a few different web services and some fancy-shmancy CSS files. God I love hacker news! Ok fine there will be some of my code in there, hopefully not for long because soon there will be a tool to solve any other problem as well :P As long as these don't get too pricey.