5 ms·

Node.js stream handbook

113 points by hoggle 12y ago

dpweb 12y ago
This is a great write-up on streams. For curiosity sake, I'd be interested to know the overhead of var stream = fs.createReadStream(__dirname + '/data.txt'); stream.pipe(res); vs. fs.readFile('./data.txt) and res(). and if streams become a better and better option as response sizes get larger and larger. The streams syntax is nicer looking in any case.
- sehrope 12y ago
  > ... and if streams become a better and better option as response sizes get larger and larger. The streams syntax is nicer looking in any case. fs.readFile(...) reads the entire contents of the file into memory. For small files that may be acceptable but if you're dealing with anything that could be large it's not going to be very pleasant or even work properly. I just tried it out on my laptop for a 100MB and a 1GB file. For a 100MB file using streams is about 25% slower. However the sync method failed for the 1GB file: Style 100MB 1GB ===== ===== === fs.readFileSync .176s <failed> streams/pipe .234s 1.25s
  - hoggle 12y ago
    By adapting the highWaterMark option one could probably tweak it even further. Default buffer size seems to be 64k: https://github.com/joyent/node/blob/912b5e05811fd24f09f9d65200a1561a4482f166/lib/fs.js#L1468 https://github.com/joyent/node/blob/912b5e05811fd24f09f9d652... Stream buffering: http://www.nodejs.org/api/stream.html#stream_buffering http://www.nodejs.org/api/stream.html#stream_buffering
- domenicd 12y ago
  In addition to the other responses, the pipe version will get initial chunks of data across the HTTP connection sooner, so that the user starts seeing data on their screen.
hoggle 12y ago
Also by the magnificent substack, learning by doing done right: the node.js stream adventure! Install via: npm install -g stream-adventure Now simply go forth and start your adventure by typing: stream-adventure Have fun and don't forget to thank the man: https://github.com/substack/stream-adventure https://github.com/substack/stream-adventure https://twitter.com/substack https://twitter.com/substack
shortstuffsushi 12y ago
Not hoping to start any sort of flame war, but could someone try to explain this difference between streams and promises? The conceptual difference doesn't seem completely clear to me.
- todd3834 12y ago
  Promises hep deal with asynchronous functions by providing a convenient way to pass success/fail methods in a way some people prefer more than classic callbacks. Streams are a totally different use case. Streams are used when you want to process data one part at a time without having to have all of the data in memory at once. *edit: typo
  - muraiki 12y ago
    Promises are actually about trust, not syntax. With a callback you have to trust that the function that will invoke your callback (which might not be something you wrote) will only ever call it once, that it will pass through errors, that it won't call both your success and failure callbacks, etc. With a promise (or rather, following the promise specification) you instead get back an object that you choose how to handle. That object is either pending or else an immutable success or failure result. Either success or failure will be called (not both), and whichever result is called can only be called once. As such, promises allow you to avoid inversion of control and to safely interact with potentially untrusted code. This great series of articles helped me understand this: http://blog.getify.com/promises-part-2/ http://blog.getify.com/promises-part-2/
- Touche 12y ago
  With streams you don't wait for them to end. They may never end, in fact.
  - shortstuffsushi 12y ago
    Which is unlike Promises, which guarantee a single "end," but also don't care when that ends (could have already happened, could happen eventually). Seems similar in nature to a socket connection.
fibo 12y ago
Can't believe it was not in HN yet
- alphonse23 12y ago
  it was, it's been posted several times -- at least I know for sure in comments. I was about to ask, "why are so many links getting re-posted in hackernews?"
yudong 12y ago
Will it work with LIVE video streaming? For example, can you use ffmpeg to pull live video via rtsp, re-encode and pipe to nodejs for it to stream down to browser and to be consumed by html - with or without a plugin? Any working code sample?
- gabipurcaru 12y ago
  Yup, it does. I do have a quick working example, with a plain multipart form :) You can check it out here: https://github.com/gabipurcaru/peggy https://github.com/gabipurcaru/peggy While this is not exactly what you are talking about, I think it demonstrates that node.js streams can do that too.

113 points by hoggle 12y ago

dpweb 12y ago

This is a great write-up on streams. For curiosity sake, I'd be interested to know the overhead of var stream = fs.createReadStream(__dirname + '/data.txt'); stream.pipe(res); vs. fs.readFile('./data.txt) and res(). and if streams become a better and better option as response sizes get larger and larger. The streams syntax is nicer looking in any case.

sehrope 12y ago
> ... and if streams become a better and better option as response sizes get larger and larger. The streams syntax is nicer looking in any case. fs.readFile(...) reads the entire contents of the file into memory. For small files that may be acceptable but if you're dealing with anything that could be large it's not going to be very pleasant or even work properly. I just tried it out on my laptop for a 100MB and a 1GB file. For a 100MB file using streams is about 25% slower. However the sync method failed for the 1GB file: Style 100MB 1GB ===== ===== === fs.readFileSync .176s <failed> streams/pipe .234s 1.25s
- hoggle 12y ago
  By adapting the highWaterMark option one could probably tweak it even further. Default buffer size seems to be 64k: https://github.com/joyent/node/blob/912b5e05811fd24f09f9d65200a1561a4482f166/lib/fs.js#L1468 https://github.com/joyent/node/blob/912b5e05811fd24f09f9d652... Stream buffering: http://www.nodejs.org/api/stream.html#stream_buffering http://www.nodejs.org/api/stream.html#stream_buffering
domenicd 12y ago
In addition to the other responses, the pipe version will get initial chunks of data across the HTTP connection sooner, so that the user starts seeing data on their screen.

hoggle 12y ago

Also by the magnificent substack, learning by doing done right: the node.js stream adventure! Install via: npm install -g stream-adventure Now simply go forth and start your adventure by typing: stream-adventure Have fun and don't forget to thank the man: https://github.com/substack/stream-adventure https://github.com/substack/stream-adventure https://twitter.com/substack https://twitter.com/substack

shortstuffsushi 12y ago

Not hoping to start any sort of flame war, but could someone try to explain this difference between streams and promises? The conceptual difference doesn't seem completely clear to me.

todd3834 12y ago
Promises hep deal with asynchronous functions by providing a convenient way to pass success/fail methods in a way some people prefer more than classic callbacks. Streams are a totally different use case. Streams are used when you want to process data one part at a time without having to have all of the data in memory at once. *edit: typo
- muraiki 12y ago
  Promises are actually about trust, not syntax. With a callback you have to trust that the function that will invoke your callback (which might not be something you wrote) will only ever call it once, that it will pass through errors, that it won't call both your success and failure callbacks, etc. With a promise (or rather, following the promise specification) you instead get back an object that you choose how to handle. That object is either pending or else an immutable success or failure result. Either success or failure will be called (not both), and whichever result is called can only be called once. As such, promises allow you to avoid inversion of control and to safely interact with potentially untrusted code. This great series of articles helped me understand this: http://blog.getify.com/promises-part-2/ http://blog.getify.com/promises-part-2/
Touche 12y ago
With streams you don't wait for them to end. They may never end, in fact.
- shortstuffsushi 12y ago
  Which is unlike Promises, which guarantee a single "end," but also don't care when that ends (could have already happened, could happen eventually). Seems similar in nature to a socket connection.

fibo 12y ago

Can't believe it was not in HN yet

alphonse23 12y ago
it was, it's been posted several times -- at least I know for sure in comments. I was about to ask, "why are so many links getting re-posted in hackernews?"

yudong 12y ago

Will it work with LIVE video streaming? For example, can you use ffmpeg to pull live video via rtsp, re-encode and pipe to nodejs for it to stream down to browser and to be consumed by html - with or without a plugin? Any working code sample?

gabipurcaru 12y ago
Yup, it does. I do have a quick working example, with a plain multipart form :) You can check it out here: https://github.com/gabipurcaru/peggy https://github.com/gabipurcaru/peggy While this is not exactly what you are talking about, I think it demonstrates that node.js streams can do that too.