5 ms·

Node.js Stream Playground

130 points by jeresig 13y ago

jedireza 13y ago
Wow. nodestreams.com is great. It's really helpful to see the results of different chained combinations instantly. Thanks for making this John.
marknadal 13y ago
Anybody know how to dynamically zip things on the fly? Like if you want to generate a download for a user that has files and folders... how can you do this fast and in one go? Currently I have to literally write to disk, then zip it, and only then can I finally send a response. Any help?
- jeresig 13y ago
  Absolutely! Poking around I found one module that could do it - I'm sure there are more (and better ones, too): https://github.com/wellawaretech/node-zipstream https://github.com/wellawaretech/node-zipstream In the example there you would .pipe() out to an HTTP response rather than a file, presumably.
  - marknadal 13y ago
    Sweet, thanks. I think I had found this a while ago but lost it, looks great! ...holy smokes, John Resig just replied to me. Thanks so much for jQuery, it has changed my life and is what made me become a programmer. (I now do client work for 20<20Thiel Fellows, and my startup has been in WSJ - none of which would have been the case if I hadn't started programming with jQuery!)
  - maxogden 13y ago
    for an example of doing this with tar instead of zip check out https://npmjs.org/package/dir-tar-stream https://npmjs.org/package/dir-tar-stream
- contrahax 13y ago
  node-archiver is the best ive used so far... imo
k3n 13y ago
My only complaint about Resig's blogging is that he doesn't do it enough! It's rare that I read one of his posts and don't come away having learned a lot. Not as much to learn today, but with the really neat tool I can certainly learn from it, so I can't complain.
ilteris 13y ago
another useful resource for getting familiar with streams: https://github.com/whatwg/streams https://github.com/whatwg/streams
ilaksh 13y ago
Great tool. However, since this is an extremely obvious example, I have to attempt once again to point out that textual programming languages are far from ideal representations in many problem domains. The inability or refusal of programmers to accept visual and interactive tools is the main thing holding back software development. I think the main reasons are related to false outdated beliefs about the nature of programming and a fear of being labeled as a novice, since interactive graphical tools are always categorized as being for novices. http://stackoverflow.com/questions/461796/dataflow-programming-languages http://stackoverflow.com/questions/461796/dataflow-programmi... http://noflojs.org/ http://noflojs.org/
- lambda 13y ago
  The problem with every visual programming language that I've ever tried is not that it's aimed at novices and I'm afraid of being labelled that way. It's that there's a whole wealth of tools for dealing with text based programming that just can't be used with visual programming languages without a huge amount of effort (if at all). Visual programming languages don't work well with revision control; they usually don't have diffing and merging tools. They don't work well with grep. They tend to have a fairly low information density on the screen. When looking at a simple program that all fits on a single screen, that's fine. But when working with an actually real world code base, when I don't understand exactly what it's doing, I need to be able to see as much on screen as possible at once to keep it all in context, while browsing through to figure out how all of the pieces link together. All of the empty space taken up in a visual language makes it a lot harder to get as much information on the screen at a time. And the programs can frequently be quite hard to follow. Tracing all of the wires between different nodes, figuring out what's stacked on what. There's a reason that despite the very obvious circuit diagrams you could represent electronics with, hardware engineers for integrated circuits don't use a visual tool for laying out their hardware, they use Verilog or VHDL, only compiling it down to the actual netlist and laying it out at the very end. I have never seen a visual programming language that I would say could scale to anything beyond a few basic demos, which could just as easily be accomplished in a page of code.
  - austinz 13y ago
gobengo 13y ago
Those who enjoyed this and streams2+ in general may find useful this library we've made to use objectMode streams in the browser. http://github.com/Livefyre/stream http://github.com/Livefyre/stream There are quite a few tests if you want to get a feel for how streams work: https://github.com/Livefyre/stream/blob/master/tests/spec/readable.js https://github.com/Livefyre/stream/blob/master/tests/spec/re... For a concrete implementation, check out our SDK. We use streams to abstract away the details of paging out Content from our APIs. There is an example in the README. https://github.com/Livefyre/streamhub-sdk https://github.com/Livefyre/streamhub-sdk These streams power the bottom 5 apps you see here: http://apps.livefyre.com/ http://apps.livefyre.com/
rashthedude 13y ago
I started looking into Streams 2 days ago and boom this headline greets me on hacker news today. Thanks a bunch Mr Resig.

130 points by jeresig 13y ago

Wow. nodestreams.com is great. It's really helpful to see the results of different chained combinations instantly. Thanks for making this John.

marknadal 13y ago

Anybody know how to dynamically zip things on the fly? Like if you want to generate a download for a user that has files and folders... how can you do this fast and in one go? Currently I have to literally write to disk, then zip it, and only then can I finally send a response. Any help?

jeresig 13y ago
Absolutely! Poking around I found one module that could do it - I'm sure there are more (and better ones, too): https://github.com/wellawaretech/node-zipstream https://github.com/wellawaretech/node-zipstream In the example there you would .pipe() out to an HTTP response rather than a file, presumably.
- marknadal 13y ago
  Sweet, thanks. I think I had found this a while ago but lost it, looks great! ...holy smokes, John Resig just replied to me. Thanks so much for jQuery, it has changed my life and is what made me become a programmer. (I now do client work for 20<20Thiel Fellows, and my startup has been in WSJ - none of which would have been the case if I hadn't started programming with jQuery!)
- maxogden 13y ago
  for an example of doing this with tar instead of zip check out https://npmjs.org/package/dir-tar-stream https://npmjs.org/package/dir-tar-stream
contrahax 13y ago
node-archiver is the best ive used so far... imo

k3n 13y ago

My only complaint about Resig's blogging is that he doesn't do it enough! It's rare that I read one of his posts and don't come away having learned a lot. Not as much to learn today, but with the really neat tool I can certainly learn from it, so I can't complain.

ilteris 13y ago

another useful resource for getting familiar with streams: https://github.com/whatwg/streams https://github.com/whatwg/streams

ilaksh 13y ago

Great tool. However, since this is an extremely obvious example, I have to attempt once again to point out that textual programming languages are far from ideal representations in many problem domains. The inability or refusal of programmers to accept visual and interactive tools is the main thing holding back software development. I think the main reasons are related to false outdated beliefs about the nature of programming and a fear of being labeled as a novice, since interactive graphical tools are always categorized as being for novices. http://stackoverflow.com/questions/461796/dataflow-programming-languages http://stackoverflow.com/questions/461796/dataflow-programmi... http://noflojs.org/ http://noflojs.org/

lambda 13y ago
The problem with every visual programming language that I've ever tried is not that it's aimed at novices and I'm afraid of being labelled that way. It's that there's a whole wealth of tools for dealing with text based programming that just can't be used with visual programming languages without a huge amount of effort (if at all). Visual programming languages don't work well with revision control; they usually don't have diffing and merging tools. They don't work well with grep. They tend to have a fairly low information density on the screen. When looking at a simple program that all fits on a single screen, that's fine. But when working with an actually real world code base, when I don't understand exactly what it's doing, I need to be able to see as much on screen as possible at once to keep it all in context, while browsing through to figure out how all of the pieces link together. All of the empty space taken up in a visual language makes it a lot harder to get as much information on the screen at a time. And the programs can frequently be quite hard to follow. Tracing all of the wires between different nodes, figuring out what's stacked on what. There's a reason that despite the very obvious circuit diagrams you could represent electronics with, hardware engineers for integrated circuits don't use a visual tool for laying out their hardware, they use Verilog or VHDL, only compiling it down to the actual netlist and laying it out at the very end. I have never seen a visual programming language that I would say could scale to anything beyond a few basic demos, which could just as easily be accomplished in a page of code.
- austinz 13y ago

gobengo 13y ago

Those who enjoyed this and streams2+ in general may find useful this library we've made to use objectMode streams in the browser. http://github.com/Livefyre/stream http://github.com/Livefyre/stream There are quite a few tests if you want to get a feel for how streams work: https://github.com/Livefyre/stream/blob/master/tests/spec/readable.js https://github.com/Livefyre/stream/blob/master/tests/spec/re... For a concrete implementation, check out our SDK. We use streams to abstract away the details of paging out Content from our APIs. There is an example in the README. https://github.com/Livefyre/streamhub-sdk https://github.com/Livefyre/streamhub-sdk These streams power the bottom 5 apps you see here: http://apps.livefyre.com/ http://apps.livefyre.com/

rashthedude 13y ago

I started looking into Streams 2 days ago and boom this headline greets me on hacker news today. Thanks a bunch Mr Resig.