3 ms·
The new node streams API is nice, but for processing a lot of data like this we've found them unsuitable for a few reasons: 1) objectMode is considered an abom
by rgarcia 13y ago
The new node streams API is nice, but for processing a lot of data like this we've found them unsuitable for a few reasons:
1) objectMode is considered an abomination by most of node-core[1], so if you're putting anything other than a string or a buffer through a stream, you're "doing it wrong" (I disagree, but I also am not a node maintainer).
2) If you want to process data in parallel you're out of luck, since Writable handles writes one at a time. We've created some workarounds but I'm not super happy with them [2].
3) Once you jerry-rig Writable/Transform to run in parallel, you're stuck with one core. We've also created a workaround for this [3].
We've started moving towards go's channels because of all these issues.
[1] https://github.com/joyent/node/pull/4835 https://github.com/joyent/node/pull/4835
[2] https://github.com/Clever/writable-stream-parallel https://github.com/Clever/writable-stream-parallel https://github.com/clever/understream https://github.com/clever/understream
[3] https://github.com/Clever/async-forkqueue https://github.com/Clever/async-forkqueue
- casual_slacker 13y agoAs per questions 2 and 3, besides the naturally single-threaded nature of node, is that having parallel producers requires a locking mechanism, and parallel consumers doesn't really make sense with these types of streams. How do you avoid these issues with go?
- rgarcia 13y agoIn many stream pipelines you don't care about the order of the data. You just have a bunch of unordered input data that you need to transform as fast as possible. Go's channels [1] let you essentially construct loosely-typed streams. The channel's type lets you be as opinionated as you want about what should/shouldn't be passed within the "stream" (interface{}, []byte, string, MyAwesomeType, etc.). If you use buffered channels you have "backpressure". You can use > 1 core by using goroutines. [1] http://golangtutorials.blogspot.com/2011/06/channels-in-go.html http://golangtutorials.blogspot.com/2011/06/channels-in-go.h...
- apaprocki 13y agoI agree re: 1) -- Streams are pretty handy for normalizing raw binary wire protocol (dechunking, decompressing, etc) and ultimately turning the raw data into logical objects akin to structs for each layer in a protocol. I'm not sure why blending binary/objects isn't a more valid use case for lower-level protocol work. Using this paradigm, I found it really easy to make quick work of an existing binary protocol in nice discrete stream layers that could be combined as needed.