4 ms·

Forgive me for being "That Guy" but I really think Javascript is ill-suited for this paradigm! Streams, honestly, are hard to keep straight when the program ge

by ixmatus 13y ago

Forgive me for being "That Guy" but I really think Javascript is ill-suited for this paradigm! Streams, honestly, are hard to keep straight when the program gets big without a stronger type system. IMHO. Some really sharp people have been working on stream computing software in Haskell for a while - Gabriel's Pipes package is a good example of generalized stream computing with strong equational reasoning as its foundation. Maybe if you really want to try and do this in Node you can gain some inspiration from his journey: http://hackage.haskell.org/package/pipes http://hackage.haskell.org/package/pipes

jkrems 13y ago
Streams are being used in node.js already with a lot of success. Both basic byte chunk streams and object streams. As long as components are well behaved and only emit uniform data (everything they emit is of the same type), it's not better or worse than any function call.
- ixmatus 13y ago
  Right, but the burden is on the programmer to ensure those types. If the stream composition is of simple things, like: cat some.txt | sort | uniq > yay.txt Then it isn't a problem - it's obvious and simple, but I think there will be difficulties as the programs get larger and type-level awareness occupies more space in the programmer's brain vs. it being handled by the compiler...
  - jkrems 13y ago
    I didn't say loose/dynamic typing would not put the burden on the programmer to ensure the types. And it definitely has it's problems. I just don't think that this is specific to streams, it's true for every kind of composition.
  - gcb0 13y ago
    interesting example. the fact you know you have to call sort before uniq already shows that even with plain gnu unix util the user must know what he is doing.
aegiso 13y ago
Here's the main enlightenment of becoming a node.js guy: Node follows the unix way. Everything is a stream. It's just Buffers and JS objects flying around. It's really stupid, and sometimes it's nasty. This isn't helped by Javascript's warts. But there's an enormous upside to this: following the stupid Unix way means that no matter what you need to do with your data, there's an npm module for it. Just .pipe() your stream in and your code is done. This is amazing. And it's possible only because of how bare-bones and loose the Buffer stream API is. Strong typing has its place, but it would ruin node's biggest selling point. It's hard to realize this without trying it.
- hamburglar 13y ago
  Yes, and "the unix way" isn't just "everything's a stream", but rather "everything's a text stream." In that regard, node's version of "everything's a stream" is actually a half-step up in abstraction.
  - ixmatus 13y ago
    Isn't it more accurately a byte stream (I don't know, which is why I'm asking)?
    - jkrems 13y ago
      The default build-in streams found in the stdlib are byte streams (byte chunk streams, to be more exact). But user-level there are object streams as well, those are also mentioned in the docs: http://nodejs.org/api/stream.html#stream_object_mode http://nodejs.org/api/stream.html#stream_object_mode - so they are kind-of official.
  - deleted 13y ago
    [deleted]
- ixmatus 13y ago
  I don't know node very well and only Javascript minimally but I understand what streaming I/O is and have used generators / iteratees for a long time in many languages. I understand the benefits and that's why Pipes in HS is such an exciting thing because it gives us a formally reasoned and general set of stream computing tools - you can compute anything with type-level guarantees. It's just as flexible and general as, say, Unix pipes but better because there are guarantees of the library's tooling and there are guarantees of the programs you produce! You can't say that in Node / JS, Python, Ruby, etc... JS's lack of strong typing limits your ability to reason about streams (a lot more than just streams, too) and further limits your ability to write performant stream computing software. Pipes, in Haskell, give you the big three: Effects (I/O), Composition (function composition with fusion), and Streams (generators and iteratees); because of the type system Haskell (and some nudges here and there by the library author) can fuse and optimize that code to a ridiculous degree in addition to all of the other nice guarantees you get from the type system (separate of I/O from pure code, etc...) I personally don't think dynamic typing is a selling point, ever - I write software faster and with fewer bugs in Haskell than I ever have before in Python, Ruby, Erlang, or Scheme. But that's a totally different topic and I don't want to derail this one. Don't misunderstand me as being aggressive, please. I fully respect what people decide to like and work on, I'm just trying to expand the awareness that there are tools in existence that do it better.
zmanian 13y ago
I'm a Haskeller but I'll defend Node a bit. The absolute most compelling thing about Node is how it has hacked organizational dynamics in large companies. Walmart and PayPal basically used it to completely liberate their frontend groups from their backend systems using facade system with huge improvements on customer systems. What is sad is that all the other high concurrency systems are going to end up implementing much of the GHC runtime without the reliability of Haskell...

by ixmatus 13y ago

jkrems 13y ago

Streams are being used in node.js already with a lot of success. Both basic byte chunk streams and object streams. As long as components are well behaved and only emit uniform data (everything they emit is of the same type), it's not better or worse than any function call.

ixmatus 13y ago
Right, but the burden is on the programmer to ensure those types. If the stream composition is of simple things, like: cat some.txt | sort | uniq > yay.txt Then it isn't a problem - it's obvious and simple, but I think there will be difficulties as the programs get larger and type-level awareness occupies more space in the programmer's brain vs. it being handled by the compiler...
- jkrems 13y ago
  I didn't say loose/dynamic typing would not put the burden on the programmer to ensure the types. And it definitely has it's problems. I just don't think that this is specific to streams, it's true for every kind of composition.
- gcb0 13y ago
  interesting example. the fact you know you have to call sort before uniq already shows that even with plain gnu unix util the user must know what he is doing.

aegiso 13y ago

Here's the main enlightenment of becoming a node.js guy: Node follows the unix way. Everything is a stream. It's just Buffers and JS objects flying around. It's really stupid, and sometimes it's nasty. This isn't helped by Javascript's warts. But there's an enormous upside to this: following the stupid Unix way means that no matter what you need to do with your data, there's an npm module for it. Just .pipe() your stream in and your code is done. This is amazing. And it's possible only because of how bare-bones and loose the Buffer stream API is. Strong typing has its place, but it would ruin node's biggest selling point. It's hard to realize this without trying it.

hamburglar 13y ago
Yes, and "the unix way" isn't just "everything's a stream", but rather "everything's a text stream." In that regard, node's version of "everything's a stream" is actually a half-step up in abstraction.
- ixmatus 13y ago
  Isn't it more accurately a byte stream (I don't know, which is why I'm asking)?
  - jkrems 13y ago
    The default build-in streams found in the stdlib are byte streams (byte chunk streams, to be more exact). But user-level there are object streams as well, those are also mentioned in the docs: http://nodejs.org/api/stream.html#stream_object_mode http://nodejs.org/api/stream.html#stream_object_mode - so they are kind-of official.
- deleted 13y ago
  [deleted]
ixmatus 13y ago
I don't know node very well and only Javascript minimally but I understand what streaming I/O is and have used generators / iteratees for a long time in many languages. I understand the benefits and that's why Pipes in HS is such an exciting thing because it gives us a formally reasoned and general set of stream computing tools - you can compute anything with type-level guarantees. It's just as flexible and general as, say, Unix pipes but better because there are guarantees of the library's tooling and there are guarantees of the programs you produce! You can't say that in Node / JS, Python, Ruby, etc... JS's lack of strong typing limits your ability to reason about streams (a lot more than just streams, too) and further limits your ability to write performant stream computing software. Pipes, in Haskell, give you the big three: Effects (I/O), Composition (function composition with fusion), and Streams (generators and iteratees); because of the type system Haskell (and some nudges here and there by the library author) can fuse and optimize that code to a ridiculous degree in addition to all of the other nice guarantees you get from the type system (separate of I/O from pure code, etc...) I personally don't think dynamic typing is a selling point, ever - I write software faster and with fewer bugs in Haskell than I ever have before in Python, Ruby, Erlang, or Scheme. But that's a totally different topic and I don't want to derail this one. Don't misunderstand me as being aggressive, please. I fully respect what people decide to like and work on, I'm just trying to expand the awareness that there are tools in existence that do it better.

zmanian 13y ago

I'm a Haskeller but I'll defend Node a bit. The absolute most compelling thing about Node is how it has hacked organizational dynamics in large companies. Walmart and PayPal basically used it to completely liberate their frontend groups from their backend systems using facade system with huge improvements on customer systems. What is sad is that all the other high concurrency systems are going to end up implementing much of the GHC runtime without the reliability of Haskell...