4 ms·

If you're looking for a stream processing engine more close to Storm, etc. but also simple, check out Motorway: https://github.com/plecto/motorway https://githu

If you're looking for a stream processing engine more close to Storm, etc. but also simple, check out Motorway: https://github.com/plecto/motorway https://github.com/plecto/motorway :-)

reubano 10y ago
Interesting project. I hadn't come across this one yet. One difference that riko has is it's based around functions whereas this library (and practically every stream processing lib I've come across) is based around classes. I personally prefer the functional approach much better. And if you compare the word count examples on the respective readmes [1, 2], you will see riko is much more succinct. But I suppose the verbosity of the other libraries come with benefits like scaling across a cluster of servers. [1] https://github.com/plecto/motorway#word-count-example https://github.com/plecto/motorway#word-count-example [2] https://github.com/nerevu/riko#word-count https://github.com/nerevu/riko#word-count
- greenshackle 10y ago
  I just started writing a functional stream processing library in Python for some of the same reasons. We use somewhat different concepts. I tend to think of streams as infinite, so it didn't occur to me to include something like a reverse pipe operator. I'm a bit surprised, why is filter an operator rather than a processor? I would think filters usually apply per-item, not to a whole stream? I havn't worked on it very much but I'm heading towards push-based, using 0MQ for distribution/parallel processing, and using asyncio, mostly because it plays nicely with 0MQ.
  - reubano 10y ago
    > We use somewhat different concepts. I tend to think of streams as infinite, so it didn't occur to me to include something like a reverse pipe operator. We are in agreement. reverse has a notice that it isn't lazy [1]. I prefer to include pipes that aren't lazy since it can be helpful in some cases (plus the goal is to include all pipes originally in Yahoo! Pipes). The vast majority of pipes work just fine on infinite streams [2]. > I'm a bit surprised, why is filter an operator rather than a processor? I would think filters usually apply per-item, not to a whole stream? Just an implementation detail [3]. I agree it would be better if it were a processor since it could be parallelized. PRs welcome :). > I haven't worked on it very much but I'm heading towards push-based, using 0MQ for distribution/parallel processing, and using asyncio, mostly because it plays nicely with 0MQ. See my previous comments related to this [4, 5]. [1] https://github.com/nerevu/riko/blob/master/riko/modules/reverse.py#L67 https://github.com/nerevu/riko/blob/master/riko/modules/reve... [2] Assuming you're not using the async or parallel mode [3] https://github.com/nerevu/riko/blob/master/riko/modules/filter.py#L141-L155 https://github.com/nerevu/riko/blob/master/riko/modules/filt... [4] https://news.ycombinator.com/item?id=12137591 https://news.ycombinator.com/item?id=12137591 [5] https://news.ycombinator.com/item?id=12137787 https://news.ycombinator.com/item?id=12137787

If you're looking for a stream processing engine more close to Storm, etc. but also simple, check out Motorway: https://github.com/plecto/motorway https://github.com/plecto/motorway :-)

reubano 10y ago

Interesting project. I hadn't come across this one yet. One difference that riko has is it's based around functions whereas this library (and practically every stream processing lib I've come across) is based around classes. I personally prefer the functional approach much better. And if you compare the word count examples on the respective readmes [1, 2], you will see riko is much more succinct. But I suppose the verbosity of the other libraries come with benefits like scaling across a cluster of servers. [1] https://github.com/plecto/motorway#word-count-example https://github.com/plecto/motorway#word-count-example [2] https://github.com/nerevu/riko#word-count https://github.com/nerevu/riko#word-count

greenshackle 10y ago
I just started writing a functional stream processing library in Python for some of the same reasons. We use somewhat different concepts. I tend to think of streams as infinite, so it didn't occur to me to include something like a reverse pipe operator. I'm a bit surprised, why is filter an operator rather than a processor? I would think filters usually apply per-item, not to a whole stream? I havn't worked on it very much but I'm heading towards push-based, using 0MQ for distribution/parallel processing, and using asyncio, mostly because it plays nicely with 0MQ.
- reubano 10y ago
  > We use somewhat different concepts. I tend to think of streams as infinite, so it didn't occur to me to include something like a reverse pipe operator. We are in agreement. reverse has a notice that it isn't lazy [1]. I prefer to include pipes that aren't lazy since it can be helpful in some cases (plus the goal is to include all pipes originally in Yahoo! Pipes). The vast majority of pipes work just fine on infinite streams [2]. > I'm a bit surprised, why is filter an operator rather than a processor? I would think filters usually apply per-item, not to a whole stream? Just an implementation detail [3]. I agree it would be better if it were a processor since it could be parallelized. PRs welcome :). > I haven't worked on it very much but I'm heading towards push-based, using 0MQ for distribution/parallel processing, and using asyncio, mostly because it plays nicely with 0MQ. See my previous comments related to this [4, 5]. [1] https://github.com/nerevu/riko/blob/master/riko/modules/reverse.py#L67 https://github.com/nerevu/riko/blob/master/riko/modules/reve... [2] Assuming you're not using the async or parallel mode [3] https://github.com/nerevu/riko/blob/master/riko/modules/filter.py#L141-L155 https://github.com/nerevu/riko/blob/master/riko/modules/filt... [4] https://news.ycombinator.com/item?id=12137591 https://news.ycombinator.com/item?id=12137591 [5] https://news.ycombinator.com/item?id=12137787 https://news.ycombinator.com/item?id=12137787