5 ms·

Is there a SIMPLE stream processing library/framework? Something like spark or kafka streaming but that doesn't depend on hundreds of megabytes of java stuffs?

Is there a SIMPLE stream processing library/framework? Something like spark or kafka streaming but that doesn't depend on hundreds of megabytes of java stuffs? I just want to do basic windowing/counting against data streams.. I don't want to be part of a gigantic java ecosystem :-(

papercruncher 10y ago
If you can tolerate some latency on writes, AWS Kinesis is perfectly serviceable. You can write your own "stream" processing logic in a number of languages and the provided KCL/SDK from Amazon does most of plumbing. Eventually, you will probably want to move to something more sophisticated though.
- alexatkeplar 10y ago
  Exactly this - here's some simple windowing/counting in JavaScript with Kinesis, Lambda and DynamoDB, no JVM in sight :-) https://github.com/snowplow/aws-lambda-nodejs-example-project https://github.com/snowplow/aws-lambda-nodejs-example-projec...
- justinsaccount 10y ago
  a hosted service is a non-starter for my use cases :-/
hokkos 10y ago
Why not just use a RX library ? http://reactivex.io/ http://reactivex.io/
- cel1ne 10y ago
  And how do you store the data?
  - marknadal 10y ago
    You could pretty easily batch stream it to S3 for ridiculously cheap. I posted a video of our setup elsewhere in the thread. So overall I'd say this is a pretty good idea, especially when you can do some cool stuff with RxJS.
nathanielc 10y ago
Kapacitor, it has a focus on time series data, and simplicity. https://www.influxdata.com/time-series-platform/kapacitor/ https://www.influxdata.com/time-series-platform/kapacitor/ Disclaimer: I am the author.
wolle 10y ago
Depending on your requirements, a single-node solution like PipelineDB may suffice; it's a PostgreSQL extension that lets you write streaming SQL queries.
- justinsaccount 10y ago
  That is definitely along the lines of what I'm looking for.. the esper stuff was on my radar, but something that works with existing postgresql tools is a huge plus.
- avifreedman 10y ago
  PipelineDB can scale out to multiple nodes (for $).
  - wolle 10y ago
    Is there a reference regarding the number of nodes that can be deployed in the cluster, and are feasible to maintain? From the docs, I gather that "[a]ll DDL/non-stream DML statements are executed in a distributed transaction on all nodes in the cluster and committed via two-phase commit." [1] Doesn't that get in the way of low latency and availability? [1] http://enterprise.pipelinedb.com/docs/two-phase.html#two-phase-commits http://enterprise.pipelinedb.com/docs/two-phase.html#two-pha...
- Fergi 10y ago
  Jeff from PipelineDB here. We are also rolling out a SaaS product based on PipelineDB called Stride (stride.io) that enables this type of computation via an API with little to no overhead.
reubano 10y ago
I wrote riko [1] with exactly this use case in mind. Its a simple python stream processing engine. [1] https://github.com/nerevu/riko https://github.com/nerevu/riko Edit: HN submission https://news.ycombinator.com/item?id=12136011 https://news.ycombinator.com/item?id=12136011
doublerebel 10y ago
I've done a huge amount of work on Godot2 [1] (nodejs Riemann port) this past couple weeks, updating it to modern Streams2 syntax in Node. Really lightweight and deployable on all sorts of platforms. Includes many windowing features. I also updated the dashboard to match [2], including Riemann's query language [3]. I haven't had a chance to update the docs yet, but the tests have a ton of examples [4]. Reactors are just Transform streams now so it's possible to use any duplex/transform stream in the pipeline. Docs coming ASAP. [1]: https://github.com/nextorigin/godot2 https://github.com/nextorigin/godot2 [2]: https://github.com/nextorigin/godot2-dash https://github.com/nextorigin/godot2-dash [3]: https://github.com/nextorigin/riemann-query-parser https://github.com/nextorigin/riemann-query-parser [4]: https://github.com/nextorigin/godot2/tree/nextorigin/test/reactor https://github.com/nextorigin/godot2/tree/nextorigin/test/re...
capkutay 10y ago
You can try Striim (shameless plug, I work there). You build all your stream processing pipelines in a simple, SQL-like language in a nice web-based UI. It also has built in adapters to convert log data, JSON, XML, database change capture, and other complex data formats into in-memory streams. The product also comes with dashboards so you can do ad-hoc analytics and data visualization on your streams and windows. It has support for stream replay built on top of Kafka-backed persistent streams. It has a commercial license, but we're pretty flexible with startups http://www.striim.com/download-striim/ http://www.striim.com/download-striim/
gfodor 10y ago
It still requires Java and Kafka, but Kafka Streams is a library that lets you build stream processing jobs without a centralized cluster if you have Kafka running. Just embed the jar and run your class on as many machines as you like and they will self-organize and balance. Worth checking out.
- justinsaccount 10y ago
  yeah.. The kafka streams library does look nice, especially how they don't care how you run the thing. It's a bit harder to dive into java though. The thing I have a use case for stream processing is basically glorified wordcount - just over a realtime 7 day window. The data processing code is the easy part, the tooling/runtime is the hard part.
  - cel1ne 10y ago
    Regarding java tooling: check out the blog posts "not your father's java" and especially the part about capsule packaging.
nl 10y ago
Spark with Python is pretty good. You just install it, then use a python shell or scripts. You don't really need to get into the Java side at all. I'm not sure the size of the install should be the primary thing you are concerned about.
- justinsaccount 10y ago
  Indeed. I was just playing with the python bindings.. I was able to build a prototype of what I am trying to do in a few minutes. I'd prefer the kafka streams approach of deployment, but I suppose for just having it run on a single box this would work.
  - nl 10y ago
    It's actually pretty easy to setup a multi-node setup, if you don't worry about YARN/Mesos and just use the plain Spark cluster thing. I think it's like 1 config entry per node. It doesn't solve the data storage problem, so if you need that you need a way to map the same files to the same path on every node (eg, network drive, RSync). If you are just doing streaming it might not matter though.
    - justinsaccount 10y ago
      My use case is 100% streaming.. I actually implemented what I'm trying to do using some basic python code that runs against archived logs, but I just process the last weeks worth of data from scratch at each iteration. It only takes 30 seconds to run, but this method doesn't work for a 100x larger stream that I want to do the same thing for. The ability to run the data pipeline in realtime but also replay older data against new code using something like kafka would be a huge plus. Really the main thing my code is missing is the sliding window implementation, so I can either port to spark streaming which has all that stuff built in, or just implement my own window code.
      - nl 10y ago
        Spark has sliding window operations: windowedWordCounts = pairs.reduceByKeyAndWindow(lambda x, y: x + y, lambda x, y: x - y, 30, 10) Search for "window operations" on http://spark.apache.org/docs/latest/streaming-programming-guide.html#discretized-streams-dstreams http://spark.apache.org/docs/latest/streaming-programming-gu.... Unless you meant something different? You can replay streams against Spark, too. streamingContext.textFileStream will stream data from files dumped in a directory - to replay them, just dump them there again.
eikenberry 10y ago
I've been keeping an eye on this one for the next time I needed a system of this sort. Looks promising... https://github.com/chrislusf/glow https://github.com/chrislusf/glow
- justinsaccount 10y ago
  glow looks like a nifty mapreduce implemenation, but it doesn't appear to do stream processing.
oellegaard 10y ago
Check out motorway: https://github.com/plecto/motorway/ https://github.com/plecto/motorway/ - its a few hundred lines of Python that nearly does the same as eg storm
di4na 10y ago
Elixir is getting a really nice Flow lib. You can have a look here : https://hexdocs.pm/gen_stage/Experimental.Flow.html https://hexdocs.pm/gen_stage/Experimental.Flow.html

papercruncher 10y ago

If you can tolerate some latency on writes, AWS Kinesis is perfectly serviceable. You can write your own "stream" processing logic in a number of languages and the provided KCL/SDK from Amazon does most of plumbing. Eventually, you will probably want to move to something more sophisticated though.

alexatkeplar 10y ago
Exactly this - here's some simple windowing/counting in JavaScript with Kinesis, Lambda and DynamoDB, no JVM in sight :-) https://github.com/snowplow/aws-lambda-nodejs-example-project https://github.com/snowplow/aws-lambda-nodejs-example-projec...
justinsaccount 10y ago
a hosted service is a non-starter for my use cases :-/

hokkos 10y ago

Why not just use a RX library ? http://reactivex.io/ http://reactivex.io/

cel1ne 10y ago
And how do you store the data?
- marknadal 10y ago
  You could pretty easily batch stream it to S3 for ridiculously cheap. I posted a video of our setup elsewhere in the thread. So overall I'd say this is a pretty good idea, especially when you can do some cool stuff with RxJS.

nathanielc 10y ago

Kapacitor, it has a focus on time series data, and simplicity. https://www.influxdata.com/time-series-platform/kapacitor/ https://www.influxdata.com/time-series-platform/kapacitor/ Disclaimer: I am the author.

wolle 10y ago

Depending on your requirements, a single-node solution like PipelineDB may suffice; it's a PostgreSQL extension that lets you write streaming SQL queries.

justinsaccount 10y ago
That is definitely along the lines of what I'm looking for.. the esper stuff was on my radar, but something that works with existing postgresql tools is a huge plus.
avifreedman 10y ago
PipelineDB can scale out to multiple nodes (for $).
- wolle 10y ago
  Is there a reference regarding the number of nodes that can be deployed in the cluster, and are feasible to maintain? From the docs, I gather that "[a]ll DDL/non-stream DML statements are executed in a distributed transaction on all nodes in the cluster and committed via two-phase commit." [1] Doesn't that get in the way of low latency and availability? [1] http://enterprise.pipelinedb.com/docs/two-phase.html#two-phase-commits http://enterprise.pipelinedb.com/docs/two-phase.html#two-pha...
Fergi 10y ago
Jeff from PipelineDB here. We are also rolling out a SaaS product based on PipelineDB called Stride (stride.io) that enables this type of computation via an API with little to no overhead.

reubano 10y ago

I wrote riko [1] with exactly this use case in mind. Its a simple python stream processing engine. [1] https://github.com/nerevu/riko https://github.com/nerevu/riko Edit: HN submission https://news.ycombinator.com/item?id=12136011 https://news.ycombinator.com/item?id=12136011

doublerebel 10y ago

I've done a huge amount of work on Godot2 [1] (nodejs Riemann port) this past couple weeks, updating it to modern Streams2 syntax in Node. Really lightweight and deployable on all sorts of platforms. Includes many windowing features. I also updated the dashboard to match [2], including Riemann's query language [3]. I haven't had a chance to update the docs yet, but the tests have a ton of examples [4]. Reactors are just Transform streams now so it's possible to use any duplex/transform stream in the pipeline. Docs coming ASAP. [1]: https://github.com/nextorigin/godot2 https://github.com/nextorigin/godot2 [2]: https://github.com/nextorigin/godot2-dash https://github.com/nextorigin/godot2-dash [3]: https://github.com/nextorigin/riemann-query-parser https://github.com/nextorigin/riemann-query-parser [4]: https://github.com/nextorigin/godot2/tree/nextorigin/test/reactor https://github.com/nextorigin/godot2/tree/nextorigin/test/re...

capkutay 10y ago

You can try Striim (shameless plug, I work there). You build all your stream processing pipelines in a simple, SQL-like language in a nice web-based UI. It also has built in adapters to convert log data, JSON, XML, database change capture, and other complex data formats into in-memory streams. The product also comes with dashboards so you can do ad-hoc analytics and data visualization on your streams and windows. It has support for stream replay built on top of Kafka-backed persistent streams. It has a commercial license, but we're pretty flexible with startups http://www.striim.com/download-striim/ http://www.striim.com/download-striim/

gfodor 10y ago

It still requires Java and Kafka, but Kafka Streams is a library that lets you build stream processing jobs without a centralized cluster if you have Kafka running. Just embed the jar and run your class on as many machines as you like and they will self-organize and balance. Worth checking out.

justinsaccount 10y ago
yeah.. The kafka streams library does look nice, especially how they don't care how you run the thing. It's a bit harder to dive into java though. The thing I have a use case for stream processing is basically glorified wordcount - just over a realtime 7 day window. The data processing code is the easy part, the tooling/runtime is the hard part.
- cel1ne 10y ago
  Regarding java tooling: check out the blog posts "not your father's java" and especially the part about capsule packaging.

nl 10y ago

Spark with Python is pretty good. You just install it, then use a python shell or scripts. You don't really need to get into the Java side at all. I'm not sure the size of the install should be the primary thing you are concerned about.

justinsaccount 10y ago
Indeed. I was just playing with the python bindings.. I was able to build a prototype of what I am trying to do in a few minutes. I'd prefer the kafka streams approach of deployment, but I suppose for just having it run on a single box this would work.
- nl 10y ago
  It's actually pretty easy to setup a multi-node setup, if you don't worry about YARN/Mesos and just use the plain Spark cluster thing. I think it's like 1 config entry per node. It doesn't solve the data storage problem, so if you need that you need a way to map the same files to the same path on every node (eg, network drive, RSync). If you are just doing streaming it might not matter though.
  - justinsaccount 10y ago
    My use case is 100% streaming.. I actually implemented what I'm trying to do using some basic python code that runs against archived logs, but I just process the last weeks worth of data from scratch at each iteration. It only takes 30 seconds to run, but this method doesn't work for a 100x larger stream that I want to do the same thing for. The ability to run the data pipeline in realtime but also replay older data against new code using something like kafka would be a huge plus. Really the main thing my code is missing is the sliding window implementation, so I can either port to spark streaming which has all that stuff built in, or just implement my own window code.
    - nl 10y ago
      Spark has sliding window operations: windowedWordCounts = pairs.reduceByKeyAndWindow(lambda x, y: x + y, lambda x, y: x - y, 30, 10) Search for "window operations" on http://spark.apache.org/docs/latest/streaming-programming-guide.html#discretized-streams-dstreams http://spark.apache.org/docs/latest/streaming-programming-gu.... Unless you meant something different? You can replay streams against Spark, too. streamingContext.textFileStream will stream data from files dumped in a directory - to replay them, just dump them there again.

eikenberry 10y ago

I've been keeping an eye on this one for the next time I needed a system of this sort. Looks promising... https://github.com/chrislusf/glow https://github.com/chrislusf/glow

justinsaccount 10y ago
glow looks like a nifty mapreduce implemenation, but it doesn't appear to do stream processing.

oellegaard 10y ago

Check out motorway: https://github.com/plecto/motorway/ https://github.com/plecto/motorway/ - its a few hundred lines of Python that nearly does the same as eg storm

di4na 10y ago

Elixir is getting a really nice Flow lib. You can have a look here : https://hexdocs.pm/gen_stage/Experimental.Flow.html https://hexdocs.pm/gen_stage/Experimental.Flow.html