3 ms·
Hey everyone, author here. We use Benthos as a general stream swiss army knife for all the dull tasks, but you can also use it as a framework for writing your o
by jeffail 8y ago
Hey everyone, author here. We use Benthos as a general stream swiss army knife for all the dull tasks, but you can also use it as a framework for writing your own stream processors in Go.
It doesn't provide any tools for idempotent calculations yet, just at-least-once delivery guarantees. But when you use it as a framework you get the benefit of all the configurable processors in your new service.
- Mikushi 8y agoNot the OP or author but for anyone interested about it this video expends on the implementation of Benthos https://www.youtube.com/watch?v=NM7X4PIUQB0 https://www.youtube.com/watch?v=NM7X4PIUQB0 As someone who is getting into Golang quite a bit I found it very interesting to peer into the thought process that led to Benthos and how it was/is implemented.
- techno_modus 8y agoAfter reading the section about processors it is not clear whether it can do stateful computations, for example, running sum or moving average. Can I define sliding windows? Can I join two or more streams?
- jeffail 8y agoAll Benthos processors are stateless, except for a few such as batch, combine, etc, which do basic tasks such as building up batches from consecutive messages. It's possible to create sliding window processors on top of Benthos, but there aren't any general tools for doing that yet.
- asavinov 8y agoFor stateful processing you need something like: * https://kafka.apache.org/documentation/streams/ https://kafka.apache.org/documentation/streams/ - Kafka Streams (and its KTable) * https://flink.apache.org/ https://flink.apache.org/ - Flink * https://spark.apache.org/streaming/ https://spark.apache.org/streaming/ - Spark Streaming * https://github.com/asavinov/bistro https://github.com/asavinov/bistro - Bistro Streams The focus of Benthod seems to be performance and resilience (implemented without persistence).
- spooneybarger 8y agoOr to toot the horn of the project I work on: * https://github.com/wallaroolabs/wallaroo https://github.com/wallaroolabs/wallaroo - Wallaroo
- jerf 8y ago"all the dull tasks" Now that is how you advertise at me. I've seen more "minimal flexible generic configurable simple performant powerful easy extensible non-bloated" libraries than I can shake a stick at. But "this helps with the dull stuff"... that's a sales pitch I can get behind! (A constructive hint to those describing their libraries: At this point, use of pretty much any of those words in your summary sentence is a massive negative to me. They are of null content, and you just burned the most valuable marketing space you had on these null-content words. Everybody says that about their libraries. Tell me what it does in the summary. If you really want to make the point for one of those words, do it later in the documentation, because if your summary sounds like you copy/pasted my quoted bit above, I'm not even going to read your extended documentation. And when you do, provide evidence; if you're "performant", show me competitive benchmarks. I get that they're going to be biased, but it at least shows me what's important. If you're "simple" show me in the sample code. Etc.)
- sgslo 8y agoAdd "elegant" onto that list and I think you've hit the nail on the head. These words aren't descriptors, they don't explain anything or give any further context about the nature of a library.
- staticautomatic 8y agoBut is it "for humans?"