4 ms·
Didn’t look too much into Benthos but what’s the difference to Nifi?
by Scotrix 4y ago
Didn’t look too much into Benthos but what’s the difference to Nifi?
- mihaitodor 4y agoBenthos is much simpler, since it's stateless and it's a single static binary written in Go. I don't really know much about NiFi, but if you need to use a messages bus with Benthos, such as Kafka, you can. However, you don't have to.
- throwthere 4y agoBenthos looks like a cool project but thanks for turning me on to Nifi. Nifi has a lot more processors out of the box. Edit: I’m speechless but see below
- mihaitodor 4y agoWhich ones would you need? Happy to add more to Benthos. Feel free to open issues here: https://github.com/benthosdev/benthos/issues https://github.com/benthosdev/benthos/issues
- throwthere 4y agoGoogle Sheets, Drive and Slack. I'm not comfortable asking for it because I can't contribute.
- mihaitodor 4y agoOK, that's not a problem. There are a bunch of other channels that you can use to propose new features. Feel free to reach out via https://www.benthos.dev/community https://www.benthos.dev/community LE: I took note of those 3 and they should be quite straightforward to add. Thanks!
- NortySpock 4y agoI don't know much about Nifi, but from re-reviewing the documentation, it looks like Nifi is a stateful application with orchestration and multi-user access, etc. Benthos requires something else handle orchestration and source control and multi-user access and the conditions that trigger an ETL run. On the plus side, the stateless nature of Benthos means there isn't any big setup process or cluster of containers to login to or anything. You just call benthos from the cli and pass in a config file. Last week I needed to get 120k rows of data from an Oracle view that would crash on a few bad rows, and shove rows that worked into a SQL Server table. Writing a Benthos config to tick through every id, fetch the relevant rows one at a time, annotate with batch information, and drop error rows and error messages in a log and healthy rows into SQL Server was, all told, about 120 lines of YAML. Roughly 3 hours to write while consulting the extensive documentation, and then trying performance tweaks until I could saturate my network connection (increase threads to 256) only took another 30 minutes. Tweaking that YAML to cover a similar second scenario as a new config then was only another 15 minutes. The week before that, I was trying Benthos at home to stress test MQTT on a Raspberry Pi with synthetic data to see which messages got dropped or mis-ordered. (Depends on your QoS and in-flight settings, obviously) I've never had "basic" performance testing be so simple. If you are stuck on a Windows platform, it even works there, which is nice for one-off runs for dev work, troubleshooting, or break-fix work.
- mihaitodor 4y agoReally glad to hear the Oracle driver I added to it is proving useful! <3
- NortySpock 4y agoLet me tell you Benthos is a real breath of fresh air after fighting a few different ETL tools that are more GUI, chrome and administrivia than function. Thanks for contributing to Benthos!
- jeffail 4y agoThanks for sharing, this is awesome to hear!
- Scotrix 4y agoNice, thanks for sharing this. I’ll have a deeper look into Benthos, sounds interesting and I certainly like the simple/easy part :-)