3 ms·
could you elaborate on the difference between what op is asking for and what nifi/airflow does? to me the use case of moving data through a couple of different
by dajohnson89 6y ago
could you elaborate on the difference between what op is asking for and what nifi/airflow does? to me the use case of moving data through a couple of different services could be solved by a message bus OR nifi.
- contravariant 6y agoNifi is designed to handle the problems that crop up when two systems can't talk directly to each other. It puts a buffer between them to allow one process to keep sending data when the other isn't quite fast enough, it can do some basic transformation when the data isn't quite in the right format, etc. However if you just need something to send messages then you're better off using a tool that does just that, you don't need the overhead of a system that can connect arbitrary applications that talk in incompatible protocols you just need a single protocol that allows your applications to send each other data. In their docs Nifi calls itself a dataflow tool and calls dataflow a necessary evil. It's the band-aid you need when you've got a mismatch between the way data is generated and the way it is consumed. It would be insane to deliberately create such a mismatch just to use Nifi.
- dajohnson89 6y agook that makes sense. but in that case wouldn't be easier to write your own adapters for each data source?
- contravariant 6y agoWell no, Nifi has a lot of adapters that you can use out of the box, making them yourself isn't easier.