3 ms·

We have a team using this at work. They had built a process and needed it to be put on a VM and run periodically. They said the requirements were a dual core ma

We have a team using this at work. They had built a process and needed it to be put on a VM and run periodically. They said the requirements were a dual core machine and 8gb of ram. The “binary” was like 1.8gb. I’m sure it included a jre and a full nifi runtime, but god damn that is ridiculous. Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and able to run with 512mb of ram.

harikb 6y ago
Well, when one considers "no payload/data" flows through this system and it is merely managing metadata & workflow status, I wonder if it is web-scale enough :P your team will now ask for a cluster of these monstrosities.
RandyRanderson 6y ago
Interesting - can you expand on this: "Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and able to run with 512mb of ram"?
MasterIdiot 6y ago
NiFi is intended as a serious workhorse, we run it on a cluster with 256GM of RAM and over 20 cores for each server (on-prem), the 2GB of disk storage is negligible in any sort of scale (Just like the dual core/8gb machine)
ismail 6y ago
From the spec (8gb) it appears that NiFi was not needed. Were you co- locating zookeeper on the same nodes? Also you say it needed to be run periodically? It’s supposed to be a long running service. If you need something that you can spin up and shutdown in a container or VM then it is probably not the solution. Use it if you have high volumes of data you need to transport from A -> B. We run on a cluster with 256gb ram, 128 core per a node.
lordlinus 6y ago
The reason the binary is huge because it contains all the jar/nar for the connectors ( https://www.nifi.rocks/apache-nifi-processors/ https://www.nifi.rocks/apache-nifi-processors/). If you only require a subset of them you can use the minifi version which is quite small ( 49 MB for java and 3.2 MB for c++ version)

harikb 6y ago

Well, when one considers "no payload/data" flows through this system and it is merely managing metadata & workflow status, I wonder if it is web-scale enough :P your team will now ask for a cluster of these monstrosities.

RandyRanderson 6y ago

Interesting - can you expand on this: "Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and able to run with 512mb of ram"?

MasterIdiot 6y ago

NiFi is intended as a serious workhorse, we run it on a cluster with 256GM of RAM and over 20 cores for each server (on-prem), the 2GB of disk storage is negligible in any sort of scale (Just like the dual core/8gb machine)

ismail 6y ago

From the spec (8gb) it appears that NiFi was not needed. Were you co- locating zookeeper on the same nodes? Also you say it needed to be run periodically? It’s supposed to be a long running service. If you need something that you can spin up and shutdown in a container or VM then it is probably not the solution. Use it if you have high volumes of data you need to transport from A -> B. We run on a cluster with 256gb ram, 128 core per a node.

lordlinus 6y ago

The reason the binary is huge because it contains all the jar/nar for the connectors ( https://www.nifi.rocks/apache-nifi-processors/ https://www.nifi.rocks/apache-nifi-processors/). If you only require a subset of them you can use the minifi version which is quite small ( 49 MB for java and 3.2 MB for c++ version)