13 ms·

I've used it a fair bit, though not for a couple of years. Few points, some of which may be out of date: * I've seen customers fall into the trap of thinking t

I've used it a fair bit, though not for a couple of years. Few points, some of which may be out of date: * I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. * It persisted its config to an XML file, including the positions of boxes on the UI. Trying to keep this config in source control with multiple devs working on it was impossible. * Some people take the view that you should use 'native' NiFi processors and not custom code. This results in huge graphs of processors with 1000s of boxes with lines between you have to follow. Made both better and worse by being able to descend and ascend levels in the graph. The complexity that way quickly becomes insane. * You're essentially programming with it. I've no doubt you could use it to write, say, an XMPP server if so inclined. Which means you can do a great many things of huge complexity. Programming tools have developed models for inheritance and composition, abstraction, static analysis, etc. which NiFi just didn't have. The amount of repeated logic I've seen it's configuration accumulate is beyond anything I've seen from any novice programmer. I ended up feeling like it could be an OK choice in a very small number of places, but I never got to work on one of those. The NSA linking together multiple systems with a light touch is possibly one such use case. For most everyone else, I couldn't recommend it.

zokier 6y ago
> I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. That is sort of the key problem I see with NiFi (and equivalents). The heavy emphasis on graphical UI and visual paradigm sort of implies that its oriented towards non-developers, but problem is that it doesn't make non-developers suddenly expert system architects or developers even if they manage to click through the UI. And many developers probably prefer just defining stuff in code instead of having fancy UIs. So it sort of falls between these to categories. Of course there is huge spectrum of skill in people, and there are probably plenty of "semi-technical" persons to whom this is perfect match, especially if supported by some more techy people.
- dmix 6y ago
  Doesn't Microsoft Access have a similar interface? It's been a long time since I've used it.
  - popinman322 6y ago
    To me, honestly, this seems more similar to Azure Data Factory.
  - EvanAnderson 6y ago
    Based on a quick look at NiFi, SQL Server Integration Services (SSIS) seems similar. The configuration management tooling for SSIS is pretty good and it's amenable to version control better than, it sounds like, NiFi is. SSIS still suffers from the potential "gotchas" the top-level poster mentions.
    - drbw 6y ago
      I haven't used SSIS since about SQL2008 - back then it terrible to use with version control - not only was it a huge blob of XML, it had more xml escaped and shoved into attributes of the main document! Whats more it seemed to re-allocate the GUIDs of the elements every time you opened a diagram, so there were always changes... Sounds like it's improved a bit since then.
- mistrial9 6y ago
  it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ? I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"
  - cameronh90 6y ago
    I don't think it's about whether a graphical tool is performant. Many programming languages aren't performant either. It's more that graphical tools don't generally serve their stated purpose of making it possible for non-programmers to get things done because it appears that the main skill behind programming is actually problem decomposition and modelling rather than syntax. Additionally, graphical tools tend to have a bunch of problems inherent to them, such as being unable to write comments and harder to store in git and work on collaboratively and only having one single editor, which is usually much buggier than a text editor and compiler. So: graphical "languages" don't make most things much easier, and make other things much harder.
closeparen 6y ago
I have not found that argument persuasive to the managers who believe that coding is inherently wasteful if what you’re trying to do is technically possible in a workflow builder GUI.
- marcinzm 6y ago
  That's true but, personally, I just avoid working for managers like that.
  - closeparen 6y ago
    I can avoid this at the line manager level, but Directors love these things.
    - tatersolid 6y ago
      No, we don’t.
- oneplane 6y ago
  Compare coding to managing and suddenly they're on board. In practical terms: a checklist for work can achieve the same thing as a manager verbalising and visualising that checklist. Every argument the manager has against that is the same as 'gui workflow' vs code.
- monstrado 6y ago
  NiFi is also very useful for visually chaining together scripts (e.g. foo | bar | baz). In NiFi you can use ExecuteProcess and ExecuteStreamCommand to basically do what pipes (|) does. Further, the intermediate data between each process is backed by a WAL. You can also mix and match your pipeline, for example you can mix-in some of your custom scripts, and periodically use built-in processors that are provided to do additional stuff (e.g. PutKafka).
tiew9Vii 6y ago
Another NiFi article was in Hacker News the other week with a fair few positive comments. I left a comment with my thoughts and we pretty much agree with the exact same issues. Good to see I’m not the only one as seeing some organisations use it more and more to the extent insisting any data ingested in to systems need to go via nifi. Granted some of these are extremely large and disfunctional companies.
ekianjo 6y ago
Its indeed drag and drop but as soon as you want to do something that's slightly a little more complex than the regular processors in place, you need to use regex filters, branching, AVRO converters, and non-tech users will be lost very quickly. I see it very useful to automate certain operations (watch S3 storage, take action as soon as object comes in and store in into a DB), as for such use cases it's pretty much drag and drop.
leoh 6y ago
Curious if anyone from the NiFi team cares to comment / has thoughts on how to work around these issues (e.g. source control)
- yenwel 6y ago
  They already have a solution for that https://nifi.apache.org/registry.html https://nifi.apache.org/registry.html
ashtonkem 6y ago
The idea that some sufficiently smart compiler/language/tool will replace expensive developers is a siren song that’s as old as COBOL. Most of these tools have historically failed to deliver much (if any) value. In the few cases where they genuinely made programming easier, the bar for what computers were expected to accomplish was raised to the point where specialized labor was once again required.

zokier 6y ago

> I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. That is sort of the key problem I see with NiFi (and equivalents). The heavy emphasis on graphical UI and visual paradigm sort of implies that its oriented towards non-developers, but problem is that it doesn't make non-developers suddenly expert system architects or developers even if they manage to click through the UI. And many developers probably prefer just defining stuff in code instead of having fancy UIs. So it sort of falls between these to categories. Of course there is huge spectrum of skill in people, and there are probably plenty of "semi-technical" persons to whom this is perfect match, especially if supported by some more techy people.

dmix 6y ago
Doesn't Microsoft Access have a similar interface? It's been a long time since I've used it.
- popinman322 6y ago
  To me, honestly, this seems more similar to Azure Data Factory.
- EvanAnderson 6y ago
  Based on a quick look at NiFi, SQL Server Integration Services (SSIS) seems similar. The configuration management tooling for SSIS is pretty good and it's amenable to version control better than, it sounds like, NiFi is. SSIS still suffers from the potential "gotchas" the top-level poster mentions.
  - drbw 6y ago
    I haven't used SSIS since about SQL2008 - back then it terrible to use with version control - not only was it a huge blob of XML, it had more xml escaped and shoved into attributes of the main document! Whats more it seemed to re-allocate the GUIDs of the elements every time you opened a diagram, so there were always changes... Sounds like it's improved a bit since then.
mistrial9 6y ago
it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ? I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"
- cameronh90 6y ago
  I don't think it's about whether a graphical tool is performant. Many programming languages aren't performant either. It's more that graphical tools don't generally serve their stated purpose of making it possible for non-programmers to get things done because it appears that the main skill behind programming is actually problem decomposition and modelling rather than syntax. Additionally, graphical tools tend to have a bunch of problems inherent to them, such as being unable to write comments and harder to store in git and work on collaboratively and only having one single editor, which is usually much buggier than a text editor and compiler. So: graphical "languages" don't make most things much easier, and make other things much harder.

closeparen 6y ago

I have not found that argument persuasive to the managers who believe that coding is inherently wasteful if what you’re trying to do is technically possible in a workflow builder GUI.

marcinzm 6y ago
That's true but, personally, I just avoid working for managers like that.
- closeparen 6y ago
  I can avoid this at the line manager level, but Directors love these things.
  - tatersolid 6y ago
    No, we don’t.
oneplane 6y ago
Compare coding to managing and suddenly they're on board. In practical terms: a checklist for work can achieve the same thing as a manager verbalising and visualising that checklist. Every argument the manager has against that is the same as 'gui workflow' vs code.
monstrado 6y ago
NiFi is also very useful for visually chaining together scripts (e.g. foo | bar | baz). In NiFi you can use ExecuteProcess and ExecuteStreamCommand to basically do what pipes (|) does. Further, the intermediate data between each process is backed by a WAL. You can also mix and match your pipeline, for example you can mix-in some of your custom scripts, and periodically use built-in processors that are provided to do additional stuff (e.g. PutKafka).

tiew9Vii 6y ago

Another NiFi article was in Hacker News the other week with a fair few positive comments. I left a comment with my thoughts and we pretty much agree with the exact same issues. Good to see I’m not the only one as seeing some organisations use it more and more to the extent insisting any data ingested in to systems need to go via nifi. Granted some of these are extremely large and disfunctional companies.

ekianjo 6y ago

Its indeed drag and drop but as soon as you want to do something that's slightly a little more complex than the regular processors in place, you need to use regex filters, branching, AVRO converters, and non-tech users will be lost very quickly. I see it very useful to automate certain operations (watch S3 storage, take action as soon as object comes in and store in into a DB), as for such use cases it's pretty much drag and drop.

leoh 6y ago

Curious if anyone from the NiFi team cares to comment / has thoughts on how to work around these issues (e.g. source control)

yenwel 6y ago
They already have a solution for that https://nifi.apache.org/registry.html https://nifi.apache.org/registry.html

ashtonkem 6y ago

The idea that some sufficiently smart compiler/language/tool will replace expensive developers is a siren song that’s as old as COBOL. Most of these tools have historically failed to deliver much (if any) value. In the few cases where they genuinely made programming easier, the bar for what computers were expected to accomplish was raised to the point where specialized labor was once again required.