3 ms·

Can someone explain what this is? I can't find anything on the website that explains it

paullth 6y ago
Been that way for at least 5 years https://news.ycombinator.com/item?id=10190846 https://news.ycombinator.com/item?id=10190846 There's a few articles in the old comments that explains it's use case a little
mcnichol 6y ago
Think TIBCO data pipelines. You create sources of data and can have non-technical (or maybe non-developer) roles wire together these data pipelines with transformations, aggregations etc.
- chipsa 6y ago
  TIBCO = ?
  - dsl 6y ago
    TIBCO is an "Enterprise Message Bus." Think of RabbitMQ or 0MQ but supported by a giant enterprise grade vendor. My last experience with TIBCO was them telling us the product didn't support anything but running on bare metal because "virtualization does not guarantee the order of writes to disk" (their words not mine)
    - zok3102 6y ago
      True.. if your last interaction with TIBCO was c. 2005. Back then consistency issues were a real challenge with VMs, “ESB” was a real software category with transactional messaging as a pattern (I make it a practice to not judge customer preference on how they use our tools) Today you will find even our traditional brokered messaging products like EMS lifted and shifted on AWS/Azure infra. OTOH, if you are building new apps - you‘re likely to deploy TIBCO’s integration or stream apps / pipelines using Flogo/Pulsar/Kafka on k8s-lambda-gcr or on a fully managed service on cloud.tibco.com. (Source / Disclaimer: I do product stuff over at TIBCO Software. Happy to chat if there are further qns: rkozhikk<remove-this-at>tibco<dot>com)
gregw134 6y ago
Sure, I've worked with it. It let's you visually build data pipelines. It's extremely useful for getting work done quickly--you just drag and drop prebuilt connectors to things like elasticsearch, s3, or twitter and you have a data pipeline, including automatic backoff and the ability to inspect the data at each step. It's visual so it's easy to tell what's going on. Biggest downside is it's not automatically distributed. You can set it up to be distributed, but you have to do the plumbing yourself on the nifi graph by dropping nodes for routing tasks between nifi servers. Overall, perfect tool for quickly building a pipeline that can be easily shown to the business and in which you can visually see all data and errors at each step.
- hestefisk 6y ago
  Why would one consider distributing it in the first place? IMHO that would require distributed transactions, which would make it hugely complex.
contravariant 6y ago
I quite like the description they give in the docs[1], a brief summary: > Over the years dataflow has been one of those necessary evils in an architecture. [...] NiFi is built to help tackle these modern dataflow challenges. [1]: http://nifi.apache.org/docs.html http://nifi.apache.org/docs.html
zokier 6y ago
Disclaimer: I haven't worked with NiFi specifically, but with another similar product As far as I know, it at least partially fulfills the role of "enterprise application integration" pattern; the idea is sort of proto-servicemesh where you have bunch of weird enterprise applications all around and you need to get them talking to each other, so you plop in a fancy EAI middleware thingy in the middle which then will talk to each of the applications individually, converting and transforming data from one format and protocol to another. But its not just a dumb hub, in addition to doing arbitrarily complex transformations, it can also make "routing" decisions based on the data itself. And this pattern being product of its time, instead of defining the network in code in some nice DSL everything is just "configured" and lots of things is achieved through clicketyclicking through GUI. I suspect this is at least partially due "configuring a turn-key product" sounding less scary to managers than "developing a system on framework". If you look at the docs page of NiFi[1] and scroll quickly through the long list in the left sidebar with all the "processors", you already can get an idea what sort of things it does. While this sort of solutions can be very useful in some situations, I wouldn't necessarily start designing a greenfield architecture around NiFi. But if you end up running it, then you might find that piece by piece it will accumulate all sorts of bits and bops of things because it's "convenient" to throw it in there, while the logic flows become more and more eldritch in nature. [1] http://nifi.apache.org/docs.html http://nifi.apache.org/docs.html

Been that way for at least 5 years https://news.ycombinator.com/item?id=10190846 https://news.ycombinator.com/item?id=10190846 There's a few articles in the old comments that explains it's use case a little

mcnichol 6y ago

Think TIBCO data pipelines. You create sources of data and can have non-technical (or maybe non-developer) roles wire together these data pipelines with transformations, aggregations etc.

chipsa 6y ago
TIBCO = ?
- dsl 6y ago
  TIBCO is an "Enterprise Message Bus." Think of RabbitMQ or 0MQ but supported by a giant enterprise grade vendor. My last experience with TIBCO was them telling us the product didn't support anything but running on bare metal because "virtualization does not guarantee the order of writes to disk" (their words not mine)
  - zok3102 6y ago
    True.. if your last interaction with TIBCO was c. 2005. Back then consistency issues were a real challenge with VMs, “ESB” was a real software category with transactional messaging as a pattern (I make it a practice to not judge customer preference on how they use our tools) Today you will find even our traditional brokered messaging products like EMS lifted and shifted on AWS/Azure infra. OTOH, if you are building new apps - you‘re likely to deploy TIBCO’s integration or stream apps / pipelines using Flogo/Pulsar/Kafka on k8s-lambda-gcr or on a fully managed service on cloud.tibco.com. (Source / Disclaimer: I do product stuff over at TIBCO Software. Happy to chat if there are further qns: rkozhikk<remove-this-at>tibco<dot>com)

gregw134 6y ago

Sure, I've worked with it. It let's you visually build data pipelines. It's extremely useful for getting work done quickly--you just drag and drop prebuilt connectors to things like elasticsearch, s3, or twitter and you have a data pipeline, including automatic backoff and the ability to inspect the data at each step. It's visual so it's easy to tell what's going on. Biggest downside is it's not automatically distributed. You can set it up to be distributed, but you have to do the plumbing yourself on the nifi graph by dropping nodes for routing tasks between nifi servers. Overall, perfect tool for quickly building a pipeline that can be easily shown to the business and in which you can visually see all data and errors at each step.

hestefisk 6y ago
Why would one consider distributing it in the first place? IMHO that would require distributed transactions, which would make it hugely complex.

contravariant 6y ago

I quite like the description they give in the docs[1], a brief summary: > Over the years dataflow has been one of those necessary evils in an architecture. [...] NiFi is built to help tackle these modern dataflow challenges. [1]: http://nifi.apache.org/docs.html http://nifi.apache.org/docs.html

zokier 6y ago

Disclaimer: I haven't worked with NiFi specifically, but with another similar product As far as I know, it at least partially fulfills the role of "enterprise application integration" pattern; the idea is sort of proto-servicemesh where you have bunch of weird enterprise applications all around and you need to get them talking to each other, so you plop in a fancy EAI middleware thingy in the middle which then will talk to each of the applications individually, converting and transforming data from one format and protocol to another. But its not just a dumb hub, in addition to doing arbitrarily complex transformations, it can also make "routing" decisions based on the data itself. And this pattern being product of its time, instead of defining the network in code in some nice DSL everything is just "configured" and lots of things is achieved through clicketyclicking through GUI. I suspect this is at least partially due "configuring a turn-key product" sounding less scary to managers than "developing a system on framework". If you look at the docs page of NiFi[1] and scroll quickly through the long list in the left sidebar with all the "processors", you already can get an idea what sort of things it does. While this sort of solutions can be very useful in some situations, I wouldn't necessarily start designing a greenfield architecture around NiFi. But if you end up running it, then you might find that piece by piece it will accumulate all sorts of bits and bops of things because it's "convenient" to throw it in there, while the logic flows become more and more eldritch in nature. [1] http://nifi.apache.org/docs.html http://nifi.apache.org/docs.html