3 ms·

Is anyone out there doing (or considering) streaming ETL as opposed to batch?

artwr 8y ago
I think a lot of people are considering it. The ideas behind Apache Beam are supposed to make the transition easier, with the ability to run in either mode. I think in practice it really depends on your use case. For a large enough amount of data, especially on the ingest side, running it streaming makes sense. Now for that daily report from aggregated table, do you really need streaming ETL/aggregation is another question. The operational cost is non trivial with streaming.
- pgwhalen 8y ago
  Can you expand on that last sentence? Not that I’d disagree, I’m just curious about what specifically you’re referring to. My team is currently underwater supporting a decades old system of batch ETL, where it’s a major challenge to understand dependencies between jobs, and therefore what to rerun when something fails. We’re exploring a move to streaming ETL (Kafka streams being the leading candidate) as a way of making the dependencies explicit: rather than having to rerun stored procs or Talend jobs that we depend on certain data, we run a fleet of 24/7 microservices that transform and move data downstream whenever it is available.
woqe 8y ago
Apache nifi (https://nifi.apache.org/ https://nifi.apache.org/) is a software which handles streaming ETL processing. From a previous post I made about it: "We have HTTP endpoints set up to receive data from our ERP's accounting system to send data to Concur and to update customers' Lawson punchout ordering systems with shipment information. The 'E' is an HTTP post with an XML payload. The 'T' consists of using the payload to query other databases to build the 'L' payload, and the 'L' is an HTTP post to the consumer's endpoints."
- chrisjc 8y ago
  There's often a misconception about Nifi being batch or file based and not streaming. This probably originates as result of data in Nifi being represented as a FlowFile, which is kind of a misnomer. As woqe stated above, Nifi can certainly doing streaming.
thibaut_barrere 8y ago
I do both depending on the project needs!

I think a lot of people are considering it. The ideas behind Apache Beam are supposed to make the transition easier, with the ability to run in either mode. I think in practice it really depends on your use case. For a large enough amount of data, especially on the ingest side, running it streaming makes sense. Now for that daily report from aggregated table, do you really need streaming ETL/aggregation is another question. The operational cost is non trivial with streaming.

pgwhalen 8y ago
Can you expand on that last sentence? Not that I’d disagree, I’m just curious about what specifically you’re referring to. My team is currently underwater supporting a decades old system of batch ETL, where it’s a major challenge to understand dependencies between jobs, and therefore what to rerun when something fails. We’re exploring a move to streaming ETL (Kafka streams being the leading candidate) as a way of making the dependencies explicit: rather than having to rerun stored procs or Talend jobs that we depend on certain data, we run a fleet of 24/7 microservices that transform and move data downstream whenever it is available.

woqe 8y ago

Apache nifi (https://nifi.apache.org/ https://nifi.apache.org/) is a software which handles streaming ETL processing. From a previous post I made about it: "We have HTTP endpoints set up to receive data from our ERP's accounting system to send data to Concur and to update customers' Lawson punchout ordering systems with shipment information. The 'E' is an HTTP post with an XML payload. The 'T' consists of using the payload to query other databases to build the 'L' payload, and the 'L' is an HTTP post to the consumer's endpoints."

chrisjc 8y ago
There's often a misconception about Nifi being batch or file based and not streaming. This probably originates as result of data in Nifi being represented as a FlowFile, which is kind of a misnomer. As woqe stated above, Nifi can certainly doing streaming.

thibaut_barrere 8y ago

I do both depending on the project needs!