4 ms·

Are there similar products on the market? How is ETL typically done today?

jakestein 10y ago
Stitch (stitchdata.com) powers ETL for hundreds of customers, and we provide the Singer integrations as part of our product. ETL definitely existed before Singer and Stitch, however, and two of the most well known enterprise-focused products are Informatica and Talend.
haney 10y ago
The most similar thing I've found is Apache Beam[1] which allows you to run ETL code on multiple engines but doesn't necessarily come with off the shelf integrations for commercial products. As far as "how ETL is done today" it's definitely going through changes Airflow[2] is a popular open source tool that can run ETL pipelines but it's more of a task runner so again you're writing your own extraction and loading logic for things off the shelf CRMs and Accounting tools. I'm excited to not have to write extraction scripts for third party tools that I'm sure other people have also written. Very cool project. 1. https://beam.apache.org/ https://beam.apache.org/ 2. https://airflow.incubator.apache.org/ https://airflow.incubator.apache.org/
deleted 10y ago
[deleted]
woqe 10y ago
We are using Apache NiFi[1] to handle a lot of our ETL use cases. We have HTTP endpoints set up to receive data from our ERP's accounting system to send data to Concur and to update customers' Lawson punchout ordering systems with shipment information. The 'E' is an HTTP post with an XML payload. The 'T' consists of using the payload to query other databases to build the 'L' payload, and the 'L' is an HTTP post to the consumer's endpoints. Further, we have NiFi handling HL7 messages inside hospitals. The 'T' is the real winner, here. NiFi has built in transformer for HL7 messages which makes them a breeze with which to work. EDIT: I wanted to add that we've also used the EventHub processor to connect NiFi to Azure's services as well, and it has been rock solid for us. NiFi's data provenance, flowfile/attribute system, back pressure settings, auto-queueing, retrying capabilities all make it very reliable and robust. 1. https://nifi.apache.org/ https://nifi.apache.org/

Stitch (stitchdata.com) powers ETL for hundreds of customers, and we provide the Singer integrations as part of our product. ETL definitely existed before Singer and Stitch, however, and two of the most well known enterprise-focused products are Informatica and Talend.

haney 10y ago

The most similar thing I've found is Apache Beam[1] which allows you to run ETL code on multiple engines but doesn't necessarily come with off the shelf integrations for commercial products. As far as "how ETL is done today" it's definitely going through changes Airflow[2] is a popular open source tool that can run ETL pipelines but it's more of a task runner so again you're writing your own extraction and loading logic for things off the shelf CRMs and Accounting tools. I'm excited to not have to write extraction scripts for third party tools that I'm sure other people have also written. Very cool project. 1. https://beam.apache.org/ https://beam.apache.org/ 2. https://airflow.incubator.apache.org/ https://airflow.incubator.apache.org/

deleted 10y ago

[deleted]

woqe 10y ago

We are using Apache NiFi[1] to handle a lot of our ETL use cases. We have HTTP endpoints set up to receive data from our ERP's accounting system to send data to Concur and to update customers' Lawson punchout ordering systems with shipment information. The 'E' is an HTTP post with an XML payload. The 'T' consists of using the payload to query other databases to build the 'L' payload, and the 'L' is an HTTP post to the consumer's endpoints. Further, we have NiFi handling HL7 messages inside hospitals. The 'T' is the real winner, here. NiFi has built in transformer for HL7 messages which makes them a breeze with which to work. EDIT: I wanted to add that we've also used the EventHub processor to connect NiFi to Azure's services as well, and it has been rock solid for us. NiFi's data provenance, flowfile/attribute system, back pressure settings, auto-queueing, retrying capabilities all make it very reliable and robust. 1. https://nifi.apache.org/ https://nifi.apache.org/