3 ms·
Can you give more background on what your data pipelines are like? Are they mostly batch processes? If so, I'd strongly recommend using a workflow tool like Lu
by abd12 9y ago
Can you give more background on what your data pipelines are like? Are they mostly batch processes?
If so, I'd strongly recommend using a workflow tool like Luigi[0] or Airflow[1]. In a phrase, I'd say they're like "Make for data".
[0]: https://github.com/spotify/luigi https://github.com/spotify/luigi
[1]: https://github.com/apache/incubator-airflow https://github.com/apache/incubator-airflow
- muramira 9y agoBasically, we improved a shell scripts based ETLs with lambdas. Now we barely can maintain them, and when a something breaks, it takes an engineer an inordinate amount of energy and time to get them fixed. Since we process about 0.5 to 1.5 TB of time series(IOT) I was thinking about an architecture that combine AWS kinesis and airflow