4 ms·
Really similar to the pipelines that I engineer/manage at my current company. Although we have our Airflow on kubernetes. One optimization though is separating
by prions 7y ago
Really similar to the pipelines that I engineer/manage at my current company. Although we have our Airflow on kubernetes.
One optimization though is separating your loading tasks from compute tasks. This makes the pipeline more resilient and makes backfilling/reprocessing less of a headache.
- san089 7y agoThanks for the tip. I actually thought about such separation, but it was too late to make such changes. I already laid down the architecture till that time. But you made a good point.