3 ms·
Airflow can be used to process data that only adds up to 10s or 100s of MBs. In aggregate, a big tech company processes petabytes or exabytes, but remember, tec
by mliker 6y ago
Airflow can be used to process data that only adds up to 10s or 100s of MBs. In aggregate, a big tech company processes petabytes or exabytes, but remember, technology like Airflow isn't processing all of that simultaneously within one task. So no, I would not characterize Airflow as a solution to just "massive company problems".