3 ms·
re points 3 and 4, you can use airflow in a non-distributed manner to just run bash jobs (bash running python in my case). The telemetry it collects and web in
by mattbillenstein 8y ago
re points 3 and 4, you can use airflow in a non-distributed manner to just run bash jobs (bash running python in my case). The telemetry it collects and web interface give you a lot of visibility that you don't get with cron and plain bash jobs.
I started with Airflow in this capacity -- maybe taking a day or two to set it up figuring I'd add celery when we needed it. It's been maybe 1.5 years and task queues may never be needed; we can always just buy a bigger VM on ec2 as it's only 8-core/15GB atm.