9 ms·
How well would this work with something like airflow ? If I have jobs that run only end of day for an hour on airflow, I pay just for an hour of the database ?
by quantguy11959 7y ago
How well would this work with something like airflow ? If I have jobs that run only end of day for an hour on airflow, I pay just for an hour of the database ?
- habitue 7y agoDepends on how you've configured airflow. It polls its dag directory constantly, and may do some db queries for each poll (don't quote me on this, I definitely don't know everything about the internals of airflow). From what I remember you can reduce the polling period in config
- quantguy11959 7y agoYeah I’ll need to do some research, if I can only poll the db for just the time I need this will be a huge cost saving for me. I think the db only gets queried if a dag actually runs but I’m not 100% sure.