3 ms·
Just put everything in a warehouse( via fivetran or some such thing) and just use DBT. Use airflow as cron runner for dbt. If you don't need realtime metrics,
by dominotw 4y ago
Just put everything in a warehouse( via fivetran or some such thing) and just use DBT.
Use airflow as cron runner for dbt.
If you don't need realtime metrics, this formula works way better than convoluted airflow dags.
- ricklamers 4y agoWe have had a great experience scheduling Meltano/dbt inside Orchest for our Metabase dashboards. As a pattern, combining these declarative/configuration CLI tools with a flexible orchestration layer (Orchest can run any containerized task, it will containerize transparently for you) really shines.
- kmbfjr 4y agoIf you need cron, use cron or a cronjob pod. Airflow is a poor cron scheduler at any sort of scale.
- dominotw 4y agoyea for sure. Not sure what a good cron runner with web ui with logs is, jenkins?