3 ms·
+1 In my org we never use Airflow to compute anything. We only use a single operator: PythonOperator. All business and data access logic is encapsuled in REST
by pid-1 5y ago
+1
In my org we never use Airflow to compute anything. We only use a single operator: PythonOperator.
All business and data access logic is encapsuled in REST APIs. Our DAGs are responsible for calling those APIs in the correct order, in the correct time, retrying when needed. I really dislike using Airflow for anything else, as it generally becomes a huge mess.
- psychometry 5y agoIf it's all Python, why not use Prefect?
- yashasolutions 5y agoFirst time I read about Prefect, can you briefly share why you would use prefect over airflow if using only python? (that's what I am about to be doing soon...)
- jonahbenton 5y agoHave been a Prefect user since their first release. In a nutshell, it's a native pythonic DAG tool, where Airflow is a DAG tool that happens to be written in Python.
- pid-1 5y agoIt didn't exist when we bootstrapped our department. Nowadays Airflow is a dev favorite and we have no good reason to seek alternatives.