4 ms·
Vineyard serves as the data back end for airflow
- sighingnow 5y agoVineyard, an open-source immutable in-memory data-manager that is designed for optimize the data exchange between tasks inside a bigdata analytical workflow, now supports airflow and can serve as the data backend for efficient data sharing between tasks. See how vineyard helps for ETL jobs that large volume of complex data need to be shared between tasks: https://registry.astronomer.io/dags/v6d-etl-pandas https://registry.astronomer.io/dags/v6d-etl-pandas There's also a tutorial to get started: https://v6d.io/notes/airflow.html https://v6d.io/notes/airflow.html