3 ms·
Could you point me to the component responsible for storing the intermediate data of the DAG run? I was looking for this but couldn’t find it from quickly scann
by barefeg 5y ago
Could you point me to the component responsible for storing the intermediate data of the DAG run? I was looking for this but couldn’t find it from quickly scanning the docs
- joelschw 5y agoHere you go - https://kedro.readthedocs.io/en/0.17.6/09_development/03_commands_reference.html#modifying-a-kedro-run https://kedro.readthedocs.io/en/0.17.6/09_development/03_com...
- waylonwalker 5y agoBy adding a catalog entry for each of your datasets with a filepath argument, it will save them to that place. All their datasets use fsspec under the hood so this path can be an s3 bucket or a local path and it all just works.