3 ms·
Luigi – Python based ETL
- mindcrash 11y agoTitle is slightly misleading, per the documentation: "The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else." Especially mind the "These tasks can be anything" part.