3 ms·

I would recommend looking into Python-based workflow managers: Luigi[0] then Airflow[1], to get a hang of scheduling, DAGs, etc. Both are fairly simple to get s

I would recommend looking into Python-based workflow managers: Luigi[0] then Airflow[1], to get a hang of scheduling, DAGs, etc. Both are fairly simple to get started with (especially for a seasoned Python developer) and are used in production environments as well. [0] https://github.com/spotify/luigi https://github.com/spotify/luigi [1] https://github.com/apache/incubator-airflow https://github.com/apache/incubator-airflow

tomnipotent 8y ago
Only just look - don't actually try to use one of these on a side project unless you want to waste a LOT of time. They're amazing for large projects moving 100GB+ a day and running dozens or hundreds of different jobs to orchestrate a moving system, but are a total time sink for anything small.
- ljw1001 8y ago
  100 GB per day is pretty small, but your point holds