3 ms·
Check out Luigi (Python -- https://github.com/spotify/luigi https://github.com/spotify/luigi). I've built (or worked on) a few bespoke systems myself, but Luig
by rch 8y ago
Check out Luigi (Python -- https://github.com/spotify/luigi https://github.com/spotify/luigi).
I've built (or worked on) a few bespoke systems myself, but Luigi covers better than 80% of what I typically need.
- dfsegoat 8y agoAlso using luigi here in production since 2015. We use it to manage a multi-day pipeline that is essentially semi-automatic (requires occasional human intervention).
- tedmiston 8y agoThere was a good tutorial at PyCon this past weekend called Workflow Engines Up and Running [1] on Python workflow automation frameworks, specifically comparing Luigi vs Airflow. The video is on YouTube as well [2]. [1]: https://us.pycon.org/2018/schedule/presentation/58/ https://us.pycon.org/2018/schedule/presentation/58/ [2]: https://youtu.be/kw0RL9LZk9s https://youtu.be/kw0RL9LZk9s