6 ms·
Seems like it is getting to be a trend to write your own workflow engine and open source it. Spotify has luigi, LinkedIn has Azkaban and Airbnb has Airflow. I w
by chuckcode 11y ago
Seems like it is getting to be a trend to write your own workflow engine and open source it. Spotify has luigi, LinkedIn has Azkaban and Airbnb has Airflow. I wonder at what market capitalization companies start rolling their own language like .net, go and swift?
- firloop 11y agoWell, Facebook spun PHP off into Hack if that counts.
- thejosh 11y agoAll a small framework called 'Ruby on Rails', released by Basecamp.
- eva1984 11y agoDon't forget Chronos. http://mesos.github.io/chronos/ http://mesos.github.io/chronos/
- toomuchtodo 11y agoIsn't Chronos more of a distributed scheduler ("fault tolerant cron") than a workflow engine?
- eva1984 11y agoDistributed job scheduler is, in a larger sense, a workflow engine. It will need to understand what is the next job to trigger, handle different situation where job fails\timeouts and do the retry if specified.
- samkone 11y agowhich was also started at Airbnb.
- deleted 11y ago[deleted]
- carrollgt 11y agodrake[0] from factual falls into this category as well. [0] - https://github.com/Factual/drake https://github.com/Factual/drake
- kroosaidher 11y agoHey, isn't a DSL your own language, strictly speaking? So any internal company-specific XML schema etc...
- rattray 11y agoHow is that in any way surprising? The more resources you have, the more custom-tailored you can make your tools. And I can't imagine it's a shock that data workflows aren't a completely-solved-for-all-usecases problem at this point...
- perlgeek 11y agoAirflow reminded me a lot of taskflow from the OpenStack project.