3 ms·
By chance, have you tried https://gc3-uzh-ch.github.io/elasticluster/ https://gc3-uzh-ch.github.io/elasticluster/ ? Also, have you looked at Airflow? Not comp
by batbomb 10y ago
By chance, have you tried https://gc3-uzh-ch.github.io/elasticluster/ https://gc3-uzh-ch.github.io/elasticluster/ ?
Also, have you looked at Airflow?
Not comparing them, just curious.
- vtuulos 10y agoFirst time I hear about Elasticluster. Regarding cluster provisioning, auto-scaling is critical for cost and convenience. We managed to do this with a simple job queue that outputs a CloudWatch metric that in turn controls an auto-scaling group of spot instances. Works like a charm but AWS Batch should make this redundant. We took a look at Airflow, which looks nice, but we haven't had a compelling reason not to use Luigi.
- oavdeev 10y agoWe've looked at Airflow. Two main things why we went with Luigi are * time-based and dependency-based scheduling do not mix well in our experience, if you try to do both at the same time you end up with a very complicated execution model. Luigi is much simpler, it is basically a makefile. To handle time one can just add dummy dependencies where "exists()" function returns true if now()>X, and/or just run the whole thing by cron. * We heavily use dynamic dependencies in Luigi, where one job decides how many childs to spawn and what parameters to use at run time. Makes it easy to do e.g. mapreduce style jobs purely in Luigi.