3 ms·

We have been using this architecture for containerized batch processing for more than a year: http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker

by vtuulos 10y ago

We have been using this architecture for containerized batch processing for more than a year: http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker.html http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-d... It is very convenient to be able to define any jobs as Docker containers, using the best language for the task, and define dependencies between their inputs and output explicitly, not being tied to any particular paradigm like MapReduce. I am more than happy to let AWS handle plumbing for this instead of having to maintain and operate our custom solution.

batbomb 10y ago
By chance, have you tried https://gc3-uzh-ch.github.io/elasticluster/ https://gc3-uzh-ch.github.io/elasticluster/ ? Also, have you looked at Airflow? Not comparing them, just curious.
- vtuulos 10y ago
  First time I hear about Elasticluster. Regarding cluster provisioning, auto-scaling is critical for cost and convenience. We managed to do this with a simple job queue that outputs a CloudWatch metric that in turn controls an auto-scaling group of spot instances. Works like a charm but AWS Batch should make this redundant. We took a look at Airflow, which looks nice, but we haven't had a compelling reason not to use Luigi.
- oavdeev 10y ago
  We've looked at Airflow. Two main things why we went with Luigi are * time-based and dependency-based scheduling do not mix well in our experience, if you try to do both at the same time you end up with a very complicated execution model. Luigi is much simpler, it is basically a makefile. To handle time one can just add dummy dependencies where "exists()" function returns true if now()>X, and/or just run the whole thing by cron. * We heavily use dynamic dependencies in Luigi, where one job decides how many childs to spawn and what parameters to use at run time. Makes it easy to do e.g. mapreduce style jobs purely in Luigi.
jdoliner 10y ago
AdRoll has done some very cool stuff using containers for data processing. Afaik they were some of the first people to use this approach, and I think it's worked out quite well for them and given them quite an edge over their competitors. If you're interested in building this type of infrastructure for yourself, I'd recommend taking a look at http://github.com/pachyderm/pachyderm http://github.com/pachyderm/pachyderm. I'm one of the founders so this is a somewhat shameless plug, but Pachyderm is designed to get people most of the way to what AdRoll has built without having to build as much of the "plumbing" from scratch. AdRoll was some of our earliest inspiration for the design of Pachyderm.
- vtuulos 10y ago
  Thanks! Pachyderm's version control over datasets is a killer feature.

by vtuulos 10y ago

batbomb 10y ago

By chance, have you tried https://gc3-uzh-ch.github.io/elasticluster/ https://gc3-uzh-ch.github.io/elasticluster/ ? Also, have you looked at Airflow? Not comparing them, just curious.

vtuulos 10y ago
First time I hear about Elasticluster. Regarding cluster provisioning, auto-scaling is critical for cost and convenience. We managed to do this with a simple job queue that outputs a CloudWatch metric that in turn controls an auto-scaling group of spot instances. Works like a charm but AWS Batch should make this redundant. We took a look at Airflow, which looks nice, but we haven't had a compelling reason not to use Luigi.
oavdeev 10y ago
We've looked at Airflow. Two main things why we went with Luigi are * time-based and dependency-based scheduling do not mix well in our experience, if you try to do both at the same time you end up with a very complicated execution model. Luigi is much simpler, it is basically a makefile. To handle time one can just add dummy dependencies where "exists()" function returns true if now()>X, and/or just run the whole thing by cron. * We heavily use dynamic dependencies in Luigi, where one job decides how many childs to spawn and what parameters to use at run time. Makes it easy to do e.g. mapreduce style jobs purely in Luigi.

jdoliner 10y ago

AdRoll has done some very cool stuff using containers for data processing. Afaik they were some of the first people to use this approach, and I think it's worked out quite well for them and given them quite an edge over their competitors. If you're interested in building this type of infrastructure for yourself, I'd recommend taking a look at http://github.com/pachyderm/pachyderm http://github.com/pachyderm/pachyderm. I'm one of the founders so this is a somewhat shameless plug, but Pachyderm is designed to get people most of the way to what AdRoll has built without having to build as much of the "plumbing" from scratch. AdRoll was some of our earliest inspiration for the design of Pachyderm.

vtuulos 10y ago
Thanks! Pachyderm's version control over datasets is a killer feature.