4 ms·
AWS Batch – Fully Managed Batch Processing
- koolba 10y agoIf this is what I think it is, I may have finally found my Heroku Scheduler killer.
- shizcakes 10y agoI could not tell from the marketing literature - what's the difference between this and EMR?
- rad_gruchalski 10y agoI can only assume, this sits higher than EMR. EMR is a managed Hadoop (and likes). This talks about running frameworks like luigi which is a workflow engine for, for example, Hadoop jobs.
- batbomb 10y agoThe jobs management interface. This is more like SLURM, Torque, LSF, etc...
- Pyxl101 10y agoIt looks like you submit jobs to a queue and it spins up capacity to execute them. In that sense it's like EMR. However the jobs are a Docker image or a zip file with shell scripts. In that sense it's not like EMR. I haven't gotten far enough yet to see if it has any distributed compute support, or if each job runs on a single machine. EMR is designed for master/worker architectures with many workers. It looks like AWS Batch might be designed to run each job on a single machine.
- jghn 10y agoIt's more like the container service without setting up your own cluster
- eva1984 10y agoWhat is the selling point of this? Poor man's Map(no reduce)??
- jghn 10y agoFor people involved with scientific workflow engines this is the Amazon equivalent of the Google Genomics pipelines API. For most of the computational world it's silly but for us it is pretty important.
- ronack 10y agoThis seems like a step backwards to me in some ways. I'd prefer to see them evolve Lambda to support containers, longer jobs, and better workflows instead. I thought we were moving away from EC2 with its slow provisioning, spot bidding, and per-hour billing.
- hackcrafter 10y agoI think there is a useful distinction and place for this. Lambda is for shared-compute. You don't need a dedicated server to run a "function" that takes < 60 seconds and can be called in a stateless manner as an API endpoint. This is dedicated host compute-heavy batch processing. It's a pain to do this at scale! I've built systems for running large scale life science embarrassingly parallelizable problems on EC2 and wish I had something like this! Imagine you have a set of input S3 files, each needs multiple-hours of compute to produce output S3 files. Doesn't seem that hard, until EC2 instances fail, programs crash, etc. etc.
- mason55 10y agoAnd the idea here is that you have a large parallelizeable problem but you don't need the Hadoop ecosystem? To put it another way, why not just use EMR?
- hackcrafter 10y agoYes, no Hadoop. Quite a lot of life science work is stand-alone programs that are domain-specific and read and write flat-files.
- marktangotango 10y agoAlso every legacy mainframe Cobol batch system on the planet.
- mostrowskiatl 10y agoWill AWS Batch support the running of COBOL batch programs?
- vtuulos 10y agoWe have been using this architecture for containerized batch processing for more than a year: http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker.html http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-d... It is very convenient to be able to define any jobs as Docker containers, using the best language for the task, and define dependencies between their inputs and output explicitly, not being tied to any particular paradigm like MapReduce. I am more than happy to let AWS handle plumbing for this instead of having to maintain and operate our custom solution.
- batbomb 10y agoBy chance, have you tried https://gc3-uzh-ch.github.io/elasticluster/ https://gc3-uzh-ch.github.io/elasticluster/ ? Also, have you looked at Airflow? Not comparing them, just curious.
- vtuulos 10y agoFirst time I hear about Elasticluster. Regarding cluster provisioning, auto-scaling is critical for cost and convenience. We managed to do this with a simple job queue that outputs a CloudWatch metric that in turn controls an auto-scaling group of spot instances. Works like a charm but AWS Batch should make this redundant. We took a look at Airflow, which looks nice, but we haven't had a compelling reason not to use Luigi.
- oavdeev 10y agoWe've looked at Airflow. Two main things why we went with Luigi are * time-based and dependency-based scheduling do not mix well in our experience, if you try to do both at the same time you end up with a very complicated execution model. Luigi is much simpler, it is basically a makefile. To handle time one can just add dummy dependencies where "exists()" function returns true if now()>X, and/or just run the whole thing by cron. * We heavily use dynamic dependencies in Luigi, where one job decides how many childs to spawn and what parameters to use at run time. Makes it easy to do e.g. mapreduce style jobs purely in Luigi.
- jdoliner 10y ago
- RMarcus 10y ago"AWS Batch dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted." Optimal is a pretty strong claim, given that this is an NP-Hard problem. Or do we just toss the word "optimal" around now?