Please check out DNAnexus (https://dnanexus.com/ https://dnanexus.com/). (Disclosure: I work there.) It's a platform-as-a-service that allows you to do exactly

Please check out DNAnexus (https://dnanexus.com/ https://dnanexus.com/). (Disclosure: I work there.) It's a platform-as-a-service that allows you to do exactly what you're asking. You submit jobs that you want to run, can set up dependencies between jobs in a pipeline, and we do all the instance management (though you have the option to choose different instance types to optimize the runtime or cost of your jobs depending on their resource requirements). It's built on top of AWS, though from the user perspective it's cloud-agnostic (e.g., you don't have to know anything about how to bring up/down EC2 instances, nor do you even need an AWS account). Although it was designed with genomic data analysis in mind (in particular, some of the relevant security controls for HIPAA and CLIA compliance), it's actually quite general. So even if you aren't doing bioinformatics, you can think of it as a nice service for running arbitrary Linux executables on arbitrary files in the cloud with minimal setup and administrative overhead. If you have any questions, feel free to shoot me an email (jdale at the domain mentioned above).