5 ms·

Ask HN: Job manager/queue for cloud instances

4 points by folli 12y ago

I have following problem to solve and in order to spare me from reinventing the wheel, I hope for some pointers to existing solutions or services: On top of a cloud infrastructure (AWS, Rackspace...) I need to somehow be able to manage jobs (research software which can take from a couple of minutes to a couple of hours to run), i.e. starting jobs on idle instances, spin up new instances if needed and stop unused instances. Does anyone know of a solution or service which could help me in this regard?

bbgm 12y ago
You might want to try CFNcluster (https://github.com/awslabs/cfncluster https://github.com/awslabs/cfncluster) or Starcluster (http://star.mit.edu/cluster/ http://star.mit.edu/cluster/). If you are used to Grid Engine, you should find them pretty easy to use.
- folli 12y ago
  Starcluster seems to be what I was looking for! Thank you very much
jdale27 12y ago
Please check out DNAnexus (https://dnanexus.com/ https://dnanexus.com/). (Disclosure: I work there.) It's a platform-as-a-service that allows you to do exactly what you're asking. You submit jobs that you want to run, can set up dependencies between jobs in a pipeline, and we do all the instance management (though you have the option to choose different instance types to optimize the runtime or cost of your jobs depending on their resource requirements). It's built on top of AWS, though from the user perspective it's cloud-agnostic (e.g., you don't have to know anything about how to bring up/down EC2 instances, nor do you even need an AWS account). Although it was designed with genomic data analysis in mind (in particular, some of the relevant security controls for HIPAA and CLIA compliance), it's actually quite general. So even if you aren't doing bioinformatics, you can think of it as a nice service for running arbitrary Linux executables on arbitrary files in the cloud with minimal setup and administrative overhead. If you have any questions, feel free to shoot me an email (jdale at the domain mentioned above).
- folli 12y ago
  Thanks for the reply, I sent you an email.

4 points by folli 12y ago

bbgm 12y ago

You might want to try CFNcluster (https://github.com/awslabs/cfncluster https://github.com/awslabs/cfncluster) or Starcluster (http://star.mit.edu/cluster/ http://star.mit.edu/cluster/). If you are used to Grid Engine, you should find them pretty easy to use.

folli 12y ago
Starcluster seems to be what I was looking for! Thank you very much

jdale27 12y ago

Please check out DNAnexus (https://dnanexus.com/ https://dnanexus.com/). (Disclosure: I work there.) It's a platform-as-a-service that allows you to do exactly what you're asking. You submit jobs that you want to run, can set up dependencies between jobs in a pipeline, and we do all the instance management (though you have the option to choose different instance types to optimize the runtime or cost of your jobs depending on their resource requirements). It's built on top of AWS, though from the user perspective it's cloud-agnostic (e.g., you don't have to know anything about how to bring up/down EC2 instances, nor do you even need an AWS account). Although it was designed with genomic data analysis in mind (in particular, some of the relevant security controls for HIPAA and CLIA compliance), it's actually quite general. So even if you aren't doing bioinformatics, you can think of it as a nice service for running arbitrary Linux executables on arbitrary files in the cloud with minimal setup and administrative overhead. If you have any questions, feel free to shoot me an email (jdale at the domain mentioned above).

folli 12y ago
Thanks for the reply, I sent you an email.