4 ms·

Is there a spot instance-like mechanism with google cloud? I regularly request high memory spot instances (r3.8xlarge) on AWS for 1-2 days processes (genomic an

by daemonk 10y ago

Is there a spot instance-like mechanism with google cloud? I regularly request high memory spot instances (r3.8xlarge) on AWS for 1-2 days processes (genomic analysis). With spot-pricing it can be pretty cheap.

alainv 10y ago
https://cloud.google.com/preemptible-vms/ https://cloud.google.com/preemptible-vms/
dgacmu 10y ago
Yes. Google calls them "preemptible instances". Unlike the AWS spot instances, they're fixed price at a discount.
- phonon 10y ago
  Preemptible Instances only can be used for a max of 24 hours, so would not fit the above usage pattern...though I don't know if AWS spot instances are really meant to be used for that long either. Of course both allow you to use the regular non-preemptible on-demand pricing. https://cloud.google.com/compute/docs/instances/preemptible https://cloud.google.com/compute/docs/instances/preemptible
jpatokal 10y ago
Cloud Genomics is intended for use cases like yours: https://cloud.google.com/genomics/ https://cloud.google.com/genomics/
arrogantrobot 10y ago
A very clever way to run genomic analysis on GCP is via WDL ( https://software.broadinstitute.org/wdl/ https://software.broadinstitute.org/wdl/ ) workflows executed on the Cromwell ( https://github.com/broadinstitute/cromwell https://github.com/broadinstitute/cromwell ) service. Using WDL you define tasks, and compose those into workflows, then Cromwell executes those tasks on GCP using the pipelines api ( https://cloud.google.com/genomics/reference/rest/v1alpha2/pipelines https://cloud.google.com/genomics/reference/rest/v1alpha2/pi... ). So you don't need to set up any instances at all, each stage of your pipeline, a task, is executed in its own docker instance, with files from google cloud storage being materialized on that instance for you. When that task is done, outputs are sent to cloud storage buckets, and everything is torn down. Picard can pull reads directly out of the google genomics API, and more of the Broad tools are adopting the GA4GH api, so this means an increasing amount of core tools anyone doing genomics will want to use, will work directly with google genomics.

by daemonk 10y ago

alainv 10y ago

https://cloud.google.com/preemptible-vms/ https://cloud.google.com/preemptible-vms/

dgacmu 10y ago

Yes. Google calls them "preemptible instances". Unlike the AWS spot instances, they're fixed price at a discount.

phonon 10y ago
Preemptible Instances only can be used for a max of 24 hours, so would not fit the above usage pattern...though I don't know if AWS spot instances are really meant to be used for that long either. Of course both allow you to use the regular non-preemptible on-demand pricing. https://cloud.google.com/compute/docs/instances/preemptible https://cloud.google.com/compute/docs/instances/preemptible

jpatokal 10y ago

Cloud Genomics is intended for use cases like yours: https://cloud.google.com/genomics/ https://cloud.google.com/genomics/

arrogantrobot 10y ago

A very clever way to run genomic analysis on GCP is via WDL ( https://software.broadinstitute.org/wdl/ https://software.broadinstitute.org/wdl/ ) workflows executed on the Cromwell ( https://github.com/broadinstitute/cromwell https://github.com/broadinstitute/cromwell ) service. Using WDL you define tasks, and compose those into workflows, then Cromwell executes those tasks on GCP using the pipelines api ( https://cloud.google.com/genomics/reference/rest/v1alpha2/pipelines https://cloud.google.com/genomics/reference/rest/v1alpha2/pi... ). So you don't need to set up any instances at all, each stage of your pipeline, a task, is executed in its own docker instance, with files from google cloud storage being materialized on that instance for you. When that task is done, outputs are sent to cloud storage buckets, and everything is torn down. Picard can pull reads directly out of the google genomics API, and more of the Broad tools are adopting the GA4GH api, so this means an increasing amount of core tools anyone doing genomics will want to use, will work directly with google genomics.