4 ms·

Does someone have experience using this (or other) cloud solutions in a scientific HPC context? The only obvious disadvantage I can think of is that every stude

Does someone have experience using this (or other) cloud solutions in a scientific HPC context? The only obvious disadvantage I can think of is that every student running something incurs a certain cost whereas after one has bought the actual computers, running jobs is somewhat free.

dagw 9y ago
The biggest downside (other than cost) I've found is that each vCPU core is quite a bit slower that what you'll get on equivalent real hardware. So any code that doesn't scale more or less linearly across an arbitrary number of cores will suffer.
- mrep 9y ago
  vCPU's are hyperthreads and Intel packs 2 of those per core so you're in actuality getting half the number of full cores.
  - ddorian43 9y ago
    Isn't the cloud amazing ?
brutos 9y ago
Depends heavily on the projected utilization. If you know your compute node is going to be computing for the next 3 years with at least medium utilization, then the self hosted metal is probably going to be quite a bit cheaper. Its amazing how much hardware you can pack into a single machine for 10k€. Last year our group bought two additional high-memory (768GB) nodes for around that price each (including support for a couple years from the vendor). A few years before we bought 40 nodes with 128GB RAM each, for a similar price to last years high-memory nodes (and a fast interconnect and a lot of storage). If you are at a larger research institution, you probably also have an IT department that can co-locate your hardware for next to nothing (compared to cloud). There you also will save a lot of ingress/egress, storage, backup, etc. costs. Regarding the per student costs, even with cloud instances I would consider running a traditional HPC job system (grid engine, lsf, torque, ...). The MIT had a nice solution with Starcluster [1] to easily deploy a SGE on AWS. It looks a bit dead now though. [1] http://star.mit.edu/cluster/ http://star.mit.edu/cluster/
- dx034 9y ago
  > Depends heavily on the projected utilization. If you know your compute node is going to be computing for the next 3 years with at least medium utilization, then the self hosted metal is probably going to be quite a bit cheaper. Isn't that already the case for 1 month? Bare metal doesn't mean own data centre or colocation. If you go with a hosting provider most offer dedicated hardware on a monthly contract. As long as you need them longer than 1-2 months that should be significantly cheaper than Google/AWS.
  - vidarh 9y ago
    That's usually the case for almost everything on AWS/Google. If you're using them for specific features, or for very bursty work (e.g. if you use the instances less than about 6-8 hours a day), they can be cost effective, but the moment you use instances full time and don't leverage/depend on a ton of extra services, you're paying way above the odds.
    - brianwawok 9y ago
      Kubernetes helps this a bit with bin packing. Much easier to keep 3 32 core servers loaded than 32 4 core servers.
      - vidarh 9y ago
        But that's the case whether you're using AWS or self-hosted, so it doesn't really alter that calculation much

dagw 9y ago

The biggest downside (other than cost) I've found is that each vCPU core is quite a bit slower that what you'll get on equivalent real hardware. So any code that doesn't scale more or less linearly across an arbitrary number of cores will suffer.

mrep 9y ago
vCPU's are hyperthreads and Intel packs 2 of those per core so you're in actuality getting half the number of full cores.
- ddorian43 9y ago
  Isn't the cloud amazing ?

brutos 9y ago

Depends heavily on the projected utilization. If you know your compute node is going to be computing for the next 3 years with at least medium utilization, then the self hosted metal is probably going to be quite a bit cheaper. Its amazing how much hardware you can pack into a single machine for 10k€. Last year our group bought two additional high-memory (768GB) nodes for around that price each (including support for a couple years from the vendor). A few years before we bought 40 nodes with 128GB RAM each, for a similar price to last years high-memory nodes (and a fast interconnect and a lot of storage). If you are at a larger research institution, you probably also have an IT department that can co-locate your hardware for next to nothing (compared to cloud). There you also will save a lot of ingress/egress, storage, backup, etc. costs. Regarding the per student costs, even with cloud instances I would consider running a traditional HPC job system (grid engine, lsf, torque, ...). The MIT had a nice solution with Starcluster [1] to easily deploy a SGE on AWS. It looks a bit dead now though. [1] http://star.mit.edu/cluster/ http://star.mit.edu/cluster/

dx034 9y ago
> Depends heavily on the projected utilization. If you know your compute node is going to be computing for the next 3 years with at least medium utilization, then the self hosted metal is probably going to be quite a bit cheaper. Isn't that already the case for 1 month? Bare metal doesn't mean own data centre or colocation. If you go with a hosting provider most offer dedicated hardware on a monthly contract. As long as you need them longer than 1-2 months that should be significantly cheaper than Google/AWS.
- vidarh 9y ago
  That's usually the case for almost everything on AWS/Google. If you're using them for specific features, or for very bursty work (e.g. if you use the instances less than about 6-8 hours a day), they can be cost effective, but the moment you use instances full time and don't leverage/depend on a ton of extra services, you're paying way above the odds.
  - brianwawok 9y ago
    Kubernetes helps this a bit with bin packing. Much easier to keep 3 32 core servers loaded than 32 4 core servers.
    - vidarh 9y ago
      But that's the case whether you're using AWS or self-hosted, so it doesn't really alter that calculation much