3 ms·

I currently work on a research group that does all of its bioinformatics analysis in AWS. The bill is expensive, sure, 6 figures regularly, maybe 7 in its lifet

I currently work on a research group that does all of its bioinformatics analysis in AWS. The bill is expensive, sure, 6 figures regularly, maybe 7 in its lifetime, but provisioning our own cluster (let alone a supercomputer) would've been way more costly and time consuming. You also now need people to maintain it, etc... And at the end of the day, it could well be the case that you still end up using AWS for something (hosting, storage, w/e) I think it's a W, so far.

deleted 3y ago
[deleted]
fock 3y ago
because we essentially did spend exactly that to build a cluster: what amount of ressources are you using (coreh/instances/storage)?
- jasonjmcghee 3y ago
  There are tradeoffs in both cases. Some issues with building a cluster are, you're locked into the tech you bought, need space, expertise to manage it, cost to run/maintain. You can potentially recoup some of that investment by selling it later, usually a quickly depreciating asset though. But, no AWS or AWS premium.
  - littlestymaar 3y ago
    There's also the third way, of going for intermediate-size cloud providers, you get the lack of Capex and actual hardware to deal with without the AWS premium. I don't understand why so many people act as if the only alternative was AWS or self-hosted.
    - moralestapia 3y ago
      Never tried to suggest that, but tbh, the value added services on both AWS and GCP are hard to emulate, and they "just work" already. Sure, I could spend a few weeks (months?) compiling and trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ...
      - fock 3y ago
        The main value is that you can pay money to have decent network and storage with the hyperscalers. Which none of the intermediate-size cloud services offer to my knowledge? > I could spend a few weeks (months?) compiling why would you compile infra-stuff?! Usually this is nicely packaged... > random GNU-certified "alternatives" for Cloud Services but For application software you have to take care of this also with the cloud or are you just using the one precompiled environment they offer? Which cloud services do you need anyway - in our area (MatSci), things are very, very POSIX-centered and that is simple to set up.
        moralestapia 3y ago
        >Usually this is nicely packaged... That has almost never been my experience with "alternatives" but if you can provide a few links I would like to learn about them.
        fock 3y ago
        well, I don't know what you use? however - ZFS is in Ubuntu now, you can host a nice NFS-server easily. MiniIO is a single Go binary, databases are packaged too. - Slurm is packaged too! I have to admit this gets hairy if you want to have something like https://github.com/NVIDIA/pyxis https://github.com/NVIDIA/pyxis but still this is far from arcane "alternative" software but the standard. If you buy Nvidia, it comes with your DGX-server just like its rented out by Amazon... the main remaining pain point I would see is actually netbooting/autoprovisioning the machines (at least this was annoying with us).
      - littlestymaar 3y ago
        What kind of “services” do they provide that provide value in the kind of HPC setting you were talking about? > trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ... A significant fraction, if not the majority, of those services are in fact hosted versions of open-source projets[1] so there's no need to be snobbish. [1]: and that's why things like Teraform and Redis are going source-available in recent days, to fight against cloud vendors parasitic behavior
  - fock 3y ago
    I guess having built the physical system I am aware of the tradeoffs... Though due to the funding structure and 0-cost colocation (for our unit), there was not a lot to be discussed and thus I'd be interested in actual numbers for comparison!
KaiserPro 3y ago
For a startup I ran, we too built an HPC cluster on AWS. It was very much a big win, as sourcing and installing that amount of kit would have been far too much upfront. I should have really mentioned your(our) use case, as smallish department without dedicated access to a cluster or people who can run a real steel cluster. AWS's one click lustre clustre service (FSx) combined with Batch is a really quick and powerful basic HPC cluster. Before we got bought out, we had planned for creating a "base load" of real steel machines to process the continuous jobs, and spin up the rest of the cluster as and when demand needed

deleted 3y ago

[deleted]

fock 3y ago

because we essentially did spend exactly that to build a cluster: what amount of ressources are you using (coreh/instances/storage)?

jasonjmcghee 3y ago
There are tradeoffs in both cases. Some issues with building a cluster are, you're locked into the tech you bought, need space, expertise to manage it, cost to run/maintain. You can potentially recoup some of that investment by selling it later, usually a quickly depreciating asset though. But, no AWS or AWS premium.
- littlestymaar 3y ago
  There's also the third way, of going for intermediate-size cloud providers, you get the lack of Capex and actual hardware to deal with without the AWS premium. I don't understand why so many people act as if the only alternative was AWS or self-hosted.
  - moralestapia 3y ago
    Never tried to suggest that, but tbh, the value added services on both AWS and GCP are hard to emulate, and they "just work" already. Sure, I could spend a few weeks (months?) compiling and trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ...
    - fock 3y ago
      The main value is that you can pay money to have decent network and storage with the hyperscalers. Which none of the intermediate-size cloud services offer to my knowledge? > I could spend a few weeks (months?) compiling why would you compile infra-stuff?! Usually this is nicely packaged... > random GNU-certified "alternatives" for Cloud Services but For application software you have to take care of this also with the cloud or are you just using the one precompiled environment they offer? Which cloud services do you need anyway - in our area (MatSci), things are very, very POSIX-centered and that is simple to set up.
      - moralestapia 3y ago
        >Usually this is nicely packaged... That has almost never been my experience with "alternatives" but if you can provide a few links I would like to learn about them.
        fock 3y ago
        well, I don't know what you use? however - ZFS is in Ubuntu now, you can host a nice NFS-server easily. MiniIO is a single Go binary, databases are packaged too. - Slurm is packaged too! I have to admit this gets hairy if you want to have something like https://github.com/NVIDIA/pyxis https://github.com/NVIDIA/pyxis but still this is far from arcane "alternative" software but the standard. If you buy Nvidia, it comes with your DGX-server just like its rented out by Amazon... the main remaining pain point I would see is actually netbooting/autoprovisioning the machines (at least this was annoying with us).
    - littlestymaar 3y ago
      What kind of “services” do they provide that provide value in the kind of HPC setting you were talking about? > trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ... A significant fraction, if not the majority, of those services are in fact hosted versions of open-source projets[1] so there's no need to be snobbish. [1]: and that's why things like Teraform and Redis are going source-available in recent days, to fight against cloud vendors parasitic behavior
- fock 3y ago
  I guess having built the physical system I am aware of the tradeoffs... Though due to the funding structure and 0-cost colocation (for our unit), there was not a lot to be discussed and thus I'd be interested in actual numbers for comparison!

KaiserPro 3y ago

For a startup I ran, we too built an HPC cluster on AWS. It was very much a big win, as sourcing and installing that amount of kit would have been far too much upfront. I should have really mentioned your(our) use case, as smallish department without dedicated access to a cluster or people who can run a real steel cluster. AWS's one click lustre clustre service (FSx) combined with Batch is a really quick and powerful basic HPC cluster. Before we got bought out, we had planned for creating a "base load" of real steel machines to process the continuous jobs, and spin up the rest of the cluster as and when demand needed