4 ms·

because we essentially did spend exactly that to build a cluster: what amount of ressources are you using (coreh/instances/storage)?

by fock 2y ago

jasonjmcghee 2y ago
There are tradeoffs in both cases. Some issues with building a cluster are, you're locked into the tech you bought, need space, expertise to manage it, cost to run/maintain. You can potentially recoup some of that investment by selling it later, usually a quickly depreciating asset though. But, no AWS or AWS premium.
- littlestymaar 2y ago
  There's also the third way, of going for intermediate-size cloud providers, you get the lack of Capex and actual hardware to deal with without the AWS premium. I don't understand why so many people act as if the only alternative was AWS or self-hosted.
  - moralestapia 2y ago
    Never tried to suggest that, but tbh, the value added services on both AWS and GCP are hard to emulate, and they "just work" already. Sure, I could spend a few weeks (months?) compiling and trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ...
    - fock 2y ago
      The main value is that you can pay money to have decent network and storage with the hyperscalers. Which none of the intermediate-size cloud services offer to my knowledge? > I could spend a few weeks (months?) compiling why would you compile infra-stuff?! Usually this is nicely packaged... > random GNU-certified "alternatives" for Cloud Services but For application software you have to take care of this also with the cloud or are you just using the one precompiled environment they offer? Which cloud services do you need anyway - in our area (MatSci), things are very, very POSIX-centered and that is simple to set up.
      - moralestapia 2y ago
        >Usually this is nicely packaged... That has almost never been my experience with "alternatives" but if you can provide a few links I would like to learn about them.
        fock 2y ago
        well, I don't know what you use? however - ZFS is in Ubuntu now, you can host a nice NFS-server easily. MiniIO is a single Go binary, databases are packaged too. - Slurm is packaged too! I have to admit this gets hairy if you want to have something like https://github.com/NVIDIA/pyxis https://github.com/NVIDIA/pyxis but still this is far from arcane "alternative" software but the standard. If you buy Nvidia, it comes with your DGX-server just like its rented out by Amazon... the main remaining pain point I would see is actually netbooting/autoprovisioning the machines (at least this was annoying with us).
    - littlestymaar 2y ago
      What kind of “services” do they provide that provide value in the kind of HPC setting you were talking about? > trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ... A significant fraction, if not the majority, of those services are in fact hosted versions of open-source projets[1] so there's no need to be snobbish. [1]: and that's why things like Teraform and Redis are going source-available in recent days, to fight against cloud vendors parasitic behavior
    - Symbiote 2y ago
      It sounds like you're spending 7 figures of your research money without having done the most basic investigation of alternatives. I spend low 6 figures of research money on hardware + staff each year, and this avoids us spending 7 figures on cloud costs + staff.
- fock 2y ago
  I guess having built the physical system I am aware of the tradeoffs... Though due to the funding structure and 0-cost colocation (for our unit), there was not a lot to be discussed and thus I'd be interested in actual numbers for comparison!

because we essentially did spend exactly that to build a cluster: what amount of ressources are you using (coreh/instances/storage)?

by fock 2y ago

because we essentially did spend exactly that to build a cluster: what amount of ressources are you using (coreh/instances/storage)?

jasonjmcghee 2y ago

There are tradeoffs in both cases. Some issues with building a cluster are, you're locked into the tech you bought, need space, expertise to manage it, cost to run/maintain. You can potentially recoup some of that investment by selling it later, usually a quickly depreciating asset though. But, no AWS or AWS premium.

littlestymaar 2y ago
There's also the third way, of going for intermediate-size cloud providers, you get the lack of Capex and actual hardware to deal with without the AWS premium. I don't understand why so many people act as if the only alternative was AWS or self-hosted.
- moralestapia 2y ago
  Never tried to suggest that, but tbh, the value added services on both AWS and GCP are hard to emulate, and they "just work" already. Sure, I could spend a few weeks (months?) compiling and trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ...
  - fock 2y ago
    The main value is that you can pay money to have decent network and storage with the hyperscalers. Which none of the intermediate-size cloud services offer to my knowledge? > I could spend a few weeks (months?) compiling why would you compile infra-stuff?! Usually this is nicely packaged... > random GNU-certified "alternatives" for Cloud Services but For application software you have to take care of this also with the cloud or are you just using the one precompiled environment they offer? Which cloud services do you need anyway - in our area (MatSci), things are very, very POSIX-centered and that is simple to set up.
    - moralestapia 2y ago
      >Usually this is nicely packaged... That has almost never been my experience with "alternatives" but if you can provide a few links I would like to learn about them.
      - fock 2y ago
        well, I don't know what you use? however - ZFS is in Ubuntu now, you can host a nice NFS-server easily. MiniIO is a single Go binary, databases are packaged too. - Slurm is packaged too! I have to admit this gets hairy if you want to have something like https://github.com/NVIDIA/pyxis https://github.com/NVIDIA/pyxis but still this is far from arcane "alternative" software but the standard. If you buy Nvidia, it comes with your DGX-server just like its rented out by Amazon... the main remaining pain point I would see is actually netbooting/autoprovisioning the machines (at least this was annoying with us).
  - littlestymaar 2y ago
    What kind of “services” do they provide that provide value in the kind of HPC setting you were talking about? > trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ... A significant fraction, if not the majority, of those services are in fact hosted versions of open-source projets[1] so there's no need to be snobbish. [1]: and that's why things like Teraform and Redis are going source-available in recent days, to fight against cloud vendors parasitic behavior
  - Symbiote 2y ago
    It sounds like you're spending 7 figures of your research money without having done the most basic investigation of alternatives. I spend low 6 figures of research money on hardware + staff each year, and this avoids us spending 7 figures on cloud costs + staff.
fock 2y ago
I guess having built the physical system I am aware of the tradeoffs... Though due to the funding structure and 0-cost colocation (for our unit), there was not a lot to be discussed and thus I'd be interested in actual numbers for comparison!