4 ms·

> I can't even imagine the hourly bill Yes, and no. As a replacement for a reasonably well used cluster, its going to be super expensive. Something like 3-5x t

> I can't even imagine the hourly bill Yes, and no. As a replacement for a reasonably well used cluster, its going to be super expensive. Something like 3-5x the price. For providing burst capacity, its significantly cheaper/faster, and can be billed directly. But would I recommend running your stuff directly on AWS for compute heavy batch processing? no. if you are thinking about hiring machines in a datacenter, then yes.

littlestymaar 3y ago
> For providing burst capacity, its significantly cheaper/faster, and can be billed directly. Cloud is cheaper for such a workload, yes. But you still wouldn't want to pay the AWS premium for that. But I guess nobody ever got fired for choosing AWS.
- KaiserPro 3y ago
  Depends. If you want high speed storage, thats reliable and zero cost to setup (ie doesn't require staff to do) then AWS and spot instances is very much a cheap option. If you have in house expertise and can knock up a reliable high speed storage cluster (no porting to s3 is not the right option, that takes years and means you can't edit files inline.) then running on one of the other cloud providers is an option. But they often can't cope with the scale, or don't provide the right support.
  - littlestymaar 3y ago
    > and zero cost to setup (ie doesn't require staff to do) Ah, the fallacy of the cloud not requiring staff to magically work. Funnily enough, every time I heard this IRL, it was coming from dudes who was in fact paid to manage AWS instances and services for their company as their day job but somehow never included his salary and the one of his colleagues. > But they often can't cope with the scale Most people vastly overestimate the scale they need or underestimate the scale of even second tier cloud providers but there's not many workload that could cause storage trouble anywhere. For the record a single rack can host 5/10 PB, how many people needs multiple exabytes of storage again? > or don't provide the right support. I've never been a whole-data-centers-scale customer but my experience with AWS support didn't leave me in awe.
    - KaiserPro 3y ago
      > Ah, the fallacy of the cloud not requiring staff to magically work. its not a fallacy, its a tradeoff. If you want to have 100nodes doing batch work in a datacentre, you need to buy, install, commission and monitor hardware, storage and networking. Now, That is possible, but hard to do on your own. So realistically you'd get a managed service to install that stuff in a data centre. As your experience will show you, the level of service you get is "patchy". You really need some one who has worked on real steel to specify that kind of work. Those people are incredibly rare, especially if you want someone who can talk the rest of the stack as well. (That is one massive downside of the cloud, there are hardly any new versatile sysadmins being created.) As a former VFX sysadmin who looked after a number of large HPC clusters, its fairly simple to get something working, but something running reliably and fast is another matter. > Most people vastly overestimate the scale they need Yes, but we are talking about an HPC cluster that is >100k nodes. Something fast enough to cause hilarious QoS issues on most storage systems. especially single rack arrays with undersized metadata controllers. Even on isilon with its "scale out" clustering, one job can easily nail the entire cluster by doing too many metadata operations. (protip single namespace clusters are rarely worth the effort outside of a strict set of problems. Its much better to have a bunch of smaller fileservers grouped together in an automount. Think of it like DB sharding, but with folders.) Second tier cloud providers are rarely able to provide 10k GPUs on the spot, certainly not at a reasonable price. Most don't have the concept of spot pricing, so you're deep into contract negotiations. > AWS support didn't leave me in awe. Its not great unless you pay. but you will eventually get to someone who can answer your question. Azure on the other hand, less so.
      - littlestymaar 3y ago
        > its not a fallacy, its a tradeoff. If you want to have 100nodes doing batch work in a datacentre, you need to buy, install, commission and monitor hardware, storage and networking. The fallacy is that it costs zero staff. But in reality it doesn't, it reduces the amount of staff you need, but only to an extend, because you still need plenty of people to run and monitor your cloud infra. > Yes, but we are talking about an HPC cluster that is >100k nodes. It is also much higher requirements than anything AWS is used to run, so you either have the know-how in house or don't have it at all. > Second tier cloud providers are rarely able to provide 10k GPUs on the spot, If you need 10k GPUs on the spot, then you have a serious planning issue… I know that many companies have incompetent management, but that's not the kind of thing you're supposed to brag about in an internet discussion. And using AWS to cope with management and planning deficiency is only going to get you this far, at some point you'll need to address them eventually.
moralestapia 3y ago
I currently work on a research group that does all of its bioinformatics analysis in AWS. The bill is expensive, sure, 6 figures regularly, maybe 7 in its lifetime, but provisioning our own cluster (let alone a supercomputer) would've been way more costly and time consuming. You also now need people to maintain it, etc... And at the end of the day, it could well be the case that you still end up using AWS for something (hosting, storage, w/e) I think it's a W, so far.
- deleted 3y ago
  [deleted]
- fock 3y ago
  because we essentially did spend exactly that to build a cluster: what amount of ressources are you using (coreh/instances/storage)?
  - jasonjmcghee 3y ago
    There are tradeoffs in both cases. Some issues with building a cluster are, you're locked into the tech you bought, need space, expertise to manage it, cost to run/maintain. You can potentially recoup some of that investment by selling it later, usually a quickly depreciating asset though. But, no AWS or AWS premium.
    - littlestymaar 3y ago
      There's also the third way, of going for intermediate-size cloud providers, you get the lack of Capex and actual hardware to deal with without the AWS premium. I don't understand why so many people act as if the only alternative was AWS or self-hosted.
      - moralestapia 3y ago
        Never tried to suggest that, but tbh, the value added services on both AWS and GCP are hard to emulate, and they "just work" already. Sure, I could spend a few weeks (months?) compiling and trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ...

littlestymaar 3y ago

> For providing burst capacity, its significantly cheaper/faster, and can be billed directly. Cloud is cheaper for such a workload, yes. But you still wouldn't want to pay the AWS premium for that. But I guess nobody ever got fired for choosing AWS.

KaiserPro 3y ago
Depends. If you want high speed storage, thats reliable and zero cost to setup (ie doesn't require staff to do) then AWS and spot instances is very much a cheap option. If you have in house expertise and can knock up a reliable high speed storage cluster (no porting to s3 is not the right option, that takes years and means you can't edit files inline.) then running on one of the other cloud providers is an option. But they often can't cope with the scale, or don't provide the right support.
- littlestymaar 3y ago
  > and zero cost to setup (ie doesn't require staff to do) Ah, the fallacy of the cloud not requiring staff to magically work. Funnily enough, every time I heard this IRL, it was coming from dudes who was in fact paid to manage AWS instances and services for their company as their day job but somehow never included his salary and the one of his colleagues. > But they often can't cope with the scale Most people vastly overestimate the scale they need or underestimate the scale of even second tier cloud providers but there's not many workload that could cause storage trouble anywhere. For the record a single rack can host 5/10 PB, how many people needs multiple exabytes of storage again? > or don't provide the right support. I've never been a whole-data-centers-scale customer but my experience with AWS support didn't leave me in awe.
  - KaiserPro 3y ago
    > Ah, the fallacy of the cloud not requiring staff to magically work. its not a fallacy, its a tradeoff. If you want to have 100nodes doing batch work in a datacentre, you need to buy, install, commission and monitor hardware, storage and networking. Now, That is possible, but hard to do on your own. So realistically you'd get a managed service to install that stuff in a data centre. As your experience will show you, the level of service you get is "patchy". You really need some one who has worked on real steel to specify that kind of work. Those people are incredibly rare, especially if you want someone who can talk the rest of the stack as well. (That is one massive downside of the cloud, there are hardly any new versatile sysadmins being created.) As a former VFX sysadmin who looked after a number of large HPC clusters, its fairly simple to get something working, but something running reliably and fast is another matter. > Most people vastly overestimate the scale they need Yes, but we are talking about an HPC cluster that is >100k nodes. Something fast enough to cause hilarious QoS issues on most storage systems. especially single rack arrays with undersized metadata controllers. Even on isilon with its "scale out" clustering, one job can easily nail the entire cluster by doing too many metadata operations. (protip single namespace clusters are rarely worth the effort outside of a strict set of problems. Its much better to have a bunch of smaller fileservers grouped together in an automount. Think of it like DB sharding, but with folders.) Second tier cloud providers are rarely able to provide 10k GPUs on the spot, certainly not at a reasonable price. Most don't have the concept of spot pricing, so you're deep into contract negotiations. > AWS support didn't leave me in awe. Its not great unless you pay. but you will eventually get to someone who can answer your question. Azure on the other hand, less so.
    - littlestymaar 3y ago
      > its not a fallacy, its a tradeoff. If you want to have 100nodes doing batch work in a datacentre, you need to buy, install, commission and monitor hardware, storage and networking. The fallacy is that it costs zero staff. But in reality it doesn't, it reduces the amount of staff you need, but only to an extend, because you still need plenty of people to run and monitor your cloud infra. > Yes, but we are talking about an HPC cluster that is >100k nodes. It is also much higher requirements than anything AWS is used to run, so you either have the know-how in house or don't have it at all. > Second tier cloud providers are rarely able to provide 10k GPUs on the spot, If you need 10k GPUs on the spot, then you have a serious planning issue… I know that many companies have incompetent management, but that's not the kind of thing you're supposed to brag about in an internet discussion. And using AWS to cope with management and planning deficiency is only going to get you this far, at some point you'll need to address them eventually.

moralestapia 3y ago

I currently work on a research group that does all of its bioinformatics analysis in AWS. The bill is expensive, sure, 6 figures regularly, maybe 7 in its lifetime, but provisioning our own cluster (let alone a supercomputer) would've been way more costly and time consuming. You also now need people to maintain it, etc... And at the end of the day, it could well be the case that you still end up using AWS for something (hosting, storage, w/e) I think it's a W, so far.

deleted 3y ago
[deleted]
fock 3y ago
because we essentially did spend exactly that to build a cluster: what amount of ressources are you using (coreh/instances/storage)?
- jasonjmcghee 3y ago
  There are tradeoffs in both cases. Some issues with building a cluster are, you're locked into the tech you bought, need space, expertise to manage it, cost to run/maintain. You can potentially recoup some of that investment by selling it later, usually a quickly depreciating asset though. But, no AWS or AWS premium.
  - littlestymaar 3y ago
    There's also the third way, of going for intermediate-size cloud providers, you get the lack of Capex and actual hardware to deal with without the AWS premium. I don't understand why so many people act as if the only alternative was AWS or self-hosted.
    - moralestapia 3y ago
      Never tried to suggest that, but tbh, the value added services on both AWS and GCP are hard to emulate, and they "just work" already. Sure, I could spend a few weeks (months?) compiling and trying some random GNU-certified "alternatives" for Cloud Services but ... just nah ...