4 ms·

Depends. If you want high speed storage, thats reliable and zero cost to setup (ie doesn't require staff to do) then AWS and spot instances is very much a chea

Depends. If you want high speed storage, thats reliable and zero cost to setup (ie doesn't require staff to do) then AWS and spot instances is very much a cheap option. If you have in house expertise and can knock up a reliable high speed storage cluster (no porting to s3 is not the right option, that takes years and means you can't edit files inline.) then running on one of the other cloud providers is an option. But they often can't cope with the scale, or don't provide the right support.

littlestymaar 3y ago
> and zero cost to setup (ie doesn't require staff to do) Ah, the fallacy of the cloud not requiring staff to magically work. Funnily enough, every time I heard this IRL, it was coming from dudes who was in fact paid to manage AWS instances and services for their company as their day job but somehow never included his salary and the one of his colleagues. > But they often can't cope with the scale Most people vastly overestimate the scale they need or underestimate the scale of even second tier cloud providers but there's not many workload that could cause storage trouble anywhere. For the record a single rack can host 5/10 PB, how many people needs multiple exabytes of storage again? > or don't provide the right support. I've never been a whole-data-centers-scale customer but my experience with AWS support didn't leave me in awe.
- KaiserPro 3y ago
  > Ah, the fallacy of the cloud not requiring staff to magically work. its not a fallacy, its a tradeoff. If you want to have 100nodes doing batch work in a datacentre, you need to buy, install, commission and monitor hardware, storage and networking. Now, That is possible, but hard to do on your own. So realistically you'd get a managed service to install that stuff in a data centre. As your experience will show you, the level of service you get is "patchy". You really need some one who has worked on real steel to specify that kind of work. Those people are incredibly rare, especially if you want someone who can talk the rest of the stack as well. (That is one massive downside of the cloud, there are hardly any new versatile sysadmins being created.) As a former VFX sysadmin who looked after a number of large HPC clusters, its fairly simple to get something working, but something running reliably and fast is another matter. > Most people vastly overestimate the scale they need Yes, but we are talking about an HPC cluster that is >100k nodes. Something fast enough to cause hilarious QoS issues on most storage systems. especially single rack arrays with undersized metadata controllers. Even on isilon with its "scale out" clustering, one job can easily nail the entire cluster by doing too many metadata operations. (protip single namespace clusters are rarely worth the effort outside of a strict set of problems. Its much better to have a bunch of smaller fileservers grouped together in an automount. Think of it like DB sharding, but with folders.) Second tier cloud providers are rarely able to provide 10k GPUs on the spot, certainly not at a reasonable price. Most don't have the concept of spot pricing, so you're deep into contract negotiations. > AWS support didn't leave me in awe. Its not great unless you pay. but you will eventually get to someone who can answer your question. Azure on the other hand, less so.
  - littlestymaar 3y ago
    > its not a fallacy, its a tradeoff. If you want to have 100nodes doing batch work in a datacentre, you need to buy, install, commission and monitor hardware, storage and networking. The fallacy is that it costs zero staff. But in reality it doesn't, it reduces the amount of staff you need, but only to an extend, because you still need plenty of people to run and monitor your cloud infra. > Yes, but we are talking about an HPC cluster that is >100k nodes. It is also much higher requirements than anything AWS is used to run, so you either have the know-how in house or don't have it at all. > Second tier cloud providers are rarely able to provide 10k GPUs on the spot, If you need 10k GPUs on the spot, then you have a serious planning issue… I know that many companies have incompetent management, but that's not the kind of thing you're supposed to brag about in an internet discussion. And using AWS to cope with management and planning deficiency is only going to get you this far, at some point you'll need to address them eventually.

littlestymaar 3y ago

> and zero cost to setup (ie doesn't require staff to do) Ah, the fallacy of the cloud not requiring staff to magically work. Funnily enough, every time I heard this IRL, it was coming from dudes who was in fact paid to manage AWS instances and services for their company as their day job but somehow never included his salary and the one of his colleagues. > But they often can't cope with the scale Most people vastly overestimate the scale they need or underestimate the scale of even second tier cloud providers but there's not many workload that could cause storage trouble anywhere. For the record a single rack can host 5/10 PB, how many people needs multiple exabytes of storage again? > or don't provide the right support. I've never been a whole-data-centers-scale customer but my experience with AWS support didn't leave me in awe.

KaiserPro 3y ago
> Ah, the fallacy of the cloud not requiring staff to magically work. its not a fallacy, its a tradeoff. If you want to have 100nodes doing batch work in a datacentre, you need to buy, install, commission and monitor hardware, storage and networking. Now, That is possible, but hard to do on your own. So realistically you'd get a managed service to install that stuff in a data centre. As your experience will show you, the level of service you get is "patchy". You really need some one who has worked on real steel to specify that kind of work. Those people are incredibly rare, especially if you want someone who can talk the rest of the stack as well. (That is one massive downside of the cloud, there are hardly any new versatile sysadmins being created.) As a former VFX sysadmin who looked after a number of large HPC clusters, its fairly simple to get something working, but something running reliably and fast is another matter. > Most people vastly overestimate the scale they need Yes, but we are talking about an HPC cluster that is >100k nodes. Something fast enough to cause hilarious QoS issues on most storage systems. especially single rack arrays with undersized metadata controllers. Even on isilon with its "scale out" clustering, one job can easily nail the entire cluster by doing too many metadata operations. (protip single namespace clusters are rarely worth the effort outside of a strict set of problems. Its much better to have a bunch of smaller fileservers grouped together in an automount. Think of it like DB sharding, but with folders.) Second tier cloud providers are rarely able to provide 10k GPUs on the spot, certainly not at a reasonable price. Most don't have the concept of spot pricing, so you're deep into contract negotiations. > AWS support didn't leave me in awe. Its not great unless you pay. but you will eventually get to someone who can answer your question. Azure on the other hand, less so.
- littlestymaar 3y ago
  > its not a fallacy, its a tradeoff. If you want to have 100nodes doing batch work in a datacentre, you need to buy, install, commission and monitor hardware, storage and networking. The fallacy is that it costs zero staff. But in reality it doesn't, it reduces the amount of staff you need, but only to an extend, because you still need plenty of people to run and monitor your cloud infra. > Yes, but we are talking about an HPC cluster that is >100k nodes. It is also much higher requirements than anything AWS is used to run, so you either have the know-how in house or don't have it at all. > Second tier cloud providers are rarely able to provide 10k GPUs on the spot, If you need 10k GPUs on the spot, then you have a serious planning issue… I know that many companies have incompetent management, but that's not the kind of thing you're supposed to brag about in an internet discussion. And using AWS to cope with management and planning deficiency is only going to get you this far, at some point you'll need to address them eventually.