3 ms·

Thank you! This looks indeed cheap. Can you share more stories on data persistence / checkpointing? If I have a job that requires 8 V100 with 3 days, what type

by liuliu 6y ago

Thank you! This looks indeed cheap. Can you share more stories on data persistence / checkpointing? If I have a job that requires 8 V100 with 3 days, what types of reliability am I looking at?

ilmoi 6y ago
Exactly the same as you would be had you been running an EC2 instance at AWS. The machines are hosted, maintained and managed in exactly the same way in a whitelabel DC.
- sacheendra 6y ago
  This I highly unlikely. Hyperscalers like AWS have custom power delivery, rack organization, redundant networking, etc. which make their instances reliable. Long running jobs often use checkpoints which require high speed networking and storage, which I don't see an option for. Eg, I cam get EC2 instances with 100gbps networking. Great job starting the service! But, I think you have ways to go before reaching Hyperscalers level reliability.
  - shrubble 6y ago
    It's highly likely that the reliability of this service will be as good or better than AWS. I have servers hosted in a similar quality datacenter with literally 9 years of uptime. It would be 10 but the customer shut down his services... By the way, unless something has changed, you can't get more than 10gbps of throughput of a single tcp stream in those 100gbps setups...
    - nixgeek 6y ago
      Why do you believe an individual stream is limited to 10Gbps?
      - shrubble 6y ago
        They have stated as much in the documentation. See various speeds and feeds depending on instance size and type, and whether inside or outside the VPC at e.g. here: https://d1.awsstatic.com/events/reinvent/2019/REPEAT_2_Deep-dive_into_100G_networking_&_Elastic_Fabric_Adapter_on_Amazon_EC2_CMP334-R2.pdf https://d1.awsstatic.com/events/reinvent/2019/REPEAT_2_Deep-...
joecot 6y ago
I'm not very familiar with machine learning setups, but in general with hosts you trade SLA for pricing. You trade the 99.999% uptime and data consistency of hyperscaled hosts for cheaper pricing at smaller hosts. Assuming you can backup your data at checkpoints, consider running it on a setup like this for very cheap, and then back it up every hour/day to a consistently reliable data storage. AWS EC2 is very expensive for computing, but S3 is relatively cheap for storage. Backblaze B2 storage is also even cheaper, but with less guaranteed reliability than S3. But the odds of both this going down on you and Backblaze failing at the same time are pretty low. I run a credit card processor on AWS, my important personal websites on Linode, and my fun time websites and video conferencing on a small Chicago host called Genesis Hosting with a website out of the 90s and dirt cheap pricing (but excellent support). Match your price and SLA with how much pain it's going to cause you if it goes down, and don't put pay extra to put all your stuff on the same 5 9s host if you don't really have to.

by liuliu 6y ago

Thank you! This looks indeed cheap. Can you share more stories on data persistence / checkpointing? If I have a job that requires 8 V100 with 3 days, what types of reliability am I looking at?

ilmoi 6y ago

Exactly the same as you would be had you been running an EC2 instance at AWS. The machines are hosted, maintained and managed in exactly the same way in a whitelabel DC.

sacheendra 6y ago
This I highly unlikely. Hyperscalers like AWS have custom power delivery, rack organization, redundant networking, etc. which make their instances reliable. Long running jobs often use checkpoints which require high speed networking and storage, which I don't see an option for. Eg, I cam get EC2 instances with 100gbps networking. Great job starting the service! But, I think you have ways to go before reaching Hyperscalers level reliability.
- shrubble 6y ago
  It's highly likely that the reliability of this service will be as good or better than AWS. I have servers hosted in a similar quality datacenter with literally 9 years of uptime. It would be 10 but the customer shut down his services... By the way, unless something has changed, you can't get more than 10gbps of throughput of a single tcp stream in those 100gbps setups...
  - nixgeek 6y ago
    Why do you believe an individual stream is limited to 10Gbps?
    - shrubble 6y ago
      They have stated as much in the documentation. See various speeds and feeds depending on instance size and type, and whether inside or outside the VPC at e.g. here: https://d1.awsstatic.com/events/reinvent/2019/REPEAT_2_Deep-dive_into_100G_networking_&_Elastic_Fabric_Adapter_on_Amazon_EC2_CMP334-R2.pdf https://d1.awsstatic.com/events/reinvent/2019/REPEAT_2_Deep-...

joecot 6y ago

I'm not very familiar with machine learning setups, but in general with hosts you trade SLA for pricing. You trade the 99.999% uptime and data consistency of hyperscaled hosts for cheaper pricing at smaller hosts. Assuming you can backup your data at checkpoints, consider running it on a setup like this for very cheap, and then back it up every hour/day to a consistently reliable data storage. AWS EC2 is very expensive for computing, but S3 is relatively cheap for storage. Backblaze B2 storage is also even cheaper, but with less guaranteed reliability than S3. But the odds of both this going down on you and Backblaze failing at the same time are pretty low. I run a credit card processor on AWS, my important personal websites on Linode, and my fun time websites and video conferencing on a small Chicago host called Genesis Hosting with a website out of the 90s and dirt cheap pricing (but excellent support). Match your price and SLA with how much pain it's going to cause you if it goes down, and don't put pay extra to put all your stuff on the same 5 9s host if you don't really have to.