4 ms·

someone has to go and power-cycle the machines every couple months it's chill, that's the point of not using ceph

by g413n 1y ago

paxys 1y ago
So the drives are never going to fail? PSUs are never going to burn out? You are never going to need to procure new parts? Negotiate with vendors?
- buckle8017 1y ago
  They mention data loss is acceptable, so im guessing they're only fixing big outages. Ignoring failed hdds week likely mean very little maintenance.
- theideaofcoffee 1y ago
  This concern troll that everyone trots out when anyone brings up running their own gear is just exhausting. The hyperscalers have melted people’s brains to a point where they can’t even fathom running shit for themselves. Yes, drives are going to fail. Yes, power supplies are going to burn out. Yes, god, you’re going to get new parts. Yes, you will have to actually talk to vendors. Big. Deal. This shit is -not- hard. For the amount of money you save by doing it like that, you should be clamoring to do it yourself. The concern trolling doesn’t make any sort of argument against it, it just makes you look lazy.
  - immibis 1y ago
    Very good point. There was something on the HN front page like this about self-hosted email, too. I point out to people that AWS is between ten to one hundred times more expensive than a normal server. The response is "but what if I only need it to handle peak load three hours a day?" Then you still come out ahead with your own server. We have multiple colo cages. We handle enough traffic - terabytes per second - that we'll never move those to cloud. Yet management always wants more cloud. While simultaneously complaining about how we're not making enough money.
    - ranger_danger 1y ago
      I don't think the answer is so black-and-white. IMO This only realistically applies to larger companies or ones that either push lots of traffic or have a need for large amounts of compute/storage/etc. But for smaller groups that don't have large/sustained workloads, I think they can absolutely save money compared to colo/dedicated servers using one of multiple different kinds of AWS services. I have several customers that coast along just fine with a $50/mo EC2 instance or less, compared to hundreds per month for a dedicated server... I wouldn't call that "ten times" by any stretch.
      - immibis 1y ago
        Small companies should go for the likes of Hetzner/OVH, which is still 10+ times cheaper than AWS. AWS is for anyone with a fear of committing to a particular amount of resource use, but once you've tried both and realised the price and performance differential, you realize you can easily way overcommit and still come out ahead, so it's not actually that scary. Plus, nobody's stopping you from continuing to spin up EC2s when your real servers are fully utilized.
        ranger_danger 1y ago
        Hard disagree... I think these black-and-white opinions are disingenuous, lack important nuance and are often just incorrect. I even have customers on $3/mo EC2 instances... the cheapest dedicated server on OVH is still twenty times more expensive than that. I don't think there's any way to "come out on top" with OVH in that scenario, short of maybe claiming that the customer is somehow "doing it wrong" by only paying for what they need. And yes hetzner/ovh have $3-4 cloud instances too, but now you're just directly competing with AWS and I don't see any benefit to call one any better than the other.
  - rtp4me 1y ago
    Thanks for this. I agree, there seems to be some sort of resistance to building and maintaining a CoLo infrastructure. In reality, it is not too difficult. As I mentioned above, spare parts on the shelve with the CoLo "remote hands" support and a good monitoring system can lessen the impact of almost any catastrophic issue. For the record, I have built (and currently maintain) a number of CoLo deployments. Our systems have been running for +10 years with very little failure of either drives or PSUs. In fact, our PSU failure rate is probably 1 every 3-4 years, and we probably loose a couple of drives per year. All in all, the systems are very reliable.
ttfvjktesd 1y ago
You are under the assumption that only Ceph (and similar complex software) requires staff, whereas plain 30 PB can be operated basically just by rebooting from time to time. I think that anyone with actual experience of operating thousands of physical disks in datacenters would challenge this assumption.
- devanshp 1y ago
  we have 6 months of experience operating thousands of physical disks in datacenters now! it's about a couple hours a month of employee time in steady-state.
  - ttfvjktesd 1y ago
    How about all the other infrastructure. Since you are obviously not using the cloud, you must have massive amounts of GPUs and operating systems. All of that has been working together, it's not just keep watching for the physical disks and all is set. Don't get me wrong, I buy the actual numbers regarding hardware costs, but in addition to that presenting the rest as basically a one man show in terms of maintenance hours is the point where I'm very sceptical.
    - g413n 1y ago
      oh we use cloud gpus, infiniband h100s absolutely aren't something we want to self-host. not aws tho, they're crazy overpriced; mithril and sfcompute! we also use cloudflare extensively for everything that isn't the core heap dataset, the convenience of buckets is totally worth it for most day-to-day usage. the heap is really just the main pretraining corpus and nothing else.
      - ttfvjktesd 1y ago
        How is it going to work when the GPU is in the cloud and the storage is miles away in a local colo in SF down the street? I was under the impression that the GPUs has to go multiple times over the training dataset, which means transfer 30 PB multiple times in and out of the clouds. Is the data link even fast enough? How much are you charged for data transfer fees.
datadrivenangel 1y ago
Assuming that they end up hiring a full time ops person at 500k annually total costs (250k base for a data center wizard), then that's 42k extra a month, or ~$70k. Still 200k per month lower than their next best offering.
- Symbiote 1y ago
  It's really not necessary. I have four racks rather than ten, and less storage but more compute. All purchased new from HP with warranties. Ordering each year takes a couple of days work. Racking that takes one or two. Initial setup (seeing differences with a new generation of server etc and customizing Ubuntu autoinstallation) is done in a day. So that's a week per year for setup. If we are really unlucky, add another week for a strange failure. (This happened once in the 10 years I've been doing this, a CPU needed replacement by the HP engineer.) I replaced a couple of drives in July, and a network fibre transceiver in May.

someone has to go and power-cycle the machines every couple months it's chill, that's the point of not using ceph

by g413n 1y ago

someone has to go and power-cycle the machines every couple months it's chill, that's the point of not using ceph

So the drives are never going to fail? PSUs are never going to burn out? You are never going to need to procure new parts? Negotiate with vendors?

buckle8017 1y ago
They mention data loss is acceptable, so im guessing they're only fixing big outages. Ignoring failed hdds week likely mean very little maintenance.
theideaofcoffee 1y ago
This concern troll that everyone trots out when anyone brings up running their own gear is just exhausting. The hyperscalers have melted people’s brains to a point where they can’t even fathom running shit for themselves. Yes, drives are going to fail. Yes, power supplies are going to burn out. Yes, god, you’re going to get new parts. Yes, you will have to actually talk to vendors. Big. Deal. This shit is -not- hard. For the amount of money you save by doing it like that, you should be clamoring to do it yourself. The concern trolling doesn’t make any sort of argument against it, it just makes you look lazy.
- immibis 1y ago
  Very good point. There was something on the HN front page like this about self-hosted email, too. I point out to people that AWS is between ten to one hundred times more expensive than a normal server. The response is "but what if I only need it to handle peak load three hours a day?" Then you still come out ahead with your own server. We have multiple colo cages. We handle enough traffic - terabytes per second - that we'll never move those to cloud. Yet management always wants more cloud. While simultaneously complaining about how we're not making enough money.
  - ranger_danger 1y ago
    I don't think the answer is so black-and-white. IMO This only realistically applies to larger companies or ones that either push lots of traffic or have a need for large amounts of compute/storage/etc. But for smaller groups that don't have large/sustained workloads, I think they can absolutely save money compared to colo/dedicated servers using one of multiple different kinds of AWS services. I have several customers that coast along just fine with a $50/mo EC2 instance or less, compared to hundreds per month for a dedicated server... I wouldn't call that "ten times" by any stretch.
    - immibis 1y ago
      Small companies should go for the likes of Hetzner/OVH, which is still 10+ times cheaper than AWS. AWS is for anyone with a fear of committing to a particular amount of resource use, but once you've tried both and realised the price and performance differential, you realize you can easily way overcommit and still come out ahead, so it's not actually that scary. Plus, nobody's stopping you from continuing to spin up EC2s when your real servers are fully utilized.
      - ranger_danger 1y ago
        Hard disagree... I think these black-and-white opinions are disingenuous, lack important nuance and are often just incorrect. I even have customers on $3/mo EC2 instances... the cheapest dedicated server on OVH is still twenty times more expensive than that. I don't think there's any way to "come out on top" with OVH in that scenario, short of maybe claiming that the customer is somehow "doing it wrong" by only paying for what they need. And yes hetzner/ovh have $3-4 cloud instances too, but now you're just directly competing with AWS and I don't see any benefit to call one any better than the other.
- rtp4me 1y ago
  Thanks for this. I agree, there seems to be some sort of resistance to building and maintaining a CoLo infrastructure. In reality, it is not too difficult. As I mentioned above, spare parts on the shelve with the CoLo "remote hands" support and a good monitoring system can lessen the impact of almost any catastrophic issue. For the record, I have built (and currently maintain) a number of CoLo deployments. Our systems have been running for +10 years with very little failure of either drives or PSUs. In fact, our PSU failure rate is probably 1 every 3-4 years, and we probably loose a couple of drives per year. All in all, the systems are very reliable.

ttfvjktesd 1y ago

You are under the assumption that only Ceph (and similar complex software) requires staff, whereas plain 30 PB can be operated basically just by rebooting from time to time. I think that anyone with actual experience of operating thousands of physical disks in datacenters would challenge this assumption.

devanshp 1y ago
we have 6 months of experience operating thousands of physical disks in datacenters now! it's about a couple hours a month of employee time in steady-state.
- ttfvjktesd 1y ago
  How about all the other infrastructure. Since you are obviously not using the cloud, you must have massive amounts of GPUs and operating systems. All of that has been working together, it's not just keep watching for the physical disks and all is set. Don't get me wrong, I buy the actual numbers regarding hardware costs, but in addition to that presenting the rest as basically a one man show in terms of maintenance hours is the point where I'm very sceptical.
  - g413n 1y ago
    oh we use cloud gpus, infiniband h100s absolutely aren't something we want to self-host. not aws tho, they're crazy overpriced; mithril and sfcompute! we also use cloudflare extensively for everything that isn't the core heap dataset, the convenience of buckets is totally worth it for most day-to-day usage. the heap is really just the main pretraining corpus and nothing else.
    - ttfvjktesd 1y ago
      How is it going to work when the GPU is in the cloud and the storage is miles away in a local colo in SF down the street? I was under the impression that the GPUs has to go multiple times over the training dataset, which means transfer 30 PB multiple times in and out of the clouds. Is the data link even fast enough? How much are you charged for data transfer fees.

datadrivenangel 1y ago

Assuming that they end up hiring a full time ops person at 500k annually total costs (250k base for a data center wizard), then that's 42k extra a month, or ~$70k. Still 200k per month lower than their next best offering.

Symbiote 1y ago
It's really not necessary. I have four racks rather than ten, and less storage but more compute. All purchased new from HP with warranties. Ordering each year takes a couple of days work. Racking that takes one or two. Initial setup (seeing differences with a new generation of server etc and customizing Ubuntu autoinstallation) is done in a day. So that's a week per year for setup. If we are really unlucky, add another week for a strange failure. (This happened once in the 10 years I've been doing this, a CPU needed replacement by the HP engineer.) I replaced a couple of drives in July, and a network fibre transceiver in May.