5 ms·

You should have moved to bare metal months/years ago. Things like making your servers preemptible is like putting a bandaid on a gaping bullet hole.

lma21 10y ago
I may be missing something, why is bare metal better in this case (immutable/premptible servers)?
- banachtarski 10y ago
  Just the sheer number of instances the OP is using has put the usage FAR into the territory where the premiums spent on virtualized instance costs far outweigh any clever strategy one might use to make things cheaper in the cloud. It has nothing to do with what the infrastructure is, and everything to do with raw volume.
  - phonon 10y ago
    Spot instances are super cheap...about 80% less than regular instances.
    - banachtarski 10y ago
      Yea but it's still virtualized slow molasses at the end. And still more expensive. My point is that at 25k servers, even a 5% efficiency gain is significant, and I'm willing to bet the margins are still higher.
doh 10y ago
There are many reasons why bare metal doesn't make much sense for us, at least at the moment. Here are couple: - we're very small team (13 people). We would never be able to manage this kind of infrastructure - we're growing incredibly fast. Just 6 months ago we were running on ~300 servers. We grew so quickly that we are now working with GCE's infrastructure planers to plan our resource needs - even we're not suffering from massive bursts, we don't grow gradually but rather in massive jumps. It would take us months instead of days to grow this quickly on bare metal. This way we can grow in days. - it is way more complicated to host in many geographic locations, especially in asia. We need to be present in many parts of the world and with GCE/AWS/Azure it's very easy and convenient. - we did try to work with some bare metal hosters (like OVH), but they are not able to deliver the servers on the schedule we need. They require way longer times and obviously much longer commitments - the pricing is actually not that different once you run on preemptible (spot) instances. The bare bone would give us some performance boost, but the freedom is worth every penny.
- geggam 10y ago
  Once the hockey stick flattens out then you will want to find permanent homes, assuming you have a viable long term business model
- doh 10y ago
  I agree but that's years out. We are barely scratching the surface.
- late2part 10y ago
  It's likely that a large percentage of your workload is static. It's likely that bare metal is lower cost than spot - it is for me. It's likely that your issue is allocation of capital - you'd rather put your money into burning dollar bills into AWS versus pre-paying for 18 months of servers with capital. I'm happy your company is doing great. I agree GCP is a great competitor to AWS. I think you're being close minded to building out a 20-60 cabinet data center to offload some of your workload. It's hard to do a DC with over 20 racks and pay more than 1/2 of AWS's prices.
  - doh 10y ago
    You summed up nicely, up until the last point. I understand why would think so, because you don't know our setup. Just to give you an idea, we process 16PB of data every single month. This is an ingress traffic. If we would pay for this traffic going out (egress), we would end up paying over $1M just for the traffic itself. By keeping everything in a single spot it costs us literally 0. That said, I've tried. I reached out to many hosters, like OVH and others. They just don't have the capacity we need. 20 servers will not make a change for us. We wanted to start with 500, but it would take them 9 months.
    - late2part 10y ago
      They are fools. I will build you 1k servers with 100g to AWS in 75 days. Your $1M estimate is 3X too high at 2c/GB. In two hours I'll save you $25k per month. In 6 months $300k/month. Either you are growing and should be investing in cost efficiency or you run a staid lifestyle business. Which is fine, but if you aren't growing, get off expensive cloud.

lma21 10y ago

I may be missing something, why is bare metal better in this case (immutable/premptible servers)?

banachtarski 10y ago
Just the sheer number of instances the OP is using has put the usage FAR into the territory where the premiums spent on virtualized instance costs far outweigh any clever strategy one might use to make things cheaper in the cloud. It has nothing to do with what the infrastructure is, and everything to do with raw volume.
- phonon 10y ago
  Spot instances are super cheap...about 80% less than regular instances.
  - banachtarski 10y ago
    Yea but it's still virtualized slow molasses at the end. And still more expensive. My point is that at 25k servers, even a 5% efficiency gain is significant, and I'm willing to bet the margins are still higher.

doh 10y ago

There are many reasons why bare metal doesn't make much sense for us, at least at the moment. Here are couple: - we're very small team (13 people). We would never be able to manage this kind of infrastructure - we're growing incredibly fast. Just 6 months ago we were running on ~300 servers. We grew so quickly that we are now working with GCE's infrastructure planers to plan our resource needs - even we're not suffering from massive bursts, we don't grow gradually but rather in massive jumps. It would take us months instead of days to grow this quickly on bare metal. This way we can grow in days. - it is way more complicated to host in many geographic locations, especially in asia. We need to be present in many parts of the world and with GCE/AWS/Azure it's very easy and convenient. - we did try to work with some bare metal hosters (like OVH), but they are not able to deliver the servers on the schedule we need. They require way longer times and obviously much longer commitments - the pricing is actually not that different once you run on preemptible (spot) instances. The bare bone would give us some performance boost, but the freedom is worth every penny.

geggam 10y ago
Once the hockey stick flattens out then you will want to find permanent homes, assuming you have a viable long term business model
doh 10y ago
I agree but that's years out. We are barely scratching the surface.
late2part 10y ago
It's likely that a large percentage of your workload is static. It's likely that bare metal is lower cost than spot - it is for me. It's likely that your issue is allocation of capital - you'd rather put your money into burning dollar bills into AWS versus pre-paying for 18 months of servers with capital. I'm happy your company is doing great. I agree GCP is a great competitor to AWS. I think you're being close minded to building out a 20-60 cabinet data center to offload some of your workload. It's hard to do a DC with over 20 racks and pay more than 1/2 of AWS's prices.
- doh 10y ago
  You summed up nicely, up until the last point. I understand why would think so, because you don't know our setup. Just to give you an idea, we process 16PB of data every single month. This is an ingress traffic. If we would pay for this traffic going out (egress), we would end up paying over $1M just for the traffic itself. By keeping everything in a single spot it costs us literally 0. That said, I've tried. I reached out to many hosters, like OVH and others. They just don't have the capacity we need. 20 servers will not make a change for us. We wanted to start with 500, but it would take them 9 months.
  - late2part 10y ago
    They are fools. I will build you 1k servers with 100g to AWS in 75 days. Your $1M estimate is 3X too high at 2c/GB. In two hours I'll save you $25k per month. In 6 months $300k/month. Either you are growing and should be investing in cost efficiency or you run a staid lifestyle business. Which is fine, but if you aren't growing, get off expensive cloud.