4 ms·

Why don't they host it themselves in a colo? Doing like Netflix did by paying AWS millions is wasteful and insane when managed colo is cheaper and self-managed

Why don't they host it themselves in a colo? Doing like Netflix did by paying AWS millions is wasteful and insane when managed colo is cheaper and self-managed colo is cheaper still. ByteMark.co.uk and pair.com are both great vendors that could save them money.

sz4kerto 7y ago
What we do at my company: everything is in the cloud except build farm because renting dedicated servers is much cheaper, we need lots of them and we don't care if one drops out. So we have 32c/256G boxes for running tests (a full suite finishes in 2 hrs on a box like that), they cost something like $300/mo.
- zozbot234 7y ago
  > we don't care if one drops out. Are you comparing with spot/preemptable instances? They can be a lot cheaper than 'standard' cloud services.
jcrawfordor 7y ago
Colocation would shift a good portion of the annual expense to an up-front investment in hardware (as well as setup, shipping, install fees). It would also increase the required admin time. It's quite possible that they could save some money by going this way, but it's by no means going to just fix their problem. In a way it would exacerbate it due to the capex for hardware.
- StavrosK 7y ago
  They don't have to buy their own servers, renting from Hetzner would probably be much cheaper than AWS or Gitlab, and you can automate installing the runner on the server so your administration needs get reduced to basically "reformat the server and run this provisioning script". I'm sure there are things I'm not taking into account, but it doesn't seem that bad.
  - jcrawfordor 7y ago
    Their issue is GitLab itself, not the runner. And sure, Gitlab is available in forms like a Docker image, but as someone who managed Gitlab for years... it's reasonably high effort. Additionally, I would not trust Hetzner with anything that gets this much usage. They're very cheap, and they're very cheap for a reason. When buying colo for a busy application, it's not unusual for bandwidth to cost more than the power and rack. All in all I'd expect them to be looking at at least $2k a month or $24k a year for power/rack/bandwidth. This would be something like an average, there are lower rates available in markets with cheap connectivity but then if the equipment isn't local you end up spending more on smart hands. In many places the rate would be higher. Then up-front hardware purchase could easily be $50k and probably more. I wouldn't expect to see savings at all the first year, I'd expect it to cost a chunk more. Of course that's not quite fair, the hardware purchase amortizes over at least a few years, but when you're struggling to pay the bills that doesn't make you feel any better. Renting hardware is an option, but will still cost you more than colo on year for year for equivalent capacity. Dedicated servers from a reputable source start at a couple hundred a month and quickly go up from there. It sounds like their situation will require HA. I'm just waving my hands here based on my guess of how much usage they're seeing for the $75k they estimate in AWS spend, but I have a quote in my inbox right now from a reputable datacenter operator for $18k per year just for rack/power/x-connect, that's before bandwidth, which I'm going to be paying two different providers for for diversity (and this is still worse connectivity than AWS gets you). That's a full rack, which is a lot of room, but the power/xconnect/bandwidth part of that basically don't change as you go to partial racks so the savings are not really that big. Quality colocation costs real money, and a gitlab instance with that kind of usage shouldn't be running out of a bargain-basement provider.
    - StavrosK 7y ago
      That's very informative, thank you. It's still a third of the price, though, if your calculations are ballpark correct, which is nothing to scoff at.
- marcinzm 7y ago
  Their main cost is outgoing bandwidth, AWS outgoing bandwidth is really expensive.
  - jcrawfordor 7y ago
    Yeah, but bandwidth in a DC is pretty expensive too, it's just billed on a totally different basis. That can actually make these things a bit more complicated because if you're paying for bandwidth on a 95%ile or dedicated basis you end up managing your budget by slowing down everyone's downloads... and that has a real impact on user experience that could have all those projects shifting to a different provider. Getting a big chunk (250mbps? 500mbps?) of dedicated bandwidth will really help you on the way to $75k annually.
tixe 7y ago
Can you explain the Netflix math or is there an example you can point me to?

sz4kerto 7y ago

What we do at my company: everything is in the cloud except build farm because renting dedicated servers is much cheaper, we need lots of them and we don't care if one drops out. So we have 32c/256G boxes for running tests (a full suite finishes in 2 hrs on a box like that), they cost something like $300/mo.

zozbot234 7y ago
> we don't care if one drops out. Are you comparing with spot/preemptable instances? They can be a lot cheaper than 'standard' cloud services.

jcrawfordor 7y ago

Colocation would shift a good portion of the annual expense to an up-front investment in hardware (as well as setup, shipping, install fees). It would also increase the required admin time. It's quite possible that they could save some money by going this way, but it's by no means going to just fix their problem. In a way it would exacerbate it due to the capex for hardware.

StavrosK 7y ago
They don't have to buy their own servers, renting from Hetzner would probably be much cheaper than AWS or Gitlab, and you can automate installing the runner on the server so your administration needs get reduced to basically "reformat the server and run this provisioning script". I'm sure there are things I'm not taking into account, but it doesn't seem that bad.
- jcrawfordor 7y ago
  Their issue is GitLab itself, not the runner. And sure, Gitlab is available in forms like a Docker image, but as someone who managed Gitlab for years... it's reasonably high effort. Additionally, I would not trust Hetzner with anything that gets this much usage. They're very cheap, and they're very cheap for a reason. When buying colo for a busy application, it's not unusual for bandwidth to cost more than the power and rack. All in all I'd expect them to be looking at at least $2k a month or $24k a year for power/rack/bandwidth. This would be something like an average, there are lower rates available in markets with cheap connectivity but then if the equipment isn't local you end up spending more on smart hands. In many places the rate would be higher. Then up-front hardware purchase could easily be $50k and probably more. I wouldn't expect to see savings at all the first year, I'd expect it to cost a chunk more. Of course that's not quite fair, the hardware purchase amortizes over at least a few years, but when you're struggling to pay the bills that doesn't make you feel any better. Renting hardware is an option, but will still cost you more than colo on year for year for equivalent capacity. Dedicated servers from a reputable source start at a couple hundred a month and quickly go up from there. It sounds like their situation will require HA. I'm just waving my hands here based on my guess of how much usage they're seeing for the $75k they estimate in AWS spend, but I have a quote in my inbox right now from a reputable datacenter operator for $18k per year just for rack/power/x-connect, that's before bandwidth, which I'm going to be paying two different providers for for diversity (and this is still worse connectivity than AWS gets you). That's a full rack, which is a lot of room, but the power/xconnect/bandwidth part of that basically don't change as you go to partial racks so the savings are not really that big. Quality colocation costs real money, and a gitlab instance with that kind of usage shouldn't be running out of a bargain-basement provider.
  - StavrosK 7y ago
    That's very informative, thank you. It's still a third of the price, though, if your calculations are ballpark correct, which is nothing to scoff at.
marcinzm 7y ago
Their main cost is outgoing bandwidth, AWS outgoing bandwidth is really expensive.
- jcrawfordor 7y ago
  Yeah, but bandwidth in a DC is pretty expensive too, it's just billed on a totally different basis. That can actually make these things a bit more complicated because if you're paying for bandwidth on a 95%ile or dedicated basis you end up managing your budget by slowing down everyone's downloads... and that has a real impact on user experience that could have all those projects shifting to a different provider. Getting a big chunk (250mbps? 500mbps?) of dedicated bandwidth will really help you on the way to $75k annually.

tixe 7y ago

Can you explain the Netflix math or is there an example you can point me to?