5 ms·

FreeDesktop.org financial situation regarding cloud hosting for Gitlab

58 points by mroche 7y ago

anonsivalley652 7y ago
Why don't they host it themselves in a colo? Doing like Netflix did by paying AWS millions is wasteful and insane when managed colo is cheaper and self-managed colo is cheaper still. ByteMark.co.uk and pair.com are both great vendors that could save them money.
- sz4kerto 7y ago
  What we do at my company: everything is in the cloud except build farm because renting dedicated servers is much cheaper, we need lots of them and we don't care if one drops out. So we have 32c/256G boxes for running tests (a full suite finishes in 2 hrs on a box like that), they cost something like $300/mo.
  - zozbot234 7y ago
    > we don't care if one drops out. Are you comparing with spot/preemptable instances? They can be a lot cheaper than 'standard' cloud services.
- jcrawfordor 7y ago
  Colocation would shift a good portion of the annual expense to an up-front investment in hardware (as well as setup, shipping, install fees). It would also increase the required admin time. It's quite possible that they could save some money by going this way, but it's by no means going to just fix their problem. In a way it would exacerbate it due to the capex for hardware.
  - StavrosK 7y ago
    They don't have to buy their own servers, renting from Hetzner would probably be much cheaper than AWS or Gitlab, and you can automate installing the runner on the server so your administration needs get reduced to basically "reformat the server and run this provisioning script". I'm sure there are things I'm not taking into account, but it doesn't seem that bad.
    - jcrawfordor 7y ago
      Their issue is GitLab itself, not the runner. And sure, Gitlab is available in forms like a Docker image, but as someone who managed Gitlab for years... it's reasonably high effort. Additionally, I would not trust Hetzner with anything that gets this much usage. They're very cheap, and they're very cheap for a reason. When buying colo for a busy application, it's not unusual for bandwidth to cost more than the power and rack. All in all I'd expect them to be looking at at least $2k a month or $24k a year for power/rack/bandwidth. This would be something like an average, there are lower rates available in markets with cheap connectivity but then if the equipment isn't local you end up spending more on smart hands. In many places the rate would be higher. Then up-front hardware purchase could easily be $50k and probably more. I wouldn't expect to see savings at all the first year, I'd expect it to cost a chunk more. Of course that's not quite fair, the hardware purchase amortizes over at least a few years, but when you're struggling to pay the bills that doesn't make you feel any better. Renting hardware is an option, but will still cost you more than colo on year for year for equivalent capacity. Dedicated servers from a reputable source start at a couple hundred a month and quickly go up from there. It sounds like their situation will require HA. I'm just waving my hands here based on my guess of how much usage they're seeing for the $75k they estimate in AWS spend, but I have a quote in my inbox right now from a reputable datacenter operator for $18k per year just for rack/power/x-connect, that's before bandwidth, which I'm going to be paying two different providers for for diversity (and this is still worse connectivity than AWS gets you). That's a full rack, which is a lot of room, but the power/xconnect/bandwidth part of that basically don't change as you go to partial racks so the savings are not really that big. Quality colocation costs real money, and a gitlab instance with that kind of usage shouldn't be running out of a bargain-basement provider.
      - StavrosK 7y ago
        That's very informative, thank you. It's still a third of the price, though, if your calculations are ballpark correct, which is nothing to scoff at.
  - marcinzm 7y ago
    Their main cost is outgoing bandwidth, AWS outgoing bandwidth is really expensive.
    - jcrawfordor 7y ago
      Yeah, but bandwidth in a DC is pretty expensive too, it's just billed on a totally different basis. That can actually make these things a bit more complicated because if you're paying for bandwidth on a 95%ile or dedicated basis you end up managing your budget by slowing down everyone's downloads... and that has a real impact on user experience that could have all those projects shifting to a different provider. Getting a big chunk (250mbps? 500mbps?) of dedicated bandwidth will really help you on the way to $75k annually.
- tixe 7y ago
  Can you explain the Netflix math or is there an example you can point me to?
stonogo 7y ago
Maybe it's time to acknowledge that fd.o is primarily a Red Hat operation, and ask Red Hat to step up and serve the community?
- astrodust 7y ago
  You mean IBM which could give this project a huge cloud services credit with the click of a mouse.
lucb1e 7y ago
Would switching from GitLab to something like Gitea/Gogs cut down on those costs? The GitLab system requirements start at multiple gigabytes of RAM and CPU cores for almost idle installations whereas a small Gitea I'm running takes <100MB RAM and next to zero CPU. Most people I know that self-host (i.e. privately, not commercially) a git server with issue tracker don't run GitLab due to the resource hogging. But I don't know how it scales when you put more load on Gitea vs the same load on GitLab -- does anyone have experience with that?
- stonogo 7y ago
  It says in the message it's storing and serving build artifacts that is driving costs. This means the expense is in disk and bandwidth, not CPU and RAM. Switching to Gogs would cut down on the resources which are not identified as the problem.
  - tiew9Vii 7y ago
    They probably could use some cheap blob storage like Backblaze B2 which is very cheap ($0.005 GB/month storage, $0.01 G download) for storing artifacts then put Cloudflare in-front of it. If artifacts are treated as immutable most of the traffic will be from the CDN and as they are immutable can have a far future cache-control.
- earenndil 7y ago
  The cost is bandwidth, not ram/storage.
CameronNemo 7y ago
This is exactly why Void Linux uses GitHub and Travis. Hard to pay for one's own CI.
chx 7y ago
> With reasonable estimates for continued growth we're expecting hosting expenses totalling 75k USD this year, How does that happen? My understanding here is the servers are doing something ephemeral and so... uh, why this isn't on Hetzner SB? I do not know of course whom they are using for hosting but 35 USD gets you a decent quad core 32GB RAM server there which is very hard to beat.
- earenndil 7y ago
  Problem is not server specs, but bandwidth.
  - chx 7y ago
    Great, Hetzner has unlimited bandwidth (well, 320TB since the port is 1gbps).
    - robjan 7y ago
      If they start using 320TB / month they will quickly enter "Fair Usage Policy" territory. At that rate of bandwidth usage, Hetzner would be taking a massive loss and other clients would probably be impacted.
      - chx 7y ago
        [citation needed] https://www.hetzner.com/news/traffic-limit/ https://www.hetzner.com/news/traffic-limit/ that's not what this says. Nothing at https://www.hetzner.com/rechtliches/root-server/ https://www.hetzner.com/rechtliches/root-server/ https://www.hetzner.com/rechtliches/agb https://www.hetzner.com/rechtliches/agb In fact, people have asked support before https://i.imgur.com/usd27c2.png https://i.imgur.com/usd27c2.png
mrpippy 7y ago
There's more details 2 messages later on what the cost is for: > Some clarification I got from Daniel in a private conversation, since I was confused about what the money was paying for exactly: > We're paying 75K USD for the bandwidth to transfer data from the GitLab cloud instance. i.e., for viewing the https site, for cloning/updating git repos, and for downloading CI artifacts/images to the testing machines (AFAIU). And more details on their infrastructure in general is at https://www.freedesktop.org/wiki/Infrastructure/ https://www.freedesktop.org/wiki/Infrastructure/ gitlab.freedesktop.org is hosted in GCE us-east1. One runner for CI jobs is also in us-east1, but there are also (donated) runners at Hetzner and Packet, and a lot of bandwidth goes toward communicating with those. Apparently even if they disabled CI, it would still be $30K/year, I guess for hosting and web/git traffic. Not a good situation.
- Symbiote 7y ago
  > All the other services are run on a small fleet of machines hosted at Portland State University. Would the university host the GitLab instance? (Or a VM for it, or whatever.) The bandwidth cost to them should be very small. I run infrastructure hosted at a university, and they change around $500/year for a 10Gb/s uplink. I think this is just the cost price for them.
- Slartie 7y ago
  This is laughable. Just renting another box at Hetzner with the specs necessary for their Gitlab would be around $100-$150 per month. That is maybe 1-2k per year. With 10tb/month traffic included already, IIRC, additional at a very reasonable price, and traffic to their already existing Hetzner runner box being non-metered to begin with. But woooh, that's probably not cloudey enough.
deleted 7y ago
[deleted]
drenginian 7y ago
Digital Ocean bandwidth is 1 cent per gigabyte.

58 points by mroche 7y ago

Why don't they host it themselves in a colo? Doing like Netflix did by paying AWS millions is wasteful and insane when managed colo is cheaper and self-managed colo is cheaper still. ByteMark.co.uk and pair.com are both great vendors that could save them money.

sz4kerto 7y ago
What we do at my company: everything is in the cloud except build farm because renting dedicated servers is much cheaper, we need lots of them and we don't care if one drops out. So we have 32c/256G boxes for running tests (a full suite finishes in 2 hrs on a box like that), they cost something like $300/mo.
- zozbot234 7y ago
  > we don't care if one drops out. Are you comparing with spot/preemptable instances? They can be a lot cheaper than 'standard' cloud services.
jcrawfordor 7y ago
Colocation would shift a good portion of the annual expense to an up-front investment in hardware (as well as setup, shipping, install fees). It would also increase the required admin time. It's quite possible that they could save some money by going this way, but it's by no means going to just fix their problem. In a way it would exacerbate it due to the capex for hardware.
- StavrosK 7y ago
  They don't have to buy their own servers, renting from Hetzner would probably be much cheaper than AWS or Gitlab, and you can automate installing the runner on the server so your administration needs get reduced to basically "reformat the server and run this provisioning script". I'm sure there are things I'm not taking into account, but it doesn't seem that bad.
  - jcrawfordor 7y ago
    Their issue is GitLab itself, not the runner. And sure, Gitlab is available in forms like a Docker image, but as someone who managed Gitlab for years... it's reasonably high effort. Additionally, I would not trust Hetzner with anything that gets this much usage. They're very cheap, and they're very cheap for a reason. When buying colo for a busy application, it's not unusual for bandwidth to cost more than the power and rack. All in all I'd expect them to be looking at at least $2k a month or $24k a year for power/rack/bandwidth. This would be something like an average, there are lower rates available in markets with cheap connectivity but then if the equipment isn't local you end up spending more on smart hands. In many places the rate would be higher. Then up-front hardware purchase could easily be $50k and probably more. I wouldn't expect to see savings at all the first year, I'd expect it to cost a chunk more. Of course that's not quite fair, the hardware purchase amortizes over at least a few years, but when you're struggling to pay the bills that doesn't make you feel any better. Renting hardware is an option, but will still cost you more than colo on year for year for equivalent capacity. Dedicated servers from a reputable source start at a couple hundred a month and quickly go up from there. It sounds like their situation will require HA. I'm just waving my hands here based on my guess of how much usage they're seeing for the $75k they estimate in AWS spend, but I have a quote in my inbox right now from a reputable datacenter operator for $18k per year just for rack/power/x-connect, that's before bandwidth, which I'm going to be paying two different providers for for diversity (and this is still worse connectivity than AWS gets you). That's a full rack, which is a lot of room, but the power/xconnect/bandwidth part of that basically don't change as you go to partial racks so the savings are not really that big. Quality colocation costs real money, and a gitlab instance with that kind of usage shouldn't be running out of a bargain-basement provider.
    - StavrosK 7y ago
      That's very informative, thank you. It's still a third of the price, though, if your calculations are ballpark correct, which is nothing to scoff at.
- marcinzm 7y ago
  Their main cost is outgoing bandwidth, AWS outgoing bandwidth is really expensive.
  - jcrawfordor 7y ago
    Yeah, but bandwidth in a DC is pretty expensive too, it's just billed on a totally different basis. That can actually make these things a bit more complicated because if you're paying for bandwidth on a 95%ile or dedicated basis you end up managing your budget by slowing down everyone's downloads... and that has a real impact on user experience that could have all those projects shifting to a different provider. Getting a big chunk (250mbps? 500mbps?) of dedicated bandwidth will really help you on the way to $75k annually.
tixe 7y ago
Can you explain the Netflix math or is there an example you can point me to?

stonogo 7y ago

Maybe it's time to acknowledge that fd.o is primarily a Red Hat operation, and ask Red Hat to step up and serve the community?

astrodust 7y ago
You mean IBM which could give this project a huge cloud services credit with the click of a mouse.

lucb1e 7y ago

Would switching from GitLab to something like Gitea/Gogs cut down on those costs? The GitLab system requirements start at multiple gigabytes of RAM and CPU cores for almost idle installations whereas a small Gitea I'm running takes <100MB RAM and next to zero CPU. Most people I know that self-host (i.e. privately, not commercially) a git server with issue tracker don't run GitLab due to the resource hogging. But I don't know how it scales when you put more load on Gitea vs the same load on GitLab -- does anyone have experience with that?

stonogo 7y ago
It says in the message it's storing and serving build artifacts that is driving costs. This means the expense is in disk and bandwidth, not CPU and RAM. Switching to Gogs would cut down on the resources which are not identified as the problem.
- tiew9Vii 7y ago
  They probably could use some cheap blob storage like Backblaze B2 which is very cheap ($0.005 GB/month storage, $0.01 G download) for storing artifacts then put Cloudflare in-front of it. If artifacts are treated as immutable most of the traffic will be from the CDN and as they are immutable can have a far future cache-control.
earenndil 7y ago
The cost is bandwidth, not ram/storage.

CameronNemo 7y ago

This is exactly why Void Linux uses GitHub and Travis. Hard to pay for one's own CI.

chx 7y ago

> With reasonable estimates for continued growth we're expecting hosting expenses totalling 75k USD this year, How does that happen? My understanding here is the servers are doing something ephemeral and so... uh, why this isn't on Hetzner SB? I do not know of course whom they are using for hosting but 35 USD gets you a decent quad core 32GB RAM server there which is very hard to beat.

earenndil 7y ago
Problem is not server specs, but bandwidth.
- chx 7y ago
  Great, Hetzner has unlimited bandwidth (well, 320TB since the port is 1gbps).
  - robjan 7y ago
    If they start using 320TB / month they will quickly enter "Fair Usage Policy" territory. At that rate of bandwidth usage, Hetzner would be taking a massive loss and other clients would probably be impacted.
    - chx 7y ago
      [citation needed] https://www.hetzner.com/news/traffic-limit/ https://www.hetzner.com/news/traffic-limit/ that's not what this says. Nothing at https://www.hetzner.com/rechtliches/root-server/ https://www.hetzner.com/rechtliches/root-server/ https://www.hetzner.com/rechtliches/agb https://www.hetzner.com/rechtliches/agb In fact, people have asked support before https://i.imgur.com/usd27c2.png https://i.imgur.com/usd27c2.png

mrpippy 7y ago

There's more details 2 messages later on what the cost is for: > Some clarification I got from Daniel in a private conversation, since I was confused about what the money was paying for exactly: > We're paying 75K USD for the bandwidth to transfer data from the GitLab cloud instance. i.e., for viewing the https site, for cloning/updating git repos, and for downloading CI artifacts/images to the testing machines (AFAIU). And more details on their infrastructure in general is at https://www.freedesktop.org/wiki/Infrastructure/ https://www.freedesktop.org/wiki/Infrastructure/ gitlab.freedesktop.org is hosted in GCE us-east1. One runner for CI jobs is also in us-east1, but there are also (donated) runners at Hetzner and Packet, and a lot of bandwidth goes toward communicating with those. Apparently even if they disabled CI, it would still be $30K/year, I guess for hosting and web/git traffic. Not a good situation.

Symbiote 7y ago
> All the other services are run on a small fleet of machines hosted at Portland State University. Would the university host the GitLab instance? (Or a VM for it, or whatever.) The bandwidth cost to them should be very small. I run infrastructure hosted at a university, and they change around $500/year for a 10Gb/s uplink. I think this is just the cost price for them.
Slartie 7y ago
This is laughable. Just renting another box at Hetzner with the specs necessary for their Gitlab would be around $100-$150 per month. That is maybe 1-2k per year. With 10tb/month traffic included already, IIRC, additional at a very reasonable price, and traffic to their already existing Hetzner runner box being non-metered to begin with. But woooh, that's probably not cloudey enough.

deleted 7y ago

[deleted]

drenginian 7y ago

Digital Ocean bandwidth is 1 cent per gigabyte.