6 ms·

So Gitlab is in a bit of a strange position here. Sticking to a traditional filesystem interface (distributed under the hood) seems stupid at first. Surely ther

So Gitlab is in a bit of a strange position here. Sticking to a traditional filesystem interface (distributed under the hood) seems stupid at first. Surely there are better technical solutions. However, considering they make money out of private installs of gitlab, it makes sense to keep gitlab.com as an eat-your-own-dog-food-at-scale environment. Necessary for them to keep experience with large installs of gitlab. If one of their customers that run on-prem has performance issues they can't just say: gitlab.com uses a totally different architecture so you're on your own. They need gitlab.com to be as close as possible to the standard product. Pivotal does the same thing with Pivotal Web Services, their public Cloud Foundry solution. All of their money is made in Pivotal Cloud Foundry (private installs). From a business perspective, private installs are a way of distributed computing. Pretty clever, and good way of minimizing risk.

jtwaleson 10y ago
A risk is of course that your public environment is so much bigger than your biggest private install, that it doesn't make sense anymore to keep the same technology for both tiers. I think both PWS and gitlab.com suffer from this.
- cryptarch 10y ago
  Does Gitlab support some kind of federation between instances? If so, they could (in theory) split Gitlab.com into a bunch shards which as a whole match the properties of N% of enterprise users. That'd be a pretty cool way to avoid the different-in-scale problem (although you might still run into novel problems as you're now the Gitlab instance with the most shards..).
  - woot01 10y ago
    this is what Salesforce does - they have publicly visible shards in the url (na11, na14, eu23, etc) and login.salesforce.com redirects you to your shard when it figures out who you are.
    - londons_explore 10y ago
      For anyone considering this, reconsider. It bit me many times. In general, over time, load on some shards will increase while others decrease. Migrating a customer from one shard to another will likley cause a short outage for them, and many bugs down the line when they've bookmarked all kinds of things.
      - eonw 10y ago
        could that not be resolved with a reverse proxy or load balancer trickery? ie; hide the shard name externally.
      - brazzledazzle 10y ago
        You need to preserve the name<->customer association and maintain another key that you can use to split traffic at the LB in case a customer outgrows their shard. But personally I think it looks hokey and should not be something a customer sees anywhere but perhaps a developer tool or sniffer.
merb 10y ago
well they can probably include a better storage layer. at the moment you can just setup multiple mount points, but I guess it would be superior to actually setup a moint point or object storage. I'm pretty sure that you can put git into something like s3.
- jtwaleson 10y ago
  s3fs? ;) A) I think it would be kind of difficult to store git on S3 (the magic would be in the caching / consistency layer to keep it performant) B) it would make sense for their customers that run on AWS. However, many of them just run on a single VM / physical server.
  - merb 10y ago
    well I just said s3 since it's the best known, but I guess it would be feasible to run on object storage. I heard that libgit2 can actually change his storage layer.. btw here is the alibaba approach: http://de.slideshare.net/MinqiPan/how-we-scaled-git-lab-for-a-30k-employee-company http://de.slideshare.net/MinqiPan/how-we-scaled-git-lab-for-... Edit: and I heard github uses gitrpc.
candiodari 10y ago
It's not just that. The cloud is far, far more expensive than bare metal, and that is under completely optimal financial conditions for the cloud providers (extremely low, even 0%, interest rates for the companies owning them, hence required return is only a few percent) Dedicated servers, and colocation are going to be far cheaper than the cloud, and worse, the savings directly related to the size of the infrastructure you need. That, combined with the fact that even the very best of virtualization on shared resources still kills 20-30% of performance. So there's 3 things you can use the cloud for: 1) your company is totally fucked for IT management, and the cloud beats local because of incompetence (this is very common). And you're unwilling or unable to fix this. Or "your company doesn't focus on IT and never will" in MBA speak. 2) wildly swinging resource loads, where the peak capacity is only needed for 1-5% of time at most. 3) you expect to have vastly increasing resource requirements in a short time, and you're willing to pay a 5x-20x premium over dedi/colo to achieve that The thing I don't understand is that cloud has both a lower limit (cloud is (far) more expensive than web hosting, and having a VPS) that is an extremely common case, and far more expensive once you go over a certain capacity (doesn't matter which one, CPU, Network, Disk, ... all are far more expensive in the cloud). Even if you have wildly varying loads there's an upper limit to the resource needs where cloud becomes more expensive. The thing I don't understand is why so many people are doing this. I ran a mysql-in-the-cloud instance on Amazon for years, with a 300 Mb database, serving 10-50 qps to serve as a backend to a website, and a reporting server that ran on-premise. Cost ? 105-150 euros per month. We could have easily done that either locally or on a VPS or dedicated server for a tenth of that cost. Cloud moves a capital cost into an operational cost. This can be a boon or a disaster depending on your situation. You want to run an experiment that may or may not pan out ? Off the cloud you'll have spare capacity that you can use but don't really have to pay for. On the cloud cost controls will mean you can't use extra resources. You can't loan money from the bank ? The cloud(but also dedi providers) can still get you capacity, essentially allowing you to use their bank credit for a huge premium.
- agrajag 10y ago
  I think you're missing some use cases here. Cloud can be really helpful for prototyping systems that you may not even want running in a few months. Another use case would be where infrastructure costs are minor compared to dev and ops staff costs. If hosting on AWS makes your ops team 2x as productive at a 30% infrastructure markup that can be a steal.
user5994461 10y ago
> However, considering they make money out of private installs of gitlab, it makes sense to keep gitlab.com as an eat-your-own-dog-food-at-scale environment. Depends on how they bill. If they bill on a few private installs while giving unlimited storage & projects for free on the cloud, they're setting up themselves for bankruptcy. Gotta take care of the accounting. Having an unsustainable pricing is a classic mistake of web companies.
sytse 10y ago
Indeed this is a big factor. We want our customers and users to be able to use open source to host their repositories. That is one of the reasons to use Ceph instead of a proprietary storage solution such as Nutanix or NetApp.

jtwaleson 10y ago

A risk is of course that your public environment is so much bigger than your biggest private install, that it doesn't make sense anymore to keep the same technology for both tiers. I think both PWS and gitlab.com suffer from this.

cryptarch 10y ago
Does Gitlab support some kind of federation between instances? If so, they could (in theory) split Gitlab.com into a bunch shards which as a whole match the properties of N% of enterprise users. That'd be a pretty cool way to avoid the different-in-scale problem (although you might still run into novel problems as you're now the Gitlab instance with the most shards..).
- woot01 10y ago
  this is what Salesforce does - they have publicly visible shards in the url (na11, na14, eu23, etc) and login.salesforce.com redirects you to your shard when it figures out who you are.
  - londons_explore 10y ago
    For anyone considering this, reconsider. It bit me many times. In general, over time, load on some shards will increase while others decrease. Migrating a customer from one shard to another will likley cause a short outage for them, and many bugs down the line when they've bookmarked all kinds of things.
    - eonw 10y ago
      could that not be resolved with a reverse proxy or load balancer trickery? ie; hide the shard name externally.
    - brazzledazzle 10y ago
      You need to preserve the name<->customer association and maintain another key that you can use to split traffic at the LB in case a customer outgrows their shard. But personally I think it looks hokey and should not be something a customer sees anywhere but perhaps a developer tool or sniffer.

merb 10y ago

well they can probably include a better storage layer. at the moment you can just setup multiple mount points, but I guess it would be superior to actually setup a moint point or object storage. I'm pretty sure that you can put git into something like s3.

jtwaleson 10y ago
s3fs? ;) A) I think it would be kind of difficult to store git on S3 (the magic would be in the caching / consistency layer to keep it performant) B) it would make sense for their customers that run on AWS. However, many of them just run on a single VM / physical server.
- merb 10y ago
  well I just said s3 since it's the best known, but I guess it would be feasible to run on object storage. I heard that libgit2 can actually change his storage layer.. btw here is the alibaba approach: http://de.slideshare.net/MinqiPan/how-we-scaled-git-lab-for-a-30k-employee-company http://de.slideshare.net/MinqiPan/how-we-scaled-git-lab-for-... Edit: and I heard github uses gitrpc.

candiodari 10y ago

It's not just that. The cloud is far, far more expensive than bare metal, and that is under completely optimal financial conditions for the cloud providers (extremely low, even 0%, interest rates for the companies owning them, hence required return is only a few percent) Dedicated servers, and colocation are going to be far cheaper than the cloud, and worse, the savings directly related to the size of the infrastructure you need. That, combined with the fact that even the very best of virtualization on shared resources still kills 20-30% of performance. So there's 3 things you can use the cloud for: 1) your company is totally fucked for IT management, and the cloud beats local because of incompetence (this is very common). And you're unwilling or unable to fix this. Or "your company doesn't focus on IT and never will" in MBA speak. 2) wildly swinging resource loads, where the peak capacity is only needed for 1-5% of time at most. 3) you expect to have vastly increasing resource requirements in a short time, and you're willing to pay a 5x-20x premium over dedi/colo to achieve that The thing I don't understand is that cloud has both a lower limit (cloud is (far) more expensive than web hosting, and having a VPS) that is an extremely common case, and far more expensive once you go over a certain capacity (doesn't matter which one, CPU, Network, Disk, ... all are far more expensive in the cloud). Even if you have wildly varying loads there's an upper limit to the resource needs where cloud becomes more expensive. The thing I don't understand is why so many people are doing this. I ran a mysql-in-the-cloud instance on Amazon for years, with a 300 Mb database, serving 10-50 qps to serve as a backend to a website, and a reporting server that ran on-premise. Cost ? 105-150 euros per month. We could have easily done that either locally or on a VPS or dedicated server for a tenth of that cost. Cloud moves a capital cost into an operational cost. This can be a boon or a disaster depending on your situation. You want to run an experiment that may or may not pan out ? Off the cloud you'll have spare capacity that you can use but don't really have to pay for. On the cloud cost controls will mean you can't use extra resources. You can't loan money from the bank ? The cloud(but also dedi providers) can still get you capacity, essentially allowing you to use their bank credit for a huge premium.

agrajag 10y ago
I think you're missing some use cases here. Cloud can be really helpful for prototyping systems that you may not even want running in a few months. Another use case would be where infrastructure costs are minor compared to dev and ops staff costs. If hosting on AWS makes your ops team 2x as productive at a 30% infrastructure markup that can be a steal.

user5994461 10y ago

> However, considering they make money out of private installs of gitlab, it makes sense to keep gitlab.com as an eat-your-own-dog-food-at-scale environment. Depends on how they bill. If they bill on a few private installs while giving unlimited storage & projects for free on the cloud, they're setting up themselves for bankruptcy. Gotta take care of the accounting. Having an unsustainable pricing is a classic mistake of web companies.

sytse 10y ago

Indeed this is a big factor. We want our customers and users to be able to use open source to host their repositories. That is one of the reasons to use Ceph instead of a proprietary storage solution such as Nutanix or NetApp.