4 ms·

Gitlab _could_ be a brilliant SaaS product. It has many more features than github, Runner integration allows simple and easy building, instead of having to bust

Gitlab _could_ be a brilliant SaaS product. It has many more features than github, Runner integration allows simple and easy building, instead of having to bust out to circleCI. As a self hosted product, it is difficult to be beaten. but as a Saas product, its just horribly unreliable. In the last two months it has improved, but there are still outages every two weeks or so. (https://twitter.com/gitlabstatus https://twitter.com/gitlabstatus) I'm hoping that with this extra cash, they'll be hiring in some infrastructure people (people who will look at this in horror: https://about.gitlab.com/2016/12/11/proposed-server-purchase-for-gitlab-com/ https://about.gitlab.com/2016/12/11/proposed-server-purchase... ) who actually know how to make a stable platform using _proven "boring"_ stuff, not sexy slow and supposedly HA stuff like Ceph and forcing it to serve NFS.

dsumenkovic 8y ago
Thanks for your comment, we really appreciate the feedback. Our team is working hard on improving the availability and stability of the platform. Our goal is currently to achieve 99.95% availability on GitLab.com. According to Pingdom, over the last year our availability has been 99.81%, although this includes the large (multi-hour) maintenance window on 11 August 2018 when we migrated GitLab.com from Microsoft Azure cloud to Google Cloud Platform (GCP). Since the migration, our availability has improved greatly (caveat: we obviously have much less data than from Azure). Using the data publicly available from Pingdom, here are some interesting stats: Mean-time between outages in Azure (September 2017 through August 11 2018): 1.3 days Mean-time between outages in GCP (September 2017 through August 12 2018): 7.3 days (if you ignore some problems we experienced on the first day after the migration, this rises to 12 days!) Since, migrating to GCP, our overall availability, according to Pingdom, has risen to 99.92%. Again, if you exclude the few hiccups that we experienced on the Monday immediately following the migration, this rises to 99.97%. There are multiple reasons for this improvement. We chose Google Cloud Platform because we believe that they offer the most reliable cloud platform for our workload, particularly as we move towards running GitLab.com in Kubernetes. It is worth pointing out that we also used the migration as an opportunity to improve our infrastructure, simplify some components and otherwise make things more stable and more observable. Finally, we've also been focusing on building the infrastructure team up, having hired many new team members over the past few months. This means that the team has been better able to balance the job of running GitLab.com with making it more stable.
- _salmon 8y ago
  That's all well and good, but by your own admission you're still having outages nearly once a week on average
mmt 8y ago
> hiring in some infrastructure people Part of the problem is that these, at least the ones making the fundamental decisions, are likely to be software people, with not enough ops/sysadmin experience. > who actually know how to make a stable platform using _proven "boring"_ stuff, not sexy The ones who do may have gotten bored of it, themselves, and no longer admit it, instead leaning toward as much of the "Dev" part of DevOps (as a title) as possible, since that's where the excitement (and decision making power) tends to be. The rest of us are borderline unemployable.
- KaiserPro 8y ago
  I don't quite understand that statement. With the exception of my time at a financial news paper, virtually all professional life has been trying to make highly scalable pipelines that run as smoothly as possible. In any project, you have about three innovation tokens to spend. Each token will cost you time and momentum, however they may give you lots of reward. Personally I like spending them on the bits of the software that give you the USP, Not something thats already solved. To illustrate the point: using loads of d blades with internal storage to create a Ceph storage mesh == expensive nightmare and slow, especially as you are going to be serving it over NFS. Storage bandwidth is contending serving bandwidth. Not only that you now have three SPOFs the NFS servers, the ceph cluster (no, its really not HA) and the blade fabric. three or four real file servers with proper jbods/raid arrays if you want to go down the NFS route. the smaller the better, spread the load amongst them evenly, meaning that if one goes down it degrades but not totally wipes out access. Better yet to use SAS/FC to route block access, and either pay for hardware duplication, or write the state to two different LUNs (its git after all, its fairly simple to duplicate.) failing that, just us VMware and some decent block storage, and let it figure out how to do it. It has rather reasonable HA clustering, which don't need software changes. or just host it in the cloud. Which is what they did in the end.
  - mmt 8y ago
    > I don't quite understand that statement. I didn't make a single statement, but I also made what is more of an accusation (i.e. stating the opinion of others I disagree with) than my own assertion. > Personally I like spending them on the bits of the software that give you the USP, Not something thats already solved. I'm suggesting that most people (or at least in my own "bubble", as it were) don't differentiate. Re-inventing the wheel is just fine to them, so long as it's fun. That said, I'm not convinced large (enough) "global" filesystems are a solved problem. Of course, GitLab doesn't necessarily need such a thing, but implementing that kind of storage "sharding" (for lack of a better word) may well be spending an innovation token, too. > or just host it in the cloud. Except that doesn't magically solve any of the architectural/operational problems. For all we know they're still doing Ceph-with-NFS, just paying much more to do so, without the ability to optimize the hardware for cost and/or performance.

dsumenkovic 8y ago

Thanks for your comment, we really appreciate the feedback. Our team is working hard on improving the availability and stability of the platform. Our goal is currently to achieve 99.95% availability on GitLab.com. According to Pingdom, over the last year our availability has been 99.81%, although this includes the large (multi-hour) maintenance window on 11 August 2018 when we migrated GitLab.com from Microsoft Azure cloud to Google Cloud Platform (GCP). Since the migration, our availability has improved greatly (caveat: we obviously have much less data than from Azure). Using the data publicly available from Pingdom, here are some interesting stats: Mean-time between outages in Azure (September 2017 through August 11 2018): 1.3 days Mean-time between outages in GCP (September 2017 through August 12 2018): 7.3 days (if you ignore some problems we experienced on the first day after the migration, this rises to 12 days!) Since, migrating to GCP, our overall availability, according to Pingdom, has risen to 99.92%. Again, if you exclude the few hiccups that we experienced on the Monday immediately following the migration, this rises to 99.97%. There are multiple reasons for this improvement. We chose Google Cloud Platform because we believe that they offer the most reliable cloud platform for our workload, particularly as we move towards running GitLab.com in Kubernetes. It is worth pointing out that we also used the migration as an opportunity to improve our infrastructure, simplify some components and otherwise make things more stable and more observable. Finally, we've also been focusing on building the infrastructure team up, having hired many new team members over the past few months. This means that the team has been better able to balance the job of running GitLab.com with making it more stable.

_salmon 8y ago
That's all well and good, but by your own admission you're still having outages nearly once a week on average

mmt 8y ago

> hiring in some infrastructure people Part of the problem is that these, at least the ones making the fundamental decisions, are likely to be software people, with not enough ops/sysadmin experience. > who actually know how to make a stable platform using _proven "boring"_ stuff, not sexy The ones who do may have gotten bored of it, themselves, and no longer admit it, instead leaning toward as much of the "Dev" part of DevOps (as a title) as possible, since that's where the excitement (and decision making power) tends to be. The rest of us are borderline unemployable.

KaiserPro 8y ago
I don't quite understand that statement. With the exception of my time at a financial news paper, virtually all professional life has been trying to make highly scalable pipelines that run as smoothly as possible. In any project, you have about three innovation tokens to spend. Each token will cost you time and momentum, however they may give you lots of reward. Personally I like spending them on the bits of the software that give you the USP, Not something thats already solved. To illustrate the point: using loads of d blades with internal storage to create a Ceph storage mesh == expensive nightmare and slow, especially as you are going to be serving it over NFS. Storage bandwidth is contending serving bandwidth. Not only that you now have three SPOFs the NFS servers, the ceph cluster (no, its really not HA) and the blade fabric. three or four real file servers with proper jbods/raid arrays if you want to go down the NFS route. the smaller the better, spread the load amongst them evenly, meaning that if one goes down it degrades but not totally wipes out access. Better yet to use SAS/FC to route block access, and either pay for hardware duplication, or write the state to two different LUNs (its git after all, its fairly simple to duplicate.) failing that, just us VMware and some decent block storage, and let it figure out how to do it. It has rather reasonable HA clustering, which don't need software changes. or just host it in the cloud. Which is what they did in the end.
- mmt 8y ago
  > I don't quite understand that statement. I didn't make a single statement, but I also made what is more of an accusation (i.e. stating the opinion of others I disagree with) than my own assertion. > Personally I like spending them on the bits of the software that give you the USP, Not something thats already solved. I'm suggesting that most people (or at least in my own "bubble", as it were) don't differentiate. Re-inventing the wheel is just fine to them, so long as it's fun. That said, I'm not convinced large (enough) "global" filesystems are a solved problem. Of course, GitLab doesn't necessarily need such a thing, but implementing that kind of storage "sharding" (for lack of a better word) may well be spending an innovation token, too. > or just host it in the cloud. Except that doesn't magically solve any of the architectural/operational problems. For all we know they're still doing Ceph-with-NFS, just paying much more to do so, without the ability to optimize the hardware for cost and/or performance.