19 ms·

Why we are not leaving the cloud

350 points by happy-go-lucky 10y ago

StavrosK 10y ago
I'm really glad you guys decided to listen to the experienced commenters' advice and change your mind, rather than obstinately plowing on with what seems to be a bad decision.
Dangeranger 10y ago
It's interesting that the first half of the explanation is largely quotes from a prior HN post's responses. Does anyone else feel a bit weird seeing off hand comments getting quoted in the explanation for a business decision? I guess we should all get more accustomed to our public input carrying weight in the zeitgeist. Gitlab's "develop in the open" nature really shows through here. I am not saying that's bad, it's just so different than most startups and established businesses.
- gtaylor 10y ago
  I don't think they were using this as justification or explanation in their eventual reversal. I think they provided these quotes as interesting tidbits. Kind of like pull-out quotes on an article. You can ignore them entirely and still get the gist of the post. On a more general note: It's got to be incredibly hard to do what GitLab does with their extreme transparency. I feel like we have to be careful about reading too deeply into things and nitpicking their culture or process. HN is full of "expert" advice, much of it being terrible. They weighed their options, invited feedback, then made a decision. I appreciate what GitLab does in being so transparent. None of us are owed explanations or insight into how they operate, yet they go out of their way to provide it. Kudos to syste and his team!
  - sytse 10y ago
    Thanks. The most useful advice was shared in private. People coming to our office to share war stories of regretting moving to bare metal. I also received dozens of people sending direct messages via twitter because they couldn't share their stories publicly. Some of the best advice are things that we can't share publicly. For example a major company going bare-metal and then spending a lot of time to set up an authorization system that you get for free with AWS IAM.
    - dustinmoris 10y ago
      Did you have to sign a NDA? Because if people send you random advice via email after reading a blog post then this sounds to me very much like "publicly sharing" their experience...
      - shshhdhs 10y ago
        I disagree. It's a private email. Gitlab can ask permission to post, but it would be impolite and perhaps unethical to assume a private email is now "public". If they wanted it public, they could have chosen to provide a comment on HN, Twitter, etc.
        sytse 10y ago
        Exactly, everything is private unless it was posted in public by the author or explicit permission was given. We have a transparency value, but we understand that other organizations are different. And even we assume that our private communication will stay private.
        dustinmoris 10y ago
        Apparently according to syste some of the best arguments came from emails, so he could have put those arguments anonymised and trimmed down to the main points into the blog post, which would have respected the company's privacy while still sharing good arguments, unless there was a NDA which would prohibit that. So I disagree with you, because you look at this very black and white while the reality is grey.
- hackerboos 10y ago
  This echochamber has a profound and mostly unwarranted impact on people, myself included.
- 234dd57d2c8dba 10y ago
  No surprise here. Honestly, their team seem to be lower skill or less experienced than I'd have thought. I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy. The lack of automation, especially around validating db backups, failover (not having the failover process scripted and tested is _begging_ to have a nightmare at 2am where you're reading documentation on how to fail over a db), etc. The simple thing of having your hostname / $PS1 say the machine's purpose could have stopped this. All prod machines have a bright red PS1 and a clear name of <type>-<service>-<prod/dev/etc>-<region>-<dc>.corpnet in my setups. All of this is reflected in their discussion style, meeting style, etc. Ad-hoc, not very carefully designed, random off-hand comments. Obviously a young team with a lot to learn. Nothing wrong with that, but a lot of customers are relying on their skills! Learn quick!
  - kogepathic 10y ago
    > I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy. I do agree with you it reflects poorly on GitLab for only having a primary and replica, with broken backups. BUT, they are a startup, and they need to be laser focused on growing the company and securing funding for the next quarter so they can keep the lights on. This kind of pants on fire growth in a startup often (always?) comes at the cost of redundancy and best practices. If you stop to make your platform bulletproof instead of the new features you promised customers/investors, you die. I'm not saying this is an excuse for them to permanently shrug off making their platform more redundant and engaging in best practices. But, as with many startups, they're focused on delivering features as fast as possible to grow their user base. I think, and hope, that their recent outage has been the experience they needed to prioritize their shift toward more redundancy and best practices.
    - 234dd57d2c8dba 10y ago
- timanglade 10y ago
  We quote a lot of responses from HN but that doesn’t mean it’s the only thing that led us to this decision. I was personally involved in a lot of private conversations about this, with the executive team, with investors and with people who had gone through this exercise before, and they all had as much if not more impact on our final decision than the HN thread did. It’s just harder to quote them than referencing additional (& similar) opinions noted in a public forum.
  - Dangeranger 10y ago
    That is good to hear. My comment was not meant as a negative criticism of your process, although I can see how it could be interpreted that way. It was intended more as an observation about the openness that Gitlab brings to the process of design and decision.
    - timanglade 10y ago
      No worries! I didn’t perceive your comment as negative at all — I just took it as an opportunity to clarify something that we should have made more clear in the blogpost itself really. Thanks for bringing this up and for doing so in a constructive way!
- Rapzid 10y ago
  Well... Here is something very interesting. Myself and others were "cautioning" very heavily against moving to self-hosted gear. It wasn't until their CTO talked to somebody in person he had respect for that they flipped over to staying in the cloud. This to me calls into question the entire idea of crowd-sourcing decisions and plans like this. It was a single person who moved the needle, and now there is retroactive agreement(and quoting) with the people who were against it from the start. If it went the other way there would be plenty of quotes to pull from of people supporting the move to dedicated.
  - degenerate 10y ago
    Recommending someone to build in the cloud is an easy and safe recommendation to make. Therefore, more people will go that route when asked what you should do. Pretend for a second you plucked someone from a royal family that never prepared their own food, and they asked you how to make food. Are you going to recommend they purchase land, start a farm, grown their own vegetables, and raise their own cattle? Hell no. You tell them to go to the grocery store (cloud) and if they are desperate, they can purchase some pre-made food at restaurants (SaaS providers). When you don't know anything about a person's exact needs or abilities, making the safest recommendation is easiest, and therefore it proliferates. I hate the cloud but I know how to do what I want on my own without it. You don't know that about other people.
    - cookiecaper 10y ago
      I don't think the cloud is the grocery store. It doesn't give you a pre-packaged solution that you can just rip open and immediately access. Rather, the cloud is like leasing the land from an owner. You still have to till, plant, maintain, and harvest, but you have to pay an annual rent to do so. There are a few things that the landowner is responsible to provide, but most of the day-to-day maintenance responsibility is on the lessee. People generally recognize that while renting is sometimes necessary, it's much nicer to be an owner. We really need to stop this belief that the cloud is a magic thing that can fix your problems. It splits out the responsibility for actual, real-life hardware maintenance to a hosting company, but other than that, the responsibilities, including the responsibility to take backups, set up security, and design a fault-tolerant application, remain in the cloud user's court. Cloud is just a way to rent a virtual server; you still have to make the server work correctly.
- ReidZB 10y ago
  When I evaluate any tech project for use in production, I always give HN a cursory search for materials. It's not always fruitful, but in the cases where I stumble across something big in the HN space, I pick up valuable intel here... I still try to research elsewhere too, just to cover my bases, but HN is quite helpful as an archive of knowledge. Especially knowledge about running something in production.
AgentK20 10y ago
A bit disappointed in the article, as the bulk of it is simply quoting various viewpoints, some of which disagree with each other. I'd be much more interested in the thought process that occurred for them to reach the conclusion they did. What internal doubts did they raise? What possible solutions to those doubts did they consider? Things like that.
- Ajedi32 10y ago
  The reasoning behind their decision is linked near the top of the article, under "Sid and the team decided": https://gitlab.com/gitlab-com/infrastructure/issues/727#note_20044060 https://gitlab.com/gitlab-com/infrastructure/issues/727#note...
frabbit 10y ago
Buried towards the bottom is the interesting decision to dump CephFS and go with NFS instead.
- chillydawg 10y ago
  Fundamentally not much different, though. N shards holding the data, you have to know which shard is which and maintain connections to it. Split brains could happen if that sharding is dynamic etc.
- tra3 10y ago
  I find that interesting as well. Furthermore, looking into deploying containers with cheap cloud hosters , I was curious how to share data across various nodes by looking at how Google and Amazon do it. Turns out it's still NFS. I've never looked into networked FS before, but I thought they'd move past NFS to something more advanced...and newer I suppose.
- Rezo 10y ago
  I wonder if Amazon's new managed EFS service would make sense? It's exposed as a NFS mount to the OS. The claims are: - Up to thousands of Amazon EC2 instances, from multiple AZs, can connect concurrently to a file system. - Data is stored redundantly across multiple AZs. - Low, consistent latency. - Multiple GBs per second.
  - sytse 10y ago
    EFS is pretty great. But for our use case there are a few drawbacks: 1. It doesn't scale to the size we need for all repositories, so we still need sharding. 2. It is expensive, $300/TB-month 3. You're still sending all traffic over the network, we prefer the latency of local storage we can achieve by developing Gitaly https://gitlab.com/gitlab-org/gitaly https://gitlab.com/gitlab-org/gitaly
    - jamesmiller5 10y ago
      Why develop Gitaly instead of using consensus (etcd, zookeeper, etc.) for writes to fast storage like ssds and the normal git binary? As far as I can tell it's adding a lot of complexity by caching high level RPC calls but still doesn't really address multi-master read after write consistency or coordination.
dylanha 10y ago
Where is the original thought? There were few hard numbers. There was way too much quoted material and no Gitlab thoughts on it. It looks like they made the post based on feelings instead of a metric like time or dollars.
- timanglade 10y ago
  Point well taken on the blogpost leaving some important info out — sorry about that. Having been involved in the conversation, I can guarantee you that we came to this conclusion primarily through an analysis of the numbers (both cost & effort). I’ll see what we can do about publishing them.
mschuster91 10y ago
Hmm. Nearly all the commenters seem to be "extreme", i.e. either full-cloud or full-bare-metal. Why not go hybrid? Do your baseline load with bare-metal and your spike load with AWS, GCE or Azure Cloud.
- jtwaleson 10y ago
  For most use cases that will complicate the system. Keeping everything the same and simple is preferable.
- detaro 10y ago
  Cloudbursting either a database server or their git file storage doesn't sound all that viable, and I'd expect that to be the things that matter/are the bottleneck for Gitlab?
- brianwawok 10y ago
  You are adding more points of failure and complexity. To what, save some money? Which is fine, but then most of the arguments against pure on-prem apply. Save money but a lot of complexity. If your on-prem dies, most hybrid cloud setups will die.
tabeth 10y ago
Couldn't they have easily just drawn the opposite conclusion? I don't see a strong connection between the article's contents and its conclusion.
- Rezo 10y ago
  I read it as: commenters raised a bazillion legit questions that we had not necessarily considered (unknown unknowns), and now the whole endeavor seems more risky and the TCO questionable. Let's instead gradually re-architect our application, so it fits better in the cloud, which also happens to align with what most of our enterprise customers are going to need anyway.
jtwaleson 10y ago
Nitpick for the author, you put twaleson for my quote, should be jtwaleson.
- connorshea 10y ago
  Whoops, sorry. Opened a merge request to fix that. https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/5175 https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...
  - sytse 10y ago
    And it's merged and deployed.
pestaa 10y ago
What about managed dedicated servers? You don't need to go all in full metal, and be responsible for networking and swapping disks and all that. I know it's more than a handful of deployment recipes, but it's not like they don't manage their cloud instances either... This article is a false dichotomy with a seemingly rushed decision.
- dx034 10y ago
  I was wondering the same. Why not manage servers without managing the hardware? There are several hosting providers that offer enough capacity and data centers to provide enough capacity. You'll still need people managing servers, but no one has to live close to a datacenter and you can emulate at least some cloud-like features (e.g. spinning up a new server if one server fails and dealing with the failing server later). The only companies where that doesn't work is when you need physical security for your data (locked cabinets that no one from another company can enter), but that's no concern if you already had your data in the cloud before.
keepper 10y ago
Why was hybrid not considered? With providers like megaport providing really inexpensive direct connect, this is almost a no brainer. This may seem harsh, but relying on random commenters shows a huge flaw in how you guys went about this. Physical environments DO work well, but they do require experience to run them. I have used AWS for as long as its been around, but nothing beats physical environments for "known workloads", as long as you run them EXACTLY like you would run a cloud environment. This is why the hybrid approach is such a great thing. Run what you know well in physical and reap great savings, run what you dont know well cloud ( as well as take advantage of the analytics products), and reap the speed ;) - This means thinking of physical servers are individual units. - This means redundancy at every level. - This means architecting for failure ( servers and switches do gie ). - This means no shared storage for performance critical parts ( shared storage as below the OS level ). - This means objects stores/sharding/etc as your storage layer. - This means real engineering - And this means exactly the same whether you're physical or not. I've managed environments of 50 vm's and environments of 50000 physical servers. The methodology is always the same. and yes, this means you can save some monthly cost, and apply it to staff, that can do a lot more than just maintain this infrastructure. PS: for those that think that showing up to a datacenter is required, you're doing it wrong. Pretty much any datacenter has hot-hands service, and with the right redundancy, hardware replacement is something you can do at a slower pace. PPS: im sure someone will nitpick some of my points. The reality is, there is real money savings here. For example, Snapchat spends MORE in cloud infrastructure in 2016/2017 per year, than ALL OF GOOGLE did in 2012... think about that for a second... even netflix runs openconnect to push bits..
- dasil003 10y ago
  > This may seem harsh, but relying on random commenters shows a huge flaw in how you guys went about this. Your other points and experience are well taken, but this is definitely too harsh. Think about the nature of GitLab as an open company building dev tools and paying below-market salaries. They have a huge amount of developer mindshare relative to their internal head count. They are not very competitive for hiring people with your particular skillset. Given these facts, I think an open decision making process serves them better than trying to make decisions behind closed doors. It may come off a little ham-fisted, but in actuality I think it's far less ham-fisted than a huge number (maybe even majority?) of engineering decisions made every day in companies that just don't have the expertise to make the right decision and often don't even realize they are getting burned by what they don't know they don't know.
drinchev 10y ago
HN is great way to get opinion about a decision. Although sometimes the criticism is way more than the positivism and support. Post about "We use X and is great" will usually attract lots of "X is bad, use Y" comments. I really hope that GitLab took a well informed decision based on arguments and counter-arguments.
101km 10y ago
Gitlab is hosted on Azure and like many startups received six figures worth of free credits to be in the cloud. As the credits run out the architecture reverts to the natural state - architectures tend to be the derivative of organizational structure, code base, and processes of a company. Through this lens these decisions start making more sense. It is a reactionary process. Sometimes when you start small you bounce around like that and are lucky enough to grow for years and years until you hit a wall.
nik736 10y ago
So, GitLab based a crucial part of their company on some random comments that were just quoted and could be said the other way around as well? Were is your own thinking and your own decision making? Bare metal doesn't mean colocation. It also doesn't mean going bare metal only, you could go with a hybrid approach and save the base costs while maintaining the cloud scalability.
- timanglade 10y ago
  See my comment clarifying how we came to this decision [0]. And for what it’s worth we considered colocation and hybrid models as well before coming to this conclusion. [0]: https://news.ycombinator.com/item?id=13776735 https://news.ycombinator.com/item?id=13776735
ben_pr 10y ago
I'm a little shocked at this decision. The issues faced are all solved problems from PB storage to HA on server clusters even across Data centers. Good solutions do have upfront costs but the math that cloud hosting is 5-10X more than co-location company owned/leased is still in the ball park. It sounds like they may need an architect with enterprise experience to help them out rather than random comments on HN.
- djsumdog 10y ago
  Yea, every growing company I've been at has been in the process of moving off of hosted solutions and onto their own hardware to cut costs. Two such companies made the mistake of doing that with OpenStack, which is terrible and should die in a fire. But one later switched to DC/OS and containers and it has worked really well. They've been migrating apps running on EC2 instances into docker containers than can run on marathon in our local data center and the savings is pretty substantial (even adding in the cost of the teams needed to maintain our own platforms). Managed solutions are great for startups. There is a lot of value in not having to setup, maintain and manager your own hardware .. but that does reach a limit and companies need to be prepared for that transition and avoid lockins.
tbrowbdidnso 10y ago
So everyone says the cloud is the future. I get it. But is this the truth, or what all the tech giants want people to believe? Don your foil hats for a moment and listen to me. All the original internet companies run their own hardware. They rent out excess production capacity to us peons in the form of cloud services. These companies that all run their own hardware exclusively are telling everyone that it's stupid to run your own hardware... Why are we listening? Look to the newer tech giants as a prequel of what's to come. Netflix for example, is completely at the mercy of Amazon. They might as well be "Netflix brought to you by Amazon". Their edge is in software alone, nothing that causes a huge barrier to entry like custom hardware. This makes them much easier to dethrone. Hilariously, they rent all their hardware from a direct competitor who has access to all their software secrets. Does anyone else see something wrong with this? The cloud as a money saving venture is and always has been a damn lie. It's the same tactic as when automotive companies paid off local governments to destroy Americas public transport many years ago. All the big tech companies have their hands in the cookie jar besides Facebook, who remains mostly silent but runs their own hardware as well. The major tech companies have every incentive to make you think running your own servers is nigh impossible. Don't drink the fucking koolaid. If the industry continues to consolidate rapidly AmaGooSoft will be the ONLY places you can have web servers in ten years
- clarkmoody 10y ago
  Don't forget to take the tinfoil off when you're done ;-) What are some examples of companies that started in the cloud then switched to their own metal? Surely there are some blog posts about costs before & after?
  - tomkinstinch 10y ago
    Dropbox switched from S3 to their own in-house hardware: https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire/ https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...
- lbotos 10y ago
  I mean, sure, you might not save money being in the "cloud" but running hardware is a different skillset than building software. If you can pay someone else to run your hardware, someone else who wants to be really good at running hardware, you might be able to focus more on your software, and get really good at making that?
deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
user5994461 10y ago
I love all the comments trying to explain that managing your own hardware, storage, network and backup cannot take that much of your time. Not only it does take a huge amount of time, but it takes so much time that it require multiple dedicated highly-skilled people. Last but not least, the last article about Gitlab was about a major outage where they had as much as 6 out of 6 backups unusable! The fact that they still exist today is to the sheer luck that there was an unexpected copy made manually before the disaster. That speaks at length on how that 100 people company is struggling to have the resources and the qualifications it needs to execute (what startup isn't?). Let's not send them on a suicide mission to manage their own hardware on top of everything they already have to achieve. ;)
- nailk 10y ago
  Why do you think that if you don't use a cloud service you have to own hardware? You don't have to do it. Dedicated server providers will manage your hardware, that's it.
mahyarm 10y ago
In my observing-from-a-distance experience, going bare metal makes sense once you are a certain size and have predictable baseline load to maintain. Many companies that have their own data centers still use cloud services to auto scale and deal with spikes.
lightedman 10y ago
As long as you prove yourselves incapable of handling things without the aid of third parties, I will never be using your services. Sorry, I'm just picky like that and demand true competence.
winteriscoming 10y ago
Every time I see a post from Gitlab, I worry for them. I haven't used gitlab but have checked it out a few times. On the other hand I use github regularly. But this isn't about github vs gitlab. Instead this is about gitlab's definition or rather level of transparency and something they seem to be proud of and are even being appreciated very often here at HN. I feel that gitlab have taken the transparency thing to an extreme, to the extent that they even named the engineer that caused a recent downtime. I don't think it matters that the engineer didnt object to it. Then there are posts about why they are choosing some UI framework and why they are moving away from cloud and now this. It's fine to be transparent but in a professional world it's also important to be private about certain things. Trying to please people by being transparent for the sake of it will take a toll on the team eventually. It's much more effective to just get things done instead of constantly being transparent to the external crowd, about every single detail
bernardlunn 10y ago
TL:DR = people cost more than servers = stay in cloud
dx034 10y ago
As much as I like gitlab, that decision seems weird. I'd expect a business to come up with numbers for different scenarios, not just quotes (even if those weren't the only deciding factor). What I mean is, instead of writing about 8TB disks vs 2TB disks, get hard numbers for those and calculate costs for different versions to the costs in the cloud. Instead of saying that engineers are expensive, look at market data to determine how much you have to pay to hire these people. Instead of writing about the potential of full cabinets, get quotes from data center providers of what they can provide and how they would deal with this. I'm sure Equinix has solutions for that kind of problem. And in addition to that, it seems that managed dedicated wasn't considered at all. Why hire someone to swap disks for you? Why not just renting capacity for servers and have the data center provider taking care of hardware issues? They operate at bigger scale and can do this with little overhead costs. You're still much cheaper than the cloud but don't need to drive somewhere to swap a server (you just switch to a spare one until the data center fixed the problem). With Gitlab's scale, it shouldn't be too hard to get a decent bespoke quote from companies like OVH.
- discordianfish 10y ago
  They also miss out on the dog fooding aspect. They target enterprise customers who run probably run gitlab on bare metal. By doing the same, you experience what your customers do and have much closer feedback loops on any issues that might especially affect bare metal. Regarding managed dedicated.. I had very bad experience with rackspace in that regard, which is why we moved AWS. At another job we had our own DC suite but had some company taking care of initial cabling as well as remote hands from the datacenter. If I ever do bare metal again, that's why I would do.
  - sytse 10y ago
    I agree that dogfooding is important. Our users and customers run GitLab both on bare-metal and in the cloud. GitLab runs better on bare-metal because it tends to have better IO performance. So by running it in the cloud we are optimizing where it is needed.
- CapitalistCartr 10y ago
impappl 10y ago
This commit really does tell the story of the disproportionate influence HN comments had on this decision: https://gitlab.com/gitlab-com/www-gitlab-com/commit/8f7d618d14b91c338e9d4bfca96f86c3c6e5766b https://gitlab.com/gitlab-com/www-gitlab-com/commit/8f7d618d...
jstoja 10y ago
Why didn't they chose to split it? You always have your bare-metal infrastructure and if you need a terrible growth and need more IT infra, you can use public cloud to handle to load until you find plan a good solution bare metal. Sure it's some work, the setup is sometimes not ideal, but it's a huge gain imho. In addition, many people bring the argument that infrastructure is not their core skill. It is true, but do they need to get to the level of public clouds to run their own infrastructure ? I mean, even with sub-optimal choices on hardware but smart choice on how you handle your architecture, you still can have huge savings and not so many risks by doing it in-house.
tgtweak 10y ago
Use your bare metal for your low-water mark and "scale over" to the cloud. Setting up a hybrid cloud like this is not overly complicated and AWS and other providers will offer you a direct connect network connection in many carrier neutral colocation datacenters so you can sit on your cloud vpc with a physical connection in your datacenter of choice. Get the best of both worlds. Providers like ovh can get very close to the cost of buying your own metal with a tco break-even at over 18 months, not counting network and power equipment costs. Using ec2 or GCE for a large number of persistent vms without a heavy discount seems... Wasteful.
theptip 10y ago
As a potential Gitlab customer (I would be running an instance in the cloud if I moved off of their hosted offering), this removes the main concern I had with migrating to their platform. Fundamentally the performance problems they had could be solved by either software or hardware, and I have more faith in the team's ability to learn the distributed computing / caching techniques required than to learn how to run their own metal.

350 points by happy-go-lucky 10y ago

I'm really glad you guys decided to listen to the experienced commenters' advice and change your mind, rather than obstinately plowing on with what seems to be a bad decision.

Dangeranger 10y ago

It's interesting that the first half of the explanation is largely quotes from a prior HN post's responses. Does anyone else feel a bit weird seeing off hand comments getting quoted in the explanation for a business decision? I guess we should all get more accustomed to our public input carrying weight in the zeitgeist. Gitlab's "develop in the open" nature really shows through here. I am not saying that's bad, it's just so different than most startups and established businesses.

gtaylor 10y ago
I don't think they were using this as justification or explanation in their eventual reversal. I think they provided these quotes as interesting tidbits. Kind of like pull-out quotes on an article. You can ignore them entirely and still get the gist of the post. On a more general note: It's got to be incredibly hard to do what GitLab does with their extreme transparency. I feel like we have to be careful about reading too deeply into things and nitpicking their culture or process. HN is full of "expert" advice, much of it being terrible. They weighed their options, invited feedback, then made a decision. I appreciate what GitLab does in being so transparent. None of us are owed explanations or insight into how they operate, yet they go out of their way to provide it. Kudos to syste and his team!
- sytse 10y ago
  Thanks. The most useful advice was shared in private. People coming to our office to share war stories of regretting moving to bare metal. I also received dozens of people sending direct messages via twitter because they couldn't share their stories publicly. Some of the best advice are things that we can't share publicly. For example a major company going bare-metal and then spending a lot of time to set up an authorization system that you get for free with AWS IAM.
  - dustinmoris 10y ago
    Did you have to sign a NDA? Because if people send you random advice via email after reading a blog post then this sounds to me very much like "publicly sharing" their experience...
    - shshhdhs 10y ago
      I disagree. It's a private email. Gitlab can ask permission to post, but it would be impolite and perhaps unethical to assume a private email is now "public". If they wanted it public, they could have chosen to provide a comment on HN, Twitter, etc.
      - sytse 10y ago
        Exactly, everything is private unless it was posted in public by the author or explicit permission was given. We have a transparency value, but we understand that other organizations are different. And even we assume that our private communication will stay private.
      - dustinmoris 10y ago
        Apparently according to syste some of the best arguments came from emails, so he could have put those arguments anonymised and trimmed down to the main points into the blog post, which would have respected the company's privacy while still sharing good arguments, unless there was a NDA which would prohibit that. So I disagree with you, because you look at this very black and white while the reality is grey.
hackerboos 10y ago
This echochamber has a profound and mostly unwarranted impact on people, myself included.
234dd57d2c8dba 10y ago
No surprise here. Honestly, their team seem to be lower skill or less experienced than I'd have thought. I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy. The lack of automation, especially around validating db backups, failover (not having the failover process scripted and tested is _begging_ to have a nightmare at 2am where you're reading documentation on how to fail over a db), etc. The simple thing of having your hostname / $PS1 say the machine's purpose could have stopped this. All prod machines have a bright red PS1 and a clear name of <type>-<service>-<prod/dev/etc>-<region>-<dc>.corpnet in my setups. All of this is reflected in their discussion style, meeting style, etc. Ad-hoc, not very carefully designed, random off-hand comments. Obviously a young team with a lot to learn. Nothing wrong with that, but a lot of customers are relying on their skills! Learn quick!
- kogepathic 10y ago
  > I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy. I do agree with you it reflects poorly on GitLab for only having a primary and replica, with broken backups. BUT, they are a startup, and they need to be laser focused on growing the company and securing funding for the next quarter so they can keep the lights on. This kind of pants on fire growth in a startup often (always?) comes at the cost of redundancy and best practices. If you stop to make your platform bulletproof instead of the new features you promised customers/investors, you die. I'm not saying this is an excuse for them to permanently shrug off making their platform more redundant and engaging in best practices. But, as with many startups, they're focused on delivering features as fast as possible to grow their user base. I think, and hope, that their recent outage has been the experience they needed to prioritize their shift toward more redundancy and best practices.
  - 234dd57d2c8dba 10y ago
timanglade 10y ago
We quote a lot of responses from HN but that doesn’t mean it’s the only thing that led us to this decision. I was personally involved in a lot of private conversations about this, with the executive team, with investors and with people who had gone through this exercise before, and they all had as much if not more impact on our final decision than the HN thread did. It’s just harder to quote them than referencing additional (& similar) opinions noted in a public forum.
- Dangeranger 10y ago
  That is good to hear. My comment was not meant as a negative criticism of your process, although I can see how it could be interpreted that way. It was intended more as an observation about the openness that Gitlab brings to the process of design and decision.
  - timanglade 10y ago
    No worries! I didn’t perceive your comment as negative at all — I just took it as an opportunity to clarify something that we should have made more clear in the blogpost itself really. Thanks for bringing this up and for doing so in a constructive way!
Rapzid 10y ago
Well... Here is something very interesting. Myself and others were "cautioning" very heavily against moving to self-hosted gear. It wasn't until their CTO talked to somebody in person he had respect for that they flipped over to staying in the cloud. This to me calls into question the entire idea of crowd-sourcing decisions and plans like this. It was a single person who moved the needle, and now there is retroactive agreement(and quoting) with the people who were against it from the start. If it went the other way there would be plenty of quotes to pull from of people supporting the move to dedicated.
- degenerate 10y ago
  Recommending someone to build in the cloud is an easy and safe recommendation to make. Therefore, more people will go that route when asked what you should do. Pretend for a second you plucked someone from a royal family that never prepared their own food, and they asked you how to make food. Are you going to recommend they purchase land, start a farm, grown their own vegetables, and raise their own cattle? Hell no. You tell them to go to the grocery store (cloud) and if they are desperate, they can purchase some pre-made food at restaurants (SaaS providers). When you don't know anything about a person's exact needs or abilities, making the safest recommendation is easiest, and therefore it proliferates. I hate the cloud but I know how to do what I want on my own without it. You don't know that about other people.
  - cookiecaper 10y ago
    I don't think the cloud is the grocery store. It doesn't give you a pre-packaged solution that you can just rip open and immediately access. Rather, the cloud is like leasing the land from an owner. You still have to till, plant, maintain, and harvest, but you have to pay an annual rent to do so. There are a few things that the landowner is responsible to provide, but most of the day-to-day maintenance responsibility is on the lessee. People generally recognize that while renting is sometimes necessary, it's much nicer to be an owner. We really need to stop this belief that the cloud is a magic thing that can fix your problems. It splits out the responsibility for actual, real-life hardware maintenance to a hosting company, but other than that, the responsibilities, including the responsibility to take backups, set up security, and design a fault-tolerant application, remain in the cloud user's court. Cloud is just a way to rent a virtual server; you still have to make the server work correctly.
ReidZB 10y ago
When I evaluate any tech project for use in production, I always give HN a cursory search for materials. It's not always fruitful, but in the cases where I stumble across something big in the HN space, I pick up valuable intel here... I still try to research elsewhere too, just to cover my bases, but HN is quite helpful as an archive of knowledge. Especially knowledge about running something in production.

AgentK20 10y ago

A bit disappointed in the article, as the bulk of it is simply quoting various viewpoints, some of which disagree with each other. I'd be much more interested in the thought process that occurred for them to reach the conclusion they did. What internal doubts did they raise? What possible solutions to those doubts did they consider? Things like that.

Ajedi32 10y ago
The reasoning behind their decision is linked near the top of the article, under "Sid and the team decided": https://gitlab.com/gitlab-com/infrastructure/issues/727#note_20044060 https://gitlab.com/gitlab-com/infrastructure/issues/727#note...

frabbit 10y ago

Buried towards the bottom is the interesting decision to dump CephFS and go with NFS instead.

chillydawg 10y ago
Fundamentally not much different, though. N shards holding the data, you have to know which shard is which and maintain connections to it. Split brains could happen if that sharding is dynamic etc.
tra3 10y ago
I find that interesting as well. Furthermore, looking into deploying containers with cheap cloud hosters , I was curious how to share data across various nodes by looking at how Google and Amazon do it. Turns out it's still NFS. I've never looked into networked FS before, but I thought they'd move past NFS to something more advanced...and newer I suppose.
Rezo 10y ago
I wonder if Amazon's new managed EFS service would make sense? It's exposed as a NFS mount to the OS. The claims are: - Up to thousands of Amazon EC2 instances, from multiple AZs, can connect concurrently to a file system. - Data is stored redundantly across multiple AZs. - Low, consistent latency. - Multiple GBs per second.
- sytse 10y ago
  EFS is pretty great. But for our use case there are a few drawbacks: 1. It doesn't scale to the size we need for all repositories, so we still need sharding. 2. It is expensive, $300/TB-month 3. You're still sending all traffic over the network, we prefer the latency of local storage we can achieve by developing Gitaly https://gitlab.com/gitlab-org/gitaly https://gitlab.com/gitlab-org/gitaly
  - jamesmiller5 10y ago
    Why develop Gitaly instead of using consensus (etcd, zookeeper, etc.) for writes to fast storage like ssds and the normal git binary? As far as I can tell it's adding a lot of complexity by caching high level RPC calls but still doesn't really address multi-master read after write consistency or coordination.

dylanha 10y ago

Where is the original thought? There were few hard numbers. There was way too much quoted material and no Gitlab thoughts on it. It looks like they made the post based on feelings instead of a metric like time or dollars.

timanglade 10y ago
Point well taken on the blogpost leaving some important info out — sorry about that. Having been involved in the conversation, I can guarantee you that we came to this conclusion primarily through an analysis of the numbers (both cost & effort). I’ll see what we can do about publishing them.

mschuster91 10y ago

Hmm. Nearly all the commenters seem to be "extreme", i.e. either full-cloud or full-bare-metal. Why not go hybrid? Do your baseline load with bare-metal and your spike load with AWS, GCE or Azure Cloud.

jtwaleson 10y ago
For most use cases that will complicate the system. Keeping everything the same and simple is preferable.
detaro 10y ago
Cloudbursting either a database server or their git file storage doesn't sound all that viable, and I'd expect that to be the things that matter/are the bottleneck for Gitlab?
brianwawok 10y ago
You are adding more points of failure and complexity. To what, save some money? Which is fine, but then most of the arguments against pure on-prem apply. Save money but a lot of complexity. If your on-prem dies, most hybrid cloud setups will die.

tabeth 10y ago

Couldn't they have easily just drawn the opposite conclusion? I don't see a strong connection between the article's contents and its conclusion.

Rezo 10y ago
I read it as: commenters raised a bazillion legit questions that we had not necessarily considered (unknown unknowns), and now the whole endeavor seems more risky and the TCO questionable. Let's instead gradually re-architect our application, so it fits better in the cloud, which also happens to align with what most of our enterprise customers are going to need anyway.

jtwaleson 10y ago

Nitpick for the author, you put twaleson for my quote, should be jtwaleson.

connorshea 10y ago
Whoops, sorry. Opened a merge request to fix that. https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/5175 https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...
- sytse 10y ago
  And it's merged and deployed.

pestaa 10y ago

What about managed dedicated servers? You don't need to go all in full metal, and be responsible for networking and swapping disks and all that. I know it's more than a handful of deployment recipes, but it's not like they don't manage their cloud instances either... This article is a false dichotomy with a seemingly rushed decision.

dx034 10y ago
I was wondering the same. Why not manage servers without managing the hardware? There are several hosting providers that offer enough capacity and data centers to provide enough capacity. You'll still need people managing servers, but no one has to live close to a datacenter and you can emulate at least some cloud-like features (e.g. spinning up a new server if one server fails and dealing with the failing server later). The only companies where that doesn't work is when you need physical security for your data (locked cabinets that no one from another company can enter), but that's no concern if you already had your data in the cloud before.

keepper 10y ago

Why was hybrid not considered? With providers like megaport providing really inexpensive direct connect, this is almost a no brainer. This may seem harsh, but relying on random commenters shows a huge flaw in how you guys went about this. Physical environments DO work well, but they do require experience to run them. I have used AWS for as long as its been around, but nothing beats physical environments for "known workloads", as long as you run them EXACTLY like you would run a cloud environment. This is why the hybrid approach is such a great thing. Run what you know well in physical and reap great savings, run what you dont know well cloud ( as well as take advantage of the analytics products), and reap the speed ;) - This means thinking of physical servers are individual units. - This means redundancy at every level. - This means architecting for failure ( servers and switches do gie ). - This means no shared storage for performance critical parts ( shared storage as below the OS level ). - This means objects stores/sharding/etc as your storage layer. - This means real engineering - And this means exactly the same whether you're physical or not. I've managed environments of 50 vm's and environments of 50000 physical servers. The methodology is always the same. and yes, this means you can save some monthly cost, and apply it to staff, that can do a lot more than just maintain this infrastructure. PS: for those that think that showing up to a datacenter is required, you're doing it wrong. Pretty much any datacenter has hot-hands service, and with the right redundancy, hardware replacement is something you can do at a slower pace. PPS: im sure someone will nitpick some of my points. The reality is, there is real money savings here. For example, Snapchat spends MORE in cloud infrastructure in 2016/2017 per year, than ALL OF GOOGLE did in 2012... think about that for a second... even netflix runs openconnect to push bits..

dasil003 10y ago
> This may seem harsh, but relying on random commenters shows a huge flaw in how you guys went about this. Your other points and experience are well taken, but this is definitely too harsh. Think about the nature of GitLab as an open company building dev tools and paying below-market salaries. They have a huge amount of developer mindshare relative to their internal head count. They are not very competitive for hiring people with your particular skillset. Given these facts, I think an open decision making process serves them better than trying to make decisions behind closed doors. It may come off a little ham-fisted, but in actuality I think it's far less ham-fisted than a huge number (maybe even majority?) of engineering decisions made every day in companies that just don't have the expertise to make the right decision and often don't even realize they are getting burned by what they don't know they don't know.

drinchev 10y ago

HN is great way to get opinion about a decision. Although sometimes the criticism is way more than the positivism and support. Post about "We use X and is great" will usually attract lots of "X is bad, use Y" comments. I really hope that GitLab took a well informed decision based on arguments and counter-arguments.

101km 10y ago

Gitlab is hosted on Azure and like many startups received six figures worth of free credits to be in the cloud. As the credits run out the architecture reverts to the natural state - architectures tend to be the derivative of organizational structure, code base, and processes of a company. Through this lens these decisions start making more sense. It is a reactionary process. Sometimes when you start small you bounce around like that and are lucky enough to grow for years and years until you hit a wall.

nik736 10y ago

So, GitLab based a crucial part of their company on some random comments that were just quoted and could be said the other way around as well? Were is your own thinking and your own decision making? Bare metal doesn't mean colocation. It also doesn't mean going bare metal only, you could go with a hybrid approach and save the base costs while maintaining the cloud scalability.

timanglade 10y ago
See my comment clarifying how we came to this decision [0]. And for what it’s worth we considered colocation and hybrid models as well before coming to this conclusion. [0]: https://news.ycombinator.com/item?id=13776735 https://news.ycombinator.com/item?id=13776735

ben_pr 10y ago

I'm a little shocked at this decision. The issues faced are all solved problems from PB storage to HA on server clusters even across Data centers. Good solutions do have upfront costs but the math that cloud hosting is 5-10X more than co-location company owned/leased is still in the ball park. It sounds like they may need an architect with enterprise experience to help them out rather than random comments on HN.

djsumdog 10y ago
Yea, every growing company I've been at has been in the process of moving off of hosted solutions and onto their own hardware to cut costs. Two such companies made the mistake of doing that with OpenStack, which is terrible and should die in a fire. But one later switched to DC/OS and containers and it has worked really well. They've been migrating apps running on EC2 instances into docker containers than can run on marathon in our local data center and the savings is pretty substantial (even adding in the cost of the teams needed to maintain our own platforms). Managed solutions are great for startups. There is a lot of value in not having to setup, maintain and manager your own hardware .. but that does reach a limit and companies need to be prepared for that transition and avoid lockins.

tbrowbdidnso 10y ago

So everyone says the cloud is the future. I get it. But is this the truth, or what all the tech giants want people to believe? Don your foil hats for a moment and listen to me. All the original internet companies run their own hardware. They rent out excess production capacity to us peons in the form of cloud services. These companies that all run their own hardware exclusively are telling everyone that it's stupid to run your own hardware... Why are we listening? Look to the newer tech giants as a prequel of what's to come. Netflix for example, is completely at the mercy of Amazon. They might as well be "Netflix brought to you by Amazon". Their edge is in software alone, nothing that causes a huge barrier to entry like custom hardware. This makes them much easier to dethrone. Hilariously, they rent all their hardware from a direct competitor who has access to all their software secrets. Does anyone else see something wrong with this? The cloud as a money saving venture is and always has been a damn lie. It's the same tactic as when automotive companies paid off local governments to destroy Americas public transport many years ago. All the big tech companies have their hands in the cookie jar besides Facebook, who remains mostly silent but runs their own hardware as well. The major tech companies have every incentive to make you think running your own servers is nigh impossible. Don't drink the fucking koolaid. If the industry continues to consolidate rapidly AmaGooSoft will be the ONLY places you can have web servers in ten years

clarkmoody 10y ago
Don't forget to take the tinfoil off when you're done ;-) What are some examples of companies that started in the cloud then switched to their own metal? Surely there are some blog posts about costs before & after?
- tomkinstinch 10y ago
  Dropbox switched from S3 to their own in-house hardware: https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire/ https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...
lbotos 10y ago
I mean, sure, you might not save money being in the "cloud" but running hardware is a different skillset than building software. If you can pay someone else to run your hardware, someone else who wants to be really good at running hardware, you might be able to focus more on your software, and get really good at making that?

deleted 10y ago

[deleted]

deleted 10y ago

[deleted]

user5994461 10y ago

I love all the comments trying to explain that managing your own hardware, storage, network and backup cannot take that much of your time. Not only it does take a huge amount of time, but it takes so much time that it require multiple dedicated highly-skilled people. Last but not least, the last article about Gitlab was about a major outage where they had as much as 6 out of 6 backups unusable! The fact that they still exist today is to the sheer luck that there was an unexpected copy made manually before the disaster. That speaks at length on how that 100 people company is struggling to have the resources and the qualifications it needs to execute (what startup isn't?). Let's not send them on a suicide mission to manage their own hardware on top of everything they already have to achieve. ;)

nailk 10y ago
Why do you think that if you don't use a cloud service you have to own hardware? You don't have to do it. Dedicated server providers will manage your hardware, that's it.

mahyarm 10y ago

In my observing-from-a-distance experience, going bare metal makes sense once you are a certain size and have predictable baseline load to maintain. Many companies that have their own data centers still use cloud services to auto scale and deal with spikes.

lightedman 10y ago

As long as you prove yourselves incapable of handling things without the aid of third parties, I will never be using your services. Sorry, I'm just picky like that and demand true competence.

winteriscoming 10y ago

Every time I see a post from Gitlab, I worry for them. I haven't used gitlab but have checked it out a few times. On the other hand I use github regularly. But this isn't about github vs gitlab. Instead this is about gitlab's definition or rather level of transparency and something they seem to be proud of and are even being appreciated very often here at HN. I feel that gitlab have taken the transparency thing to an extreme, to the extent that they even named the engineer that caused a recent downtime. I don't think it matters that the engineer didnt object to it. Then there are posts about why they are choosing some UI framework and why they are moving away from cloud and now this. It's fine to be transparent but in a professional world it's also important to be private about certain things. Trying to please people by being transparent for the sake of it will take a toll on the team eventually. It's much more effective to just get things done instead of constantly being transparent to the external crowd, about every single detail

bernardlunn 10y ago

TL:DR = people cost more than servers = stay in cloud

dx034 10y ago

As much as I like gitlab, that decision seems weird. I'd expect a business to come up with numbers for different scenarios, not just quotes (even if those weren't the only deciding factor). What I mean is, instead of writing about 8TB disks vs 2TB disks, get hard numbers for those and calculate costs for different versions to the costs in the cloud. Instead of saying that engineers are expensive, look at market data to determine how much you have to pay to hire these people. Instead of writing about the potential of full cabinets, get quotes from data center providers of what they can provide and how they would deal with this. I'm sure Equinix has solutions for that kind of problem. And in addition to that, it seems that managed dedicated wasn't considered at all. Why hire someone to swap disks for you? Why not just renting capacity for servers and have the data center provider taking care of hardware issues? They operate at bigger scale and can do this with little overhead costs. You're still much cheaper than the cloud but don't need to drive somewhere to swap a server (you just switch to a spare one until the data center fixed the problem). With Gitlab's scale, it shouldn't be too hard to get a decent bespoke quote from companies like OVH.

discordianfish 10y ago
They also miss out on the dog fooding aspect. They target enterprise customers who run probably run gitlab on bare metal. By doing the same, you experience what your customers do and have much closer feedback loops on any issues that might especially affect bare metal. Regarding managed dedicated.. I had very bad experience with rackspace in that regard, which is why we moved AWS. At another job we had our own DC suite but had some company taking care of initial cabling as well as remote hands from the datacenter. If I ever do bare metal again, that's why I would do.
- sytse 10y ago
  I agree that dogfooding is important. Our users and customers run GitLab both on bare-metal and in the cloud. GitLab runs better on bare-metal because it tends to have better IO performance. So by running it in the cloud we are optimizing where it is needed.
CapitalistCartr 10y ago

impappl 10y ago

This commit really does tell the story of the disproportionate influence HN comments had on this decision: https://gitlab.com/gitlab-com/www-gitlab-com/commit/8f7d618d14b91c338e9d4bfca96f86c3c6e5766b https://gitlab.com/gitlab-com/www-gitlab-com/commit/8f7d618d...

jstoja 10y ago

Why didn't they chose to split it? You always have your bare-metal infrastructure and if you need a terrible growth and need more IT infra, you can use public cloud to handle to load until you find plan a good solution bare metal. Sure it's some work, the setup is sometimes not ideal, but it's a huge gain imho. In addition, many people bring the argument that infrastructure is not their core skill. It is true, but do they need to get to the level of public clouds to run their own infrastructure ? I mean, even with sub-optimal choices on hardware but smart choice on how you handle your architecture, you still can have huge savings and not so many risks by doing it in-house.

tgtweak 10y ago

Use your bare metal for your low-water mark and "scale over" to the cloud. Setting up a hybrid cloud like this is not overly complicated and AWS and other providers will offer you a direct connect network connection in many carrier neutral colocation datacenters so you can sit on your cloud vpc with a physical connection in your datacenter of choice. Get the best of both worlds. Providers like ovh can get very close to the cost of buying your own metal with a tco break-even at over 18 months, not counting network and power equipment costs. Using ec2 or GCE for a large number of persistent vms without a heavy discount seems... Wasteful.

theptip 10y ago

As a potential Gitlab customer (I would be running an instance in the cloud if I moved off of their hosted offering), this removes the main concern I had with migrating to their platform. Fundamentally the performance problems they had could be solved by either software or hardware, and I have more faith in the team's ability to learn the distributed computing / caching techniques required than to learn how to run their own metal.