26 ms·

GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

520 points by craigcannon 10y ago

manojlds 10y ago
After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.
- cjbprime 10y ago
  Yes, seems a little tone-deaf to publish a puff piece about GitLab at the moment.
  - cookiecaper 10y ago
    Agreed. It seems that HN did delay this a bit, but they probably should've given it at least a few more weeks.
  - Perihelion 10y ago
    Not that this excuses anything, but maybe it'll help explain it: This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Now that we're aware, we are working on correcting these things. But you're right, the timing of this piece was probably not ideal.
    - noir_lord 10y ago
      Not ideal perhaps but on the other hand one bad mistake doesn't negate everything you do or say forever that would be extremely unforgiving. ...and it's not like any large web company hasn't had some howlers over the years.
    - notcolin 10y ago
      > This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Could you elaborate? I recently heard about GitLab probably less than a week ago and was under the impression they're a great company to work for.
      - connorshea 10y ago
        She's referring to issues in the backup process specifically, not in the company culture (which IMO is pretty good).
      - Perihelion 10y ago
        When I say process, I mean some of our operational processes like making sure backups work. There was an incident with data loss last week: https://news.ycombinator.com/item?id=13537052 https://news.ycombinator.com/item?id=13537052 Like any company, we have our own business operations issues too but it's still one of the best jobs I've ever had. We're pretty open about our flaws, to the point where the CEO even has a page that lists his out: https://about.gitlab.com/handbook/people-operations/ceo-preferences/#flaws https://about.gitlab.com/handbook/people-operations/ceo-pref...
        DigitalJack 10y ago
        Read The Checklist Manifesto if you haven't already. Short book, leaves a big impression.
  - scrollaway 10y ago
    Why? It's not a puff piece boasting about their backup plans, they're talking about their remote work strategy. What's tone-deaf is the vitriol some people are spilling in this thread.
    - jjawssd 10y ago
      Spot on
    - sytse 10y ago
      Thanks, I really appreciate that! But to be honest I don't think it's vitriol so much as that people rightly expect better from us.
      - scrollaway 10y ago
        Some of the reactions are understandable, but I expect most people on HN to be able to separate unrelated issues like these. Some of the comments here are absolutely unnecessarily hostile.
  - onion2k 10y ago
    Companies are run by people and consequently they're fallible. Every company makes mistakes occasionally. Gitlab handled theirs incredibly well. A more opaque company would have kept the problem, and the experience they earned handling it, to themselves. Gitlab's openness means we can all learn a little from their event. I think that's great. I wish more companies shared more when these things happen.
    - cjbprime 10y ago
      > A more opaque company would have kept the problem, and the experience they earned handling it, to themselves. I liked their transparency a lot too! But it's neither possible nor acceptable to keep six hours of data loss and eighteen hours of continuous downtime "to yourself".
  - gukov 10y ago
    Marketing 101. Flood negative with positive.
    - sytse 10y ago
      That was not our intention. This piece was recorded before the outage. We should asked to delay publication until after the postmortem was published. That would be the marketing 101 lesson, first show you corrected your mistake instead of talking about something else.
- pg_is_a_butt 10y ago
  writing things down is not the problem... failing to read the things they write down is the problem. they probably don't have time to read, because they are too busy always writing. they can't find what they need to read in the sea of endless wasteful and pointless drivel that never should have been written down in the first place. you're all idiots.
- cookiecaper 10y ago
  I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication). GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my CoL adjustment is -40%, and I live in the United States!). They explicitly state on their hiring page that they prefer people from cheaper areas. [0] [1] On top of that, just from their headcount, it sounds like they have way too many people. 160 for a technical company like this is hard to manage when the employees are local; it's even harder to manage when employees are remote. Again, for remote work, you need mature, highly-skilled staff with a strong history of successful self-direction. You're not going to get that when you're chopping their salaries in half based on their location. Bad local wages is a major reason to work remotely in the first place. [0] https://about.gitlab.com/jobs/developer/ https://about.gitlab.com/jobs/developer/ ; use the calculator at the bottom of the page [1] https://about.gitlab.com/handbook/people-operations/global-compensation-calculator https://about.gitlab.com/handbook/people-operations/global-c... ; gems include "you are required to notify us when you move and we may adjust your compensation up or down" and "[a]ll things being equal we will hire people in lower cost markets vs. higher cost markets."
  - bgxor 10y ago
    Those salary modifiers are atrocious. Their range in my city for a senior, very experienced engineer is comparable to what most entry level jobs actually offer around here. I 100% think you are on the right track. The only people from my area that would take a GitLab salary just can't get a job elsewhere.
    - pc86 10y ago
      It makes more sense for me to get a PO box in Washington DC and drive there to get my paycheck. Hell, it's probably cheaper to "rent" a single room out of someone's house just to have them forward the mail.
      - 10y ago
- geofft 10y ago
  "This" being remote work? Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)
  - ancarda 10y ago
    It may be due to downtime earlier today (this being only days after the data loss). Things don't seem to be going well for GitLab right now.
  - problems 10y ago
    > didn't do anything particularly more wrong than anyone else? This is almost completely on their admin staff, maybe other people aren't willing to say it, but I will. Test your backups. Or at least make sure they're non-zero in size. It should really be Operations 101. Whether you do this automatically or manually by setting a reminder on your calendar once a week or even month, doesn't matter. Something this simple would have solved their entire issue. I do this and we run a much smaller shop than GitLab. Heck, if we were larger I'd have hot spare database servers in another datacenter in case the primary got nuked by disk failure, network outages or mistakes.
    - fsiefken 10y ago
      As part of our disaster recovery plan when I was working as a sysadmin at a 150p company we had replicated server (database replicated and webfiles rsync) on hot standby, we just switched the front-facing servers manually. GitLab has a very short blurb on a similar styled HA setup, I'm not sure if and how they have implemented such themselves and if it would have helped in preventing or shortening the recent downtime. They have probably documented their own setup somewhere. "Automated failover can be achieved with pacemaker alongside STONITH network management. Keep in mind that application servers need to be prepared for transitioning to the new network addresses. In this situation you can also opt to synchronize the database via a database specific protocol instead of DRBD. In the documentation for each database you can find out more about the options for MySQL and the options for PostgreSQL." https://about.gitlab.com/high-availability/#filesystem-storage-and-dbms-slave-servers https://about.gitlab.com/high-availability/#filesystem-stora...
- snarf21 10y ago
  Scale is freaking hard. It is absurd to say they figured out the secret after a 7 layer failure on disaster recovery, even more so when one of the main reasons for source control is to have disaster recovery and the ability to rebuild old source.
  - mynameisvlad 10y ago
    Wat? How is their remote working practices at all related to the database disaster? For that matter, how are either of those related to source control and rebuilding from old source? What would rebuilding source help with when you just deleted your production DB?
    - emondi 10y ago
      Not talking about this case specifically, but it could be that remote working makes defining reponsibilities harder and that could lead to not making sure that there are proper backup/restoring procedures in place.
- ridruejo 10y ago
  Every company goes through rough patches, specially one that is growing as fast as they are doing. Very few companies are as open as transparent as they are, even in the fuckups. That is really hard to do and kudos to them
  - sytse 10y ago
    Thanks. I do agree that the timing of this post is not good. The video was recorded a few weeks ago. We are eating humble pie right now and work on fixing things instead of promotion.
    - coolsunglasses 10y ago
      I have nothing but respect for your willingness to take these hits but it's not just about you. This post is now fodder that can be used against me the next time I advocate for remote work at a company. Edit: it took 10 minutes for two people to pipe up on Twitter, one specifically suggesting remote work made the ops failure more likely: https://twitter.com/paul_snively/status/829850449247297536 https://twitter.com/paul_snively/status/829850449247297536
      - sytse 10y ago
        I'm sorry for hear we're making your advocacy for remote work harder :(
- aedocw 10y ago
  I don't understand why hearing some effective management practices for a distributed team would not be welcome here at any time. Their recent mistake dealt with a specific technical process and some failures around it. That incident in no way invalidates the entire company. As a remote worker myself, I'm always keen to hear about how teams make it work. I'm also a huge advocate for distributed teams (and nobody having a commute), so personally I'm really glad this link was shared today.
yunolisten 10y ago
> Write Everything Down Then rm -rf the paper.... https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/ https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/
- Spooky23 10y ago
  Not really germane to the topic. This type of op fuck-up happens everywhere. It's hard to build solid process, particularly in growth phases. Unless there are 2x a year restore tests, I personally assume a 60% backup fail rate.
  - mbiondi 10y ago
    The only reason they are still in existence is due to a chance backup they took for a tangential reason. From the sounds of it, their solution is held together with bubble gum, some tape and lots of hand waving. Being in 160 different locations probably doesn't help much either.
    - sirclueless 10y ago
      Unless I'm misreading things, the reason they only lost 6 hours of data instead of 24 hours was a chance backup, but there was never an existential crisis here. Downtime happens to pretty much every service out there. In this case the company was incredibly forthright and so we can make fun of their stupid mistakes, but really most mistakes are stupid when you look at them -- when you make thousands of decisions a day, some of them will seem silly in hindsight. We just never learn about most of them.
    - chousuke 10y ago
      I'm pretty sure most solutions on the internet consist of bubble gum, some tape and lots of handwaving. Gitlab's screwup is hardly unique, even if it was very public. It's difficult and expensive to build and maintain a solid system, and even if you want to, time and financial pressures often just don't let you, on top of the issue of just communicating the need for solid engineering, as it usually only becomes apparent when the problems start occurring.
deepaksurti 10y ago
>> 2:41 – GitLab values boring solutions: our product should be exceptional Exceptional products have exceptional UX. Gitlab IMHO has the worst UX of all git based products out there, I much rather take BitBucket over Gitlab. I tried using Gitlab, but no, I would much rather pay the 7$ to GH for my private repos. I sincerely hope they make an exceptional product. And 'should' better be 'must'!
- zenlikethat 10y ago
  I'm curious what you dislike about the UX. Personally I like the "feel" of the website generally (though it's not as polished as GitHub) and just find it to be slow, which I think they're working on improving.
  - bpicolo 10y ago
    Personally, not a fan of how a readme-less repository just has an entirely useless main page. I much prefer having the file view a'la github on that page. That's the main UX complaint for me. Still use gitlab bunches. Main complaint overall is the incredibly sluggish git push speed.
    - pedroms 10y ago
      Speed is one of our main priorities for GitLab this year. We've started Gitaly, a project dedicated to that: https://gitlab.com/gitlab-org/gitaly https://gitlab.com/gitlab-org/gitaly We expect it to become a standard component of GitLab in Q1 2017 and to reach full scope in Q3 2017.
      - bpicolo 10y ago
        Can't wait : )
    - eknkc 10y ago
      You can change that on your personal prefs to show the file list by default, with Readme on bottom: http://c.ekin.io/3z1V2v3P041b http://c.ekin.io/3z1V2v3P041b git push speed is fine on our own ce installation but gitlab.com is beyond shitty.
MrFurious 10y ago
Obviously, Gitlab don't have the secret for best uptime.
uladzislau 10y ago
Obviously what they say and what they do are two different things based on the recent events.
WalterSear 10y ago
And don't store it on gitlab.
newsat13 10y ago
Last I checked a year ago, GitLab had just 80 employees. So, it's now doubled. That is quite some growth. More often than not growing companies just collect lots of employees and then a startup comes and beats them...
Cofike 10y ago
Judging by the number of joke and low hanging responses on this thread I think they should have held this article for a little longer.
- sytse 10y ago
  I think we earned a reality check. Have such a promotional video go out after the outage was out of touch on our part. We were aware it would be posted and didn't ask to hold it.
Kiro 10y ago
Given the complete backlash I'm seeing on HN it seems like transparency actually hurt you. Very sad.
- gukov 10y ago
  It's the business-as-usual attitude and acting as if nothing happened right after a catastrophic event that riles people up. Transparency is fine. Eventually moving on is fine. But right now people are interested only in one topic when it comes to Gitlab.
  - Kiro 10y ago
    I think a lot has to do with the reason for the failure being perceived as sloppy and irresponsible. Without the transparency we wouldn't be able to bash them like this. Many of the sarcastic comments here refer to the failed processes they revealed in the postmortem.
- sytse 10y ago
  We're still committed to transparency even if it means taking a few lumps now and then, it's better in the long run and it's baked into everything we do.
  - gtaylor 10y ago
    I think many or even most appreciate your stance on this. Please don't get discouraged by the vocal but petty minority!
  - photonwins 10y ago
    Kudos to the entire team, as another commenter said, please don't be discouraged by the petty comments. It takes balls to admit mistakes in public. You have earned many people's respect.
  - lightbulbjim 10y ago
    Please keep it up. We need more companies like you.
Philipp__ 10y ago
Everybody is prone to making a mistake! As long as you learn something from it, so that it never happens again, it's ok. And I really don't get why people give them bad rep for this post. I mean it's obvious why it's posted and we all know what happened week ago, but consider their size and scale, it's not easy to manage so many people remotely.
- snarf21 10y ago
  "...but consider their size and scale, it's not easy to manage so many people remotely." This is very true but the title of the post is "GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down". That is the backlash, don't say you've figured out the secret then make a near catastrophic mistake.
  - e40 10y ago
    I don't see how their remote structure has anything to do with their mistake with backups, etc talked about here lately.
JustSomeNobody 10y ago
Wow. You all in the comments must be a bunch of perfect people, huh? Good grief. Also, nice cameo by the office cat.
- overcast 10y ago
  Imperfect doesn't mean you can be completely negligent of basics.
- sytse 10y ago
  Thanks, but the critics are right. This is not a good time to be promoting ourselves. We should be working on our postmortem and figure out how to improve our service. We're actually doing that, this video was recorded before the outage. But impressions count.
bryanh 10y ago
160 employees remote is impressive and commendable. Zapier is fully remote as well (but half the size in employee count). I'd say "write everything down" is a great shortcut to the sorts of practices you need to cultivate. We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural - organizations become complex as they grow, and individuals are busy doing their thing. You often have to bring the important data to them. On another note, working remote is awesome. I recommend everyone give it a spin once in their careers - but try to find a team that embraces it. I've heard mixed experiences from those who were the single remote person on a team.
- connorshea 10y ago
  I didn't know Zapier was a fully remote company, that's awesome :) I pretty much agree with your thoughts about over-communicating, it's super important and we've run into exactly the same things, where people aren't always caught up on everything going on at the company. In my experience this happens in physical workplaces as well, just to a lesser extent.
- Perihelion 10y ago
  Honestly, I haven't seen a company that does the hybrid remote/on-site thing well. I'm sure they exist, but every time I've worked at one the people who were remote were out of the loop on almost everything. Hybrid remote/on-site requires some great discipline. If you have a watercooler chat with someone about a feature, it's easy to forget that a remote colleague wasn't there for the discussion/not document what was said. +1 for working remotely on a team that makes an effort to make it work.
  - sotojuan 10y ago
    It works at our company because the only remote guy is the only one in his "domain" (iOS). He probably doesn't care about the frontend watercooler chat and anything important about the backend will be mentioned in standup.
the-dude 10y ago
I have the impression this is being moved off the frontpage in an accelerated manner ...
- the-dude 10y ago
  And it is back up, from place 22 to 12 again.
  - dang 10y ago
    Sometimes software and moderators do the same thing, which doubles the effect. When we notice that, we turn off the moderation and just rely on the software. That's why this one fell and then went back up.
  - johansch 10y ago
    https://about.gitlab.com/2015/03/04/gitlab-is-part-of-the-y-combinator-family/ https://about.gitlab.com/2015/03/04/gitlab-is-part-of-the-y-...
Paul-ish 10y ago
This is interesting, as I interned at Mozilla, and they have a large number of remote employees. There was only one person (other than me) on my team in my location. Mozilla also records a lot of things, but the intent is more to publish that stuff publicly, in the spirit of openness towards the community. For example, all my team meetings had public notes. I wouldn't be surprised if this tendency to record everything for the sake of the community also has positive benefits inside Mozilla.
deleted 10y ago
[deleted]
dandersh 10y ago
The most frustrating thing about working remote was the complete lack of documentation around anything -- meetings, requirements, internal frameworks/libraries, etc. Good for them for emphasizing documentation. I lost track of the amount of times I was told to "look at the source code" when asking about documentation for the internal framework. Also fun was being assigned a feature and having it's functionality explained over the phone, with any subsequent follow-up being met with "Didn't we already talk about this"? I never would have thought that a remote based team would have the worst documentation out of anywhere I've worked, but that's exactly what happened.
- bgaluszka 10y ago
  > I lost track of the amount of times I was told to "look at the source code" when asking about documentation for the internal framework. Can you elaborate on this one? This is my preferred way.
  - overcast 10y ago
    You prefer just being handed a pile of code, rather than documentation on how to use it? please.
    - bgaluszka 10y ago
      In case where I work for a given company and it's internal tool, always.
      - crpatino 10y ago
        Imagine this scenario. You work on product A. Your VP of very important things declares that every product should use library L. Library L has been designed to incestously interact with product D and makes a bunch of assumptions about what the "working environment" should be. Teams B and C have already made the transition, so L made some kludgy APIs to make B and C interaction possible, though it is maintained with low priority because everybody knows D is the money maker. Both B and C have a bunch of assumptions in common that A does not share, and they both had to compromise and emulate D environment to some exent. You are given L binaries, a few toy examples of how to use the most basic functionality in L, and access to B codebase. L codebase does not use the same source control system, so you don't even know where to begin looking for their code. Please keep in mind that your manager does not expect you to learn all the inner workings of L. Your priority task is to use L to support some new feature in A. You still have a number of other tasks to accomplish in A that are not related to L, and some of those might raise in priority with little or no notice. Do you still think it is a good idea to just browse L code, without knowing how large (or well writen) it is?
nedsma 10y ago
Glad that Gitlab is pushing the remote working boundaries even further. Yes, document everything, write easy to understand code, offer your remote coworkers to call you whenever they need help, leave feedback, praise work as if they're local. It all matters and even more when remotely.
smarx007 10y ago
It would be so much better if YC would made it a podcast instead (or in addition). Would be a perfect thing to listen on the go.
tra3 10y ago
How do you organize internal knowledge base?
johansch 10y ago
I mean, really: after figuring out after the fact that five out of five backups were malfunctioning - go here to read our secret to managing 160 employees remotely? Nah... no thank you. I think I will get my management advice from a company that is not totally broken. I don't care how transparent you are, or how remote-worker friendly you are, if you can't do the basics right. You are in the data storage business! Shut down the company and refund the money to the investors.
whoiskevin 10y ago
I have to watch a video that then tells me to write everything down? Sorry I couldn't resist. I believe and practice that as much as I can because it makes things easier regardless.
chmars 10y ago
What kind of tool would you recommend to write everything down, especially a self-hosted tool?
peterwwillis 10y ago
I think the worst team I've ever worked on was the one that had a team lead that literally never wrote anything down. Everything was verbal, and what wasn't verbal was private messaged. We would have the same discussion three times because nobody wrote it down the first or second time. Nobody even knew what the changes to the formal design were, because they never had a documented change review. I always tell people on my team that if it isn't written down, it doesn't exist. When you get into the habit (and learn how to manage all that information) it really saves your bacon. The other thing that would have saved them from the backup drama was learning how to document and date your to-do list. Suddenly someone notices that you've had a test process for backups on the to-do for over a year and they bring it up at the next meeting. And another thing: in team meetings, people can write their own meeting notes so everyone becomes responsible for documenting their responsibilities and what affects them. It's easier to do this remote than at a stand-up, because you're already at your keyboard.
gravypod 10y ago
I can't wait for a remote-work company that has a policy like this to get into some kind of legal troubles that warrents subpoenas. This is a prosecutors gold mine. I feel bad for the defense lawers already.
- yuhong 10y ago
  I am taking issues like admission of wrongdoing in settlements seriously now.
- mahyarm 10y ago
  Thats what retention policies are for.
hartator 10y ago
or "Shut Everything down". Ok, that was lame.
isaac_is_goat 10y ago
And yet their database gets blown away, and now their redis is acting up. Go Gitlab.
geoelectric 10y ago
While I was at Mozilla, John O'Duinn gave a pretty great presentation about this sort of stuff. The core message (mixed with my own takeaway) was that you have to consider all of your offices, even HQ, more like just another field office or coworking site--no location is "central" compared to others. As long as you consider accommodating remote people to be a separate task, it's a task that can be deprioritized. Ideally, anyone in your office should be able to work remotely at moment's notice with little-to-no change in procedure. Much of his presentation was outlining concrete techniques towards making this actually doable. He still has the slide deck online. http://oduinn.com/blog/2014/11/09/we-are-all-remoties-nov2014/ http://oduinn.com/blog/2014/11/09/we-are-all-remoties-nov201... In practice, of course, real life was imperfect and there's no question that you take a productivity hit over people in the same room. But if you do want a heavily location-agnostic organization as a core value, his take is a nice start.
jgalt212 10y ago
160 employees? That seems like a lot support a product that is a wrapper around another product. I know the lede is skeptical, but does anyone have any idea how those numbers break down? i.e. is there a large number of sales/support due to complex installs?
pzh 10y ago
So after losing a ton of user data and repositories about a week ago, because a remote engineer couldn't bother to check to which remote machine he was issuing 'rm -rf' commands as superuser, now GitLab is teaching us its 'secret' to success?
winteriscoming 10y ago
More than knowing how they manage remote employees, at this point, with more than a week since the data loss incident, I would be genuinely curious to know if their backups are now functional and what plan they have put in place or planning to, to verify backups work. Furthermore, I won't be surprised if they have seen more concentrated spam attack, after the news of this data loss surfaced.
- YorickPeterse 10y ago
  There's a work in progress merge request for the post mortem, which can be found at https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/4779 https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/.... This blog post mentions a whole bunch of issues that are being worked on, this list can also be found at https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108 https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108. > Furthermore, I won't be surprised if they have seen more concentrated spam attack, after the news of this data loss surfaced. Not yet. Since Akismet has been enabled for snippets it seems the amount of spam is gone, or at least not noticeable.
siliconc0w 10y ago
People seem to have some subtle insecurities around broadcasting their activities which contributes a lot to the problem of distributed teams. Ideally all communication goes to everyone and everyone can apply their own client-side filters for what they want to care about. The problem is people want to present a certain face up the org and a different one to their peers so they PM each other and create isolated channels of communication (distribution lists, slack channels, etc). Now have some cognitive overhead of whom to loop in when which creates some other complications because it means if you're receiving a message it's explicit rather than implicit. TLDR - broadcast everything to everyone by default and create a culture of transparency.
- wallstquant 10y ago
  That's impressive. Do they use open channels to communicate everything? compensation? Financials?
  - mahyarm 10y ago
    Well they do have a compensation calculator on their job listings along with the reasoning
ceejay 10y ago
I think hindsight will reveal that the things that make a "distributed company" successful are really the same things that make a "localized company" successful. I think it's just that having all employees on-site probably makes it easier for companies to "fake it" and stumble into success through sheer grit and determination. Probably a lot of times even when they have less-than-adequate (but highly motivated) human resources.
- ddebernardy 10y ago
  When everyone is on site, there's enough implicit communication going on that you can get away with not documenting things in writing for a lot longer than in a remote team.
krisdevelops 10y ago
write everything down where? and in what format?
- cmatija 10y ago
  Hi Kris, We usually tend to add every single bit of information that we attain to our handbook[0]. Other than that, we try to over-communicate over issues, slack and email. [0]https://about.gitlab.com/handbook/ https://about.gitlab.com/handbook/

520 points by craigcannon 10y ago

manojlds 10y ago

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

cjbprime 10y ago
Yes, seems a little tone-deaf to publish a puff piece about GitLab at the moment.
- cookiecaper 10y ago
  Agreed. It seems that HN did delay this a bit, but they probably should've given it at least a few more weeks.
- Perihelion 10y ago
  Not that this excuses anything, but maybe it'll help explain it: This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Now that we're aware, we are working on correcting these things. But you're right, the timing of this piece was probably not ideal.
  - noir_lord 10y ago
    Not ideal perhaps but on the other hand one bad mistake doesn't negate everything you do or say forever that would be extremely unforgiving. ...and it's not like any large web company hasn't had some howlers over the years.
  - notcolin 10y ago
    > This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Could you elaborate? I recently heard about GitLab probably less than a week ago and was under the impression they're a great company to work for.
    - connorshea 10y ago
      She's referring to issues in the backup process specifically, not in the company culture (which IMO is pretty good).
    - Perihelion 10y ago
      When I say process, I mean some of our operational processes like making sure backups work. There was an incident with data loss last week: https://news.ycombinator.com/item?id=13537052 https://news.ycombinator.com/item?id=13537052 Like any company, we have our own business operations issues too but it's still one of the best jobs I've ever had. We're pretty open about our flaws, to the point where the CEO even has a page that lists his out: https://about.gitlab.com/handbook/people-operations/ceo-preferences/#flaws https://about.gitlab.com/handbook/people-operations/ceo-pref...
      - DigitalJack 10y ago
        Read The Checklist Manifesto if you haven't already. Short book, leaves a big impression.
- scrollaway 10y ago
  Why? It's not a puff piece boasting about their backup plans, they're talking about their remote work strategy. What's tone-deaf is the vitriol some people are spilling in this thread.
  - jjawssd 10y ago
    Spot on
  - sytse 10y ago
    Thanks, I really appreciate that! But to be honest I don't think it's vitriol so much as that people rightly expect better from us.
    - scrollaway 10y ago
      Some of the reactions are understandable, but I expect most people on HN to be able to separate unrelated issues like these. Some of the comments here are absolutely unnecessarily hostile.
- onion2k 10y ago
  Companies are run by people and consequently they're fallible. Every company makes mistakes occasionally. Gitlab handled theirs incredibly well. A more opaque company would have kept the problem, and the experience they earned handling it, to themselves. Gitlab's openness means we can all learn a little from their event. I think that's great. I wish more companies shared more when these things happen.
  - cjbprime 10y ago
    > A more opaque company would have kept the problem, and the experience they earned handling it, to themselves. I liked their transparency a lot too! But it's neither possible nor acceptable to keep six hours of data loss and eighteen hours of continuous downtime "to yourself".
- gukov 10y ago
  Marketing 101. Flood negative with positive.
  - sytse 10y ago
    That was not our intention. This piece was recorded before the outage. We should asked to delay publication until after the postmortem was published. That would be the marketing 101 lesson, first show you corrected your mistake instead of talking about something else.
pg_is_a_butt 10y ago
writing things down is not the problem... failing to read the things they write down is the problem. they probably don't have time to read, because they are too busy always writing. they can't find what they need to read in the sea of endless wasteful and pointless drivel that never should have been written down in the first place. you're all idiots.
cookiecaper 10y ago
I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication). GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my CoL adjustment is -40%, and I live in the United States!). They explicitly state on their hiring page that they prefer people from cheaper areas. [0] [1] On top of that, just from their headcount, it sounds like they have way too many people. 160 for a technical company like this is hard to manage when the employees are local; it's even harder to manage when employees are remote. Again, for remote work, you need mature, highly-skilled staff with a strong history of successful self-direction. You're not going to get that when you're chopping their salaries in half based on their location. Bad local wages is a major reason to work remotely in the first place. [0] https://about.gitlab.com/jobs/developer/ https://about.gitlab.com/jobs/developer/ ; use the calculator at the bottom of the page [1] https://about.gitlab.com/handbook/people-operations/global-compensation-calculator https://about.gitlab.com/handbook/people-operations/global-c... ; gems include "you are required to notify us when you move and we may adjust your compensation up or down" and "[a]ll things being equal we will hire people in lower cost markets vs. higher cost markets."
- bgxor 10y ago
  Those salary modifiers are atrocious. Their range in my city for a senior, very experienced engineer is comparable to what most entry level jobs actually offer around here. I 100% think you are on the right track. The only people from my area that would take a GitLab salary just can't get a job elsewhere.
  - pc86 10y ago
    It makes more sense for me to get a PO box in Washington DC and drive there to get my paycheck. Hell, it's probably cheaper to "rent" a single room out of someone's house just to have them forward the mail.
    - 10y ago
geofft 10y ago
"This" being remote work? Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)
- ancarda 10y ago
  It may be due to downtime earlier today (this being only days after the data loss). Things don't seem to be going well for GitLab right now.
- problems 10y ago
  > didn't do anything particularly more wrong than anyone else? This is almost completely on their admin staff, maybe other people aren't willing to say it, but I will. Test your backups. Or at least make sure they're non-zero in size. It should really be Operations 101. Whether you do this automatically or manually by setting a reminder on your calendar once a week or even month, doesn't matter. Something this simple would have solved their entire issue. I do this and we run a much smaller shop than GitLab. Heck, if we were larger I'd have hot spare database servers in another datacenter in case the primary got nuked by disk failure, network outages or mistakes.
  - fsiefken 10y ago
    As part of our disaster recovery plan when I was working as a sysadmin at a 150p company we had replicated server (database replicated and webfiles rsync) on hot standby, we just switched the front-facing servers manually. GitLab has a very short blurb on a similar styled HA setup, I'm not sure if and how they have implemented such themselves and if it would have helped in preventing or shortening the recent downtime. They have probably documented their own setup somewhere. "Automated failover can be achieved with pacemaker alongside STONITH network management. Keep in mind that application servers need to be prepared for transitioning to the new network addresses. In this situation you can also opt to synchronize the database via a database specific protocol instead of DRBD. In the documentation for each database you can find out more about the options for MySQL and the options for PostgreSQL." https://about.gitlab.com/high-availability/#filesystem-storage-and-dbms-slave-servers https://about.gitlab.com/high-availability/#filesystem-stora...
snarf21 10y ago
Scale is freaking hard. It is absurd to say they figured out the secret after a 7 layer failure on disaster recovery, even more so when one of the main reasons for source control is to have disaster recovery and the ability to rebuild old source.
- mynameisvlad 10y ago
  Wat? How is their remote working practices at all related to the database disaster? For that matter, how are either of those related to source control and rebuilding from old source? What would rebuilding source help with when you just deleted your production DB?
  - emondi 10y ago
    Not talking about this case specifically, but it could be that remote working makes defining reponsibilities harder and that could lead to not making sure that there are proper backup/restoring procedures in place.
ridruejo 10y ago
Every company goes through rough patches, specially one that is growing as fast as they are doing. Very few companies are as open as transparent as they are, even in the fuckups. That is really hard to do and kudos to them
- sytse 10y ago
  Thanks. I do agree that the timing of this post is not good. The video was recorded a few weeks ago. We are eating humble pie right now and work on fixing things instead of promotion.
  - coolsunglasses 10y ago
    I have nothing but respect for your willingness to take these hits but it's not just about you. This post is now fodder that can be used against me the next time I advocate for remote work at a company. Edit: it took 10 minutes for two people to pipe up on Twitter, one specifically suggesting remote work made the ops failure more likely: https://twitter.com/paul_snively/status/829850449247297536 https://twitter.com/paul_snively/status/829850449247297536
    - sytse 10y ago
      I'm sorry for hear we're making your advocacy for remote work harder :(
aedocw 10y ago
I don't understand why hearing some effective management practices for a distributed team would not be welcome here at any time. Their recent mistake dealt with a specific technical process and some failures around it. That incident in no way invalidates the entire company. As a remote worker myself, I'm always keen to hear about how teams make it work. I'm also a huge advocate for distributed teams (and nobody having a commute), so personally I'm really glad this link was shared today.

yunolisten 10y ago

> Write Everything Down Then rm -rf the paper.... https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/ https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/

Spooky23 10y ago
Not really germane to the topic. This type of op fuck-up happens everywhere. It's hard to build solid process, particularly in growth phases. Unless there are 2x a year restore tests, I personally assume a 60% backup fail rate.
- mbiondi 10y ago
  The only reason they are still in existence is due to a chance backup they took for a tangential reason. From the sounds of it, their solution is held together with bubble gum, some tape and lots of hand waving. Being in 160 different locations probably doesn't help much either.
  - sirclueless 10y ago
    Unless I'm misreading things, the reason they only lost 6 hours of data instead of 24 hours was a chance backup, but there was never an existential crisis here. Downtime happens to pretty much every service out there. In this case the company was incredibly forthright and so we can make fun of their stupid mistakes, but really most mistakes are stupid when you look at them -- when you make thousands of decisions a day, some of them will seem silly in hindsight. We just never learn about most of them.
  - chousuke 10y ago
    I'm pretty sure most solutions on the internet consist of bubble gum, some tape and lots of handwaving. Gitlab's screwup is hardly unique, even if it was very public. It's difficult and expensive to build and maintain a solid system, and even if you want to, time and financial pressures often just don't let you, on top of the issue of just communicating the need for solid engineering, as it usually only becomes apparent when the problems start occurring.

deepaksurti 10y ago

>> 2:41 – GitLab values boring solutions: our product should be exceptional Exceptional products have exceptional UX. Gitlab IMHO has the worst UX of all git based products out there, I much rather take BitBucket over Gitlab. I tried using Gitlab, but no, I would much rather pay the 7$ to GH for my private repos. I sincerely hope they make an exceptional product. And 'should' better be 'must'!

zenlikethat 10y ago
I'm curious what you dislike about the UX. Personally I like the "feel" of the website generally (though it's not as polished as GitHub) and just find it to be slow, which I think they're working on improving.
- bpicolo 10y ago
  Personally, not a fan of how a readme-less repository just has an entirely useless main page. I much prefer having the file view a'la github on that page. That's the main UX complaint for me. Still use gitlab bunches. Main complaint overall is the incredibly sluggish git push speed.
  - pedroms 10y ago
    Speed is one of our main priorities for GitLab this year. We've started Gitaly, a project dedicated to that: https://gitlab.com/gitlab-org/gitaly https://gitlab.com/gitlab-org/gitaly We expect it to become a standard component of GitLab in Q1 2017 and to reach full scope in Q3 2017.
    - bpicolo 10y ago
      Can't wait : )
  - eknkc 10y ago
    You can change that on your personal prefs to show the file list by default, with Readme on bottom: http://c.ekin.io/3z1V2v3P041b http://c.ekin.io/3z1V2v3P041b git push speed is fine on our own ce installation but gitlab.com is beyond shitty.

MrFurious 10y ago

Obviously, Gitlab don't have the secret for best uptime.

uladzislau 10y ago

Obviously what they say and what they do are two different things based on the recent events.

WalterSear 10y ago

And don't store it on gitlab.

newsat13 10y ago

Last I checked a year ago, GitLab had just 80 employees. So, it's now doubled. That is quite some growth. More often than not growing companies just collect lots of employees and then a startup comes and beats them...

Cofike 10y ago

Judging by the number of joke and low hanging responses on this thread I think they should have held this article for a little longer.

sytse 10y ago
I think we earned a reality check. Have such a promotional video go out after the outage was out of touch on our part. We were aware it would be posted and didn't ask to hold it.

Kiro 10y ago

Given the complete backlash I'm seeing on HN it seems like transparency actually hurt you. Very sad.

gukov 10y ago
It's the business-as-usual attitude and acting as if nothing happened right after a catastrophic event that riles people up. Transparency is fine. Eventually moving on is fine. But right now people are interested only in one topic when it comes to Gitlab.
- Kiro 10y ago
  I think a lot has to do with the reason for the failure being perceived as sloppy and irresponsible. Without the transparency we wouldn't be able to bash them like this. Many of the sarcastic comments here refer to the failed processes they revealed in the postmortem.
sytse 10y ago
We're still committed to transparency even if it means taking a few lumps now and then, it's better in the long run and it's baked into everything we do.
- gtaylor 10y ago
  I think many or even most appreciate your stance on this. Please don't get discouraged by the vocal but petty minority!
- photonwins 10y ago
  Kudos to the entire team, as another commenter said, please don't be discouraged by the petty comments. It takes balls to admit mistakes in public. You have earned many people's respect.
- lightbulbjim 10y ago
  Please keep it up. We need more companies like you.

Philipp__ 10y ago

Everybody is prone to making a mistake! As long as you learn something from it, so that it never happens again, it's ok. And I really don't get why people give them bad rep for this post. I mean it's obvious why it's posted and we all know what happened week ago, but consider their size and scale, it's not easy to manage so many people remotely.

snarf21 10y ago
"...but consider their size and scale, it's not easy to manage so many people remotely." This is very true but the title of the post is "GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down". That is the backlash, don't say you've figured out the secret then make a near catastrophic mistake.
- e40 10y ago
  I don't see how their remote structure has anything to do with their mistake with backups, etc talked about here lately.

JustSomeNobody 10y ago

Wow. You all in the comments must be a bunch of perfect people, huh? Good grief. Also, nice cameo by the office cat.

overcast 10y ago
Imperfect doesn't mean you can be completely negligent of basics.
sytse 10y ago
Thanks, but the critics are right. This is not a good time to be promoting ourselves. We should be working on our postmortem and figure out how to improve our service. We're actually doing that, this video was recorded before the outage. But impressions count.

bryanh 10y ago

160 employees remote is impressive and commendable. Zapier is fully remote as well (but half the size in employee count). I'd say "write everything down" is a great shortcut to the sorts of practices you need to cultivate. We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural - organizations become complex as they grow, and individuals are busy doing their thing. You often have to bring the important data to them. On another note, working remote is awesome. I recommend everyone give it a spin once in their careers - but try to find a team that embraces it. I've heard mixed experiences from those who were the single remote person on a team.

connorshea 10y ago
I didn't know Zapier was a fully remote company, that's awesome :) I pretty much agree with your thoughts about over-communicating, it's super important and we've run into exactly the same things, where people aren't always caught up on everything going on at the company. In my experience this happens in physical workplaces as well, just to a lesser extent.
Perihelion 10y ago
Honestly, I haven't seen a company that does the hybrid remote/on-site thing well. I'm sure they exist, but every time I've worked at one the people who were remote were out of the loop on almost everything. Hybrid remote/on-site requires some great discipline. If you have a watercooler chat with someone about a feature, it's easy to forget that a remote colleague wasn't there for the discussion/not document what was said. +1 for working remotely on a team that makes an effort to make it work.
- sotojuan 10y ago
  It works at our company because the only remote guy is the only one in his "domain" (iOS). He probably doesn't care about the frontend watercooler chat and anything important about the backend will be mentioned in standup.

the-dude 10y ago

I have the impression this is being moved off the frontpage in an accelerated manner ...

the-dude 10y ago
And it is back up, from place 22 to 12 again.
- dang 10y ago
  Sometimes software and moderators do the same thing, which doubles the effect. When we notice that, we turn off the moderation and just rely on the software. That's why this one fell and then went back up.
- johansch 10y ago
  https://about.gitlab.com/2015/03/04/gitlab-is-part-of-the-y-combinator-family/ https://about.gitlab.com/2015/03/04/gitlab-is-part-of-the-y-...

Paul-ish 10y ago

This is interesting, as I interned at Mozilla, and they have a large number of remote employees. There was only one person (other than me) on my team in my location. Mozilla also records a lot of things, but the intent is more to publish that stuff publicly, in the spirit of openness towards the community. For example, all my team meetings had public notes. I wouldn't be surprised if this tendency to record everything for the sake of the community also has positive benefits inside Mozilla.

deleted 10y ago

[deleted]

dandersh 10y ago

The most frustrating thing about working remote was the complete lack of documentation around anything -- meetings, requirements, internal frameworks/libraries, etc. Good for them for emphasizing documentation. I lost track of the amount of times I was told to "look at the source code" when asking about documentation for the internal framework. Also fun was being assigned a feature and having it's functionality explained over the phone, with any subsequent follow-up being met with "Didn't we already talk about this"? I never would have thought that a remote based team would have the worst documentation out of anywhere I've worked, but that's exactly what happened.

bgaluszka 10y ago
> I lost track of the amount of times I was told to "look at the source code" when asking about documentation for the internal framework. Can you elaborate on this one? This is my preferred way.
- overcast 10y ago
  You prefer just being handed a pile of code, rather than documentation on how to use it? please.
  - bgaluszka 10y ago
    In case where I work for a given company and it's internal tool, always.
    - crpatino 10y ago
      Imagine this scenario. You work on product A. Your VP of very important things declares that every product should use library L. Library L has been designed to incestously interact with product D and makes a bunch of assumptions about what the "working environment" should be. Teams B and C have already made the transition, so L made some kludgy APIs to make B and C interaction possible, though it is maintained with low priority because everybody knows D is the money maker. Both B and C have a bunch of assumptions in common that A does not share, and they both had to compromise and emulate D environment to some exent. You are given L binaries, a few toy examples of how to use the most basic functionality in L, and access to B codebase. L codebase does not use the same source control system, so you don't even know where to begin looking for their code. Please keep in mind that your manager does not expect you to learn all the inner workings of L. Your priority task is to use L to support some new feature in A. You still have a number of other tasks to accomplish in A that are not related to L, and some of those might raise in priority with little or no notice. Do you still think it is a good idea to just browse L code, without knowing how large (or well writen) it is?

nedsma 10y ago

Glad that Gitlab is pushing the remote working boundaries even further. Yes, document everything, write easy to understand code, offer your remote coworkers to call you whenever they need help, leave feedback, praise work as if they're local. It all matters and even more when remotely.

smarx007 10y ago

It would be so much better if YC would made it a podcast instead (or in addition). Would be a perfect thing to listen on the go.

tra3 10y ago

How do you organize internal knowledge base?

johansch 10y ago

I mean, really: after figuring out after the fact that five out of five backups were malfunctioning - go here to read our secret to managing 160 employees remotely? Nah... no thank you. I think I will get my management advice from a company that is not totally broken. I don't care how transparent you are, or how remote-worker friendly you are, if you can't do the basics right. You are in the data storage business! Shut down the company and refund the money to the investors.

whoiskevin 10y ago

I have to watch a video that then tells me to write everything down? Sorry I couldn't resist. I believe and practice that as much as I can because it makes things easier regardless.

chmars 10y ago

What kind of tool would you recommend to write everything down, especially a self-hosted tool?

peterwwillis 10y ago

I think the worst team I've ever worked on was the one that had a team lead that literally never wrote anything down. Everything was verbal, and what wasn't verbal was private messaged. We would have the same discussion three times because nobody wrote it down the first or second time. Nobody even knew what the changes to the formal design were, because they never had a documented change review. I always tell people on my team that if it isn't written down, it doesn't exist. When you get into the habit (and learn how to manage all that information) it really saves your bacon. The other thing that would have saved them from the backup drama was learning how to document and date your to-do list. Suddenly someone notices that you've had a test process for backups on the to-do for over a year and they bring it up at the next meeting. And another thing: in team meetings, people can write their own meeting notes so everyone becomes responsible for documenting their responsibilities and what affects them. It's easier to do this remote than at a stand-up, because you're already at your keyboard.

gravypod 10y ago

I can't wait for a remote-work company that has a policy like this to get into some kind of legal troubles that warrents subpoenas. This is a prosecutors gold mine. I feel bad for the defense lawers already.

yuhong 10y ago
I am taking issues like admission of wrongdoing in settlements seriously now.
mahyarm 10y ago
Thats what retention policies are for.

hartator 10y ago

or "Shut Everything down". Ok, that was lame.

isaac_is_goat 10y ago

And yet their database gets blown away, and now their redis is acting up. Go Gitlab.

geoelectric 10y ago

While I was at Mozilla, John O'Duinn gave a pretty great presentation about this sort of stuff. The core message (mixed with my own takeaway) was that you have to consider all of your offices, even HQ, more like just another field office or coworking site--no location is "central" compared to others. As long as you consider accommodating remote people to be a separate task, it's a task that can be deprioritized. Ideally, anyone in your office should be able to work remotely at moment's notice with little-to-no change in procedure. Much of his presentation was outlining concrete techniques towards making this actually doable. He still has the slide deck online. http://oduinn.com/blog/2014/11/09/we-are-all-remoties-nov2014/ http://oduinn.com/blog/2014/11/09/we-are-all-remoties-nov201... In practice, of course, real life was imperfect and there's no question that you take a productivity hit over people in the same room. But if you do want a heavily location-agnostic organization as a core value, his take is a nice start.

jgalt212 10y ago

160 employees? That seems like a lot support a product that is a wrapper around another product. I know the lede is skeptical, but does anyone have any idea how those numbers break down? i.e. is there a large number of sales/support due to complex installs?

pzh 10y ago

So after losing a ton of user data and repositories about a week ago, because a remote engineer couldn't bother to check to which remote machine he was issuing 'rm -rf' commands as superuser, now GitLab is teaching us its 'secret' to success?

winteriscoming 10y ago

More than knowing how they manage remote employees, at this point, with more than a week since the data loss incident, I would be genuinely curious to know if their backups are now functional and what plan they have put in place or planning to, to verify backups work. Furthermore, I won't be surprised if they have seen more concentrated spam attack, after the news of this data loss surfaced.

YorickPeterse 10y ago
There's a work in progress merge request for the post mortem, which can be found at https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/4779 https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/.... This blog post mentions a whole bunch of issues that are being worked on, this list can also be found at https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108 https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108. > Furthermore, I won't be surprised if they have seen more concentrated spam attack, after the news of this data loss surfaced. Not yet. Since Akismet has been enabled for snippets it seems the amount of spam is gone, or at least not noticeable.

siliconc0w 10y ago

People seem to have some subtle insecurities around broadcasting their activities which contributes a lot to the problem of distributed teams. Ideally all communication goes to everyone and everyone can apply their own client-side filters for what they want to care about. The problem is people want to present a certain face up the org and a different one to their peers so they PM each other and create isolated channels of communication (distribution lists, slack channels, etc). Now have some cognitive overhead of whom to loop in when which creates some other complications because it means if you're receiving a message it's explicit rather than implicit. TLDR - broadcast everything to everyone by default and create a culture of transparency.

wallstquant 10y ago
That's impressive. Do they use open channels to communicate everything? compensation? Financials?
- mahyarm 10y ago
  Well they do have a compensation calculator on their job listings along with the reasoning

ceejay 10y ago

I think hindsight will reveal that the things that make a "distributed company" successful are really the same things that make a "localized company" successful. I think it's just that having all employees on-site probably makes it easier for companies to "fake it" and stumble into success through sheer grit and determination. Probably a lot of times even when they have less-than-adequate (but highly motivated) human resources.

ddebernardy 10y ago
When everyone is on site, there's enough implicit communication going on that you can get away with not documenting things in writing for a lot longer than in a remote team.

krisdevelops 10y ago

write everything down where? and in what format?

cmatija 10y ago
Hi Kris, We usually tend to add every single bit of information that we attain to our handbook[0]. Other than that, we try to over-communicate over issues, slack and email. [0]https://about.gitlab.com/handbook/ https://about.gitlab.com/handbook/