11 ms·

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

by manojlds 10y ago

cjbprime 10y ago
Yes, seems a little tone-deaf to publish a puff piece about GitLab at the moment.
- cookiecaper 10y ago
  Agreed. It seems that HN did delay this a bit, but they probably should've given it at least a few more weeks.
- Perihelion 10y ago
  Not that this excuses anything, but maybe it'll help explain it: This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Now that we're aware, we are working on correcting these things. But you're right, the timing of this piece was probably not ideal.
  - noir_lord 10y ago
    Not ideal perhaps but on the other hand one bad mistake doesn't negate everything you do or say forever that would be extremely unforgiving. ...and it's not like any large web company hasn't had some howlers over the years.
  - notcolin 10y ago
    > This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Could you elaborate? I recently heard about GitLab probably less than a week ago and was under the impression they're a great company to work for.
    - connorshea 10y ago
      She's referring to issues in the backup process specifically, not in the company culture (which IMO is pretty good).
    - Perihelion 10y ago
      When I say process, I mean some of our operational processes like making sure backups work. There was an incident with data loss last week: https://news.ycombinator.com/item?id=13537052 https://news.ycombinator.com/item?id=13537052 Like any company, we have our own business operations issues too but it's still one of the best jobs I've ever had. We're pretty open about our flaws, to the point where the CEO even has a page that lists his out: https://about.gitlab.com/handbook/people-operations/ceo-preferences/#flaws https://about.gitlab.com/handbook/people-operations/ceo-pref...
      - DigitalJack 10y ago
        Read The Checklist Manifesto if you haven't already. Short book, leaves a big impression.
- scrollaway 10y ago
  Why? It's not a puff piece boasting about their backup plans, they're talking about their remote work strategy. What's tone-deaf is the vitriol some people are spilling in this thread.
  - jjawssd 10y ago
    Spot on
  - sytse 10y ago
    Thanks, I really appreciate that! But to be honest I don't think it's vitriol so much as that people rightly expect better from us.
    - scrollaway 10y ago
      Some of the reactions are understandable, but I expect most people on HN to be able to separate unrelated issues like these. Some of the comments here are absolutely unnecessarily hostile.
- onion2k 10y ago
  Companies are run by people and consequently they're fallible. Every company makes mistakes occasionally. Gitlab handled theirs incredibly well. A more opaque company would have kept the problem, and the experience they earned handling it, to themselves. Gitlab's openness means we can all learn a little from their event. I think that's great. I wish more companies shared more when these things happen.
  - cjbprime 10y ago
    > A more opaque company would have kept the problem, and the experience they earned handling it, to themselves. I liked their transparency a lot too! But it's neither possible nor acceptable to keep six hours of data loss and eighteen hours of continuous downtime "to yourself".
- gukov 10y ago
  Marketing 101. Flood negative with positive.
  - sytse 10y ago
    That was not our intention. This piece was recorded before the outage. We should asked to delay publication until after the postmortem was published. That would be the marketing 101 lesson, first show you corrected your mistake instead of talking about something else.
pg_is_a_butt 10y ago
writing things down is not the problem... failing to read the things they write down is the problem. they probably don't have time to read, because they are too busy always writing. they can't find what they need to read in the sea of endless wasteful and pointless drivel that never should have been written down in the first place. you're all idiots.
cookiecaper 10y ago
I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication). GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my CoL adjustment is -40%, and I live in the United States!). They explicitly state on their hiring page that they prefer people from cheaper areas. [0] [1] On top of that, just from their headcount, it sounds like they have way too many people. 160 for a technical company like this is hard to manage when the employees are local; it's even harder to manage when employees are remote. Again, for remote work, you need mature, highly-skilled staff with a strong history of successful self-direction. You're not going to get that when you're chopping their salaries in half based on their location. Bad local wages is a major reason to work remotely in the first place. [0] https://about.gitlab.com/jobs/developer/ https://about.gitlab.com/jobs/developer/ ; use the calculator at the bottom of the page [1] https://about.gitlab.com/handbook/people-operations/global-compensation-calculator https://about.gitlab.com/handbook/people-operations/global-c... ; gems include "you are required to notify us when you move and we may adjust your compensation up or down" and "[a]ll things being equal we will hire people in lower cost markets vs. higher cost markets."
- bgxor 10y ago
  Those salary modifiers are atrocious. Their range in my city for a senior, very experienced engineer is comparable to what most entry level jobs actually offer around here. I 100% think you are on the right track. The only people from my area that would take a GitLab salary just can't get a job elsewhere.
  - pc86 10y ago
    It makes more sense for me to get a PO box in Washington DC and drive there to get my paycheck. Hell, it's probably cheaper to "rent" a single room out of someone's house just to have them forward the mail.
    - 10y ago
geofft 10y ago
"This" being remote work? Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)
- ancarda 10y ago
  It may be due to downtime earlier today (this being only days after the data loss). Things don't seem to be going well for GitLab right now.
- problems 10y ago
  > didn't do anything particularly more wrong than anyone else? This is almost completely on their admin staff, maybe other people aren't willing to say it, but I will. Test your backups. Or at least make sure they're non-zero in size. It should really be Operations 101. Whether you do this automatically or manually by setting a reminder on your calendar once a week or even month, doesn't matter. Something this simple would have solved their entire issue. I do this and we run a much smaller shop than GitLab. Heck, if we were larger I'd have hot spare database servers in another datacenter in case the primary got nuked by disk failure, network outages or mistakes.
  - fsiefken 10y ago
    As part of our disaster recovery plan when I was working as a sysadmin at a 150p company we had replicated server (database replicated and webfiles rsync) on hot standby, we just switched the front-facing servers manually. GitLab has a very short blurb on a similar styled HA setup, I'm not sure if and how they have implemented such themselves and if it would have helped in preventing or shortening the recent downtime. They have probably documented their own setup somewhere. "Automated failover can be achieved with pacemaker alongside STONITH network management. Keep in mind that application servers need to be prepared for transitioning to the new network addresses. In this situation you can also opt to synchronize the database via a database specific protocol instead of DRBD. In the documentation for each database you can find out more about the options for MySQL and the options for PostgreSQL." https://about.gitlab.com/high-availability/#filesystem-storage-and-dbms-slave-servers https://about.gitlab.com/high-availability/#filesystem-stora...
snarf21 10y ago
Scale is freaking hard. It is absurd to say they figured out the secret after a 7 layer failure on disaster recovery, even more so when one of the main reasons for source control is to have disaster recovery and the ability to rebuild old source.
- mynameisvlad 10y ago
  Wat? How is their remote working practices at all related to the database disaster? For that matter, how are either of those related to source control and rebuilding from old source? What would rebuilding source help with when you just deleted your production DB?
  - emondi 10y ago
    Not talking about this case specifically, but it could be that remote working makes defining reponsibilities harder and that could lead to not making sure that there are proper backup/restoring procedures in place.
ridruejo 10y ago
Every company goes through rough patches, specially one that is growing as fast as they are doing. Very few companies are as open as transparent as they are, even in the fuckups. That is really hard to do and kudos to them
- sytse 10y ago
  Thanks. I do agree that the timing of this post is not good. The video was recorded a few weeks ago. We are eating humble pie right now and work on fixing things instead of promotion.
  - coolsunglasses 10y ago
    I have nothing but respect for your willingness to take these hits but it's not just about you. This post is now fodder that can be used against me the next time I advocate for remote work at a company. Edit: it took 10 minutes for two people to pipe up on Twitter, one specifically suggesting remote work made the ops failure more likely: https://twitter.com/paul_snively/status/829850449247297536 https://twitter.com/paul_snively/status/829850449247297536
    - sytse 10y ago
      I'm sorry for hear we're making your advocacy for remote work harder :(
aedocw 10y ago
I don't understand why hearing some effective management practices for a distributed team would not be welcome here at any time. Their recent mistake dealt with a specific technical process and some failures around it. That incident in no way invalidates the entire company. As a remote worker myself, I'm always keen to hear about how teams make it work. I'm also a huge advocate for distributed teams (and nobody having a commute), so personally I'm really glad this link was shared today.

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

cjbprime 10y ago

Yes, seems a little tone-deaf to publish a puff piece about GitLab at the moment.

cookiecaper 10y ago
Agreed. It seems that HN did delay this a bit, but they probably should've given it at least a few more weeks.
Perihelion 10y ago
Not that this excuses anything, but maybe it'll help explain it: This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Now that we're aware, we are working on correcting these things. But you're right, the timing of this piece was probably not ideal.
- noir_lord 10y ago
  Not ideal perhaps but on the other hand one bad mistake doesn't negate everything you do or say forever that would be extremely unforgiving. ...and it's not like any large web company hasn't had some howlers over the years.
- notcolin 10y ago
  > This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Could you elaborate? I recently heard about GitLab probably less than a week ago and was under the impression they're a great company to work for.
  - connorshea 10y ago
    She's referring to issues in the backup process specifically, not in the company culture (which IMO is pretty good).
  - Perihelion 10y ago
    When I say process, I mean some of our operational processes like making sure backups work. There was an incident with data loss last week: https://news.ycombinator.com/item?id=13537052 https://news.ycombinator.com/item?id=13537052 Like any company, we have our own business operations issues too but it's still one of the best jobs I've ever had. We're pretty open about our flaws, to the point where the CEO even has a page that lists his out: https://about.gitlab.com/handbook/people-operations/ceo-preferences/#flaws https://about.gitlab.com/handbook/people-operations/ceo-pref...
    - DigitalJack 10y ago
      Read The Checklist Manifesto if you haven't already. Short book, leaves a big impression.
scrollaway 10y ago
Why? It's not a puff piece boasting about their backup plans, they're talking about their remote work strategy. What's tone-deaf is the vitriol some people are spilling in this thread.
- jjawssd 10y ago
  Spot on
- sytse 10y ago
  Thanks, I really appreciate that! But to be honest I don't think it's vitriol so much as that people rightly expect better from us.
  - scrollaway 10y ago
    Some of the reactions are understandable, but I expect most people on HN to be able to separate unrelated issues like these. Some of the comments here are absolutely unnecessarily hostile.
onion2k 10y ago
Companies are run by people and consequently they're fallible. Every company makes mistakes occasionally. Gitlab handled theirs incredibly well. A more opaque company would have kept the problem, and the experience they earned handling it, to themselves. Gitlab's openness means we can all learn a little from their event. I think that's great. I wish more companies shared more when these things happen.
- cjbprime 10y ago
  > A more opaque company would have kept the problem, and the experience they earned handling it, to themselves. I liked their transparency a lot too! But it's neither possible nor acceptable to keep six hours of data loss and eighteen hours of continuous downtime "to yourself".
gukov 10y ago
Marketing 101. Flood negative with positive.
- sytse 10y ago
  That was not our intention. This piece was recorded before the outage. We should asked to delay publication until after the postmortem was published. That would be the marketing 101 lesson, first show you corrected your mistake instead of talking about something else.

pg_is_a_butt 10y ago

writing things down is not the problem... failing to read the things they write down is the problem. they probably don't have time to read, because they are too busy always writing. they can't find what they need to read in the sea of endless wasteful and pointless drivel that never should have been written down in the first place. you're all idiots.

cookiecaper 10y ago

I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication). GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my CoL adjustment is -40%, and I live in the United States!). They explicitly state on their hiring page that they prefer people from cheaper areas. [0] [1] On top of that, just from their headcount, it sounds like they have way too many people. 160 for a technical company like this is hard to manage when the employees are local; it's even harder to manage when employees are remote. Again, for remote work, you need mature, highly-skilled staff with a strong history of successful self-direction. You're not going to get that when you're chopping their salaries in half based on their location. Bad local wages is a major reason to work remotely in the first place. [0] https://about.gitlab.com/jobs/developer/ https://about.gitlab.com/jobs/developer/ ; use the calculator at the bottom of the page [1] https://about.gitlab.com/handbook/people-operations/global-compensation-calculator https://about.gitlab.com/handbook/people-operations/global-c... ; gems include "you are required to notify us when you move and we may adjust your compensation up or down" and "[a]ll things being equal we will hire people in lower cost markets vs. higher cost markets."

bgxor 10y ago
Those salary modifiers are atrocious. Their range in my city for a senior, very experienced engineer is comparable to what most entry level jobs actually offer around here. I 100% think you are on the right track. The only people from my area that would take a GitLab salary just can't get a job elsewhere.
- pc86 10y ago
  It makes more sense for me to get a PO box in Washington DC and drive there to get my paycheck. Hell, it's probably cheaper to "rent" a single room out of someone's house just to have them forward the mail.
  - 10y ago

geofft 10y ago

"This" being remote work? Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)

ancarda 10y ago
It may be due to downtime earlier today (this being only days after the data loss). Things don't seem to be going well for GitLab right now.
problems 10y ago
> didn't do anything particularly more wrong than anyone else? This is almost completely on their admin staff, maybe other people aren't willing to say it, but I will. Test your backups. Or at least make sure they're non-zero in size. It should really be Operations 101. Whether you do this automatically or manually by setting a reminder on your calendar once a week or even month, doesn't matter. Something this simple would have solved their entire issue. I do this and we run a much smaller shop than GitLab. Heck, if we were larger I'd have hot spare database servers in another datacenter in case the primary got nuked by disk failure, network outages or mistakes.
- fsiefken 10y ago
  As part of our disaster recovery plan when I was working as a sysadmin at a 150p company we had replicated server (database replicated and webfiles rsync) on hot standby, we just switched the front-facing servers manually. GitLab has a very short blurb on a similar styled HA setup, I'm not sure if and how they have implemented such themselves and if it would have helped in preventing or shortening the recent downtime. They have probably documented their own setup somewhere. "Automated failover can be achieved with pacemaker alongside STONITH network management. Keep in mind that application servers need to be prepared for transitioning to the new network addresses. In this situation you can also opt to synchronize the database via a database specific protocol instead of DRBD. In the documentation for each database you can find out more about the options for MySQL and the options for PostgreSQL." https://about.gitlab.com/high-availability/#filesystem-storage-and-dbms-slave-servers https://about.gitlab.com/high-availability/#filesystem-stora...

snarf21 10y ago

Scale is freaking hard. It is absurd to say they figured out the secret after a 7 layer failure on disaster recovery, even more so when one of the main reasons for source control is to have disaster recovery and the ability to rebuild old source.

mynameisvlad 10y ago
Wat? How is their remote working practices at all related to the database disaster? For that matter, how are either of those related to source control and rebuilding from old source? What would rebuilding source help with when you just deleted your production DB?
- emondi 10y ago
  Not talking about this case specifically, but it could be that remote working makes defining reponsibilities harder and that could lead to not making sure that there are proper backup/restoring procedures in place.

ridruejo 10y ago

Every company goes through rough patches, specially one that is growing as fast as they are doing. Very few companies are as open as transparent as they are, even in the fuckups. That is really hard to do and kudos to them

sytse 10y ago
Thanks. I do agree that the timing of this post is not good. The video was recorded a few weeks ago. We are eating humble pie right now and work on fixing things instead of promotion.
- coolsunglasses 10y ago
  I have nothing but respect for your willingness to take these hits but it's not just about you. This post is now fodder that can be used against me the next time I advocate for remote work at a company. Edit: it took 10 minutes for two people to pipe up on Twitter, one specifically suggesting remote work made the ops failure more likely: https://twitter.com/paul_snively/status/829850449247297536 https://twitter.com/paul_snively/status/829850449247297536
  - sytse 10y ago
    I'm sorry for hear we're making your advocacy for remote work harder :(

aedocw 10y ago

I don't understand why hearing some effective management practices for a distributed team would not be welcome here at any time. Their recent mistake dealt with a specific technical process and some failures around it. That incident in no way invalidates the entire company. As a remote worker myself, I'm always keen to hear about how teams make it work. I'm also a huge advocate for distributed teams (and nobody having a commute), so personally I'm really glad this link was shared today.