9 ms·

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

dentemple 10y ago
Any publicity is good publicity, right? :D
testUser69 10y ago
I think the "business people" are people who hide their technological inadequacy behind supposed business expertise. A businesses success is not determined by whether or not they use tools that adhere to business micro-cultural "values". If "too much transparency" is a turn off for you, you're probably just an authoritarian trying to scheme and scam your way into profit, and you probably lack the confidence required to put whatever skill you think you have on display.
- GavinMcG 10y ago
  I suspect the comment you're replying to meant that the business people might feel negatively about trusting the company that lost data, regardless of transparency.
- tyre 10y ago
  This demonstrates a remarkable lack of empathy. Try understanding why people on the business side do what they do—it's almost never out of malice or desired authoritarianism. Working with people is hard, specifically because you have to get outside of your head and think about how they see the world.
- leesalminen 10y ago
  At my company we always heir on the side of transparency and liberally post to our status. subdomain whenever an issue is identified. Recently, our Redis cluster failed and both master & slave host machines rebooted. This caused a latency spike in our app, from roughly a 70ms to 400ms response time for less than 10 minutes. We posted to status within 60 seconds and posted 3 updates within those 10 minutes. The next day, a new customer (who hadn't gone live with the app yet) cancelled their subscription because the app was "not reliable". I guess my point is that there is a balance to strike. Our customers are not tech-savvy in any way and treat any small issue as the end of the world. Maybe there's no need to freak people out for a minutes-long latency spike.
  - jasonm23 10y ago
    FYI https://en.oxforddictionaries.com/definition/err_on_the_side_of_caution https://en.oxforddictionaries.com/definition/err_on_the_side... Just in case you ever use that phrase in critical correspondence. Better to err on the side of caution.
    - leesalminen 10y ago
      Ah, thanks! It didn't feel right but was too lazy to check. Will definitely remember this for next time.
- debaserab2 10y ago
  It really wouldn't have been that hard to write your opinion in a way that doesn't disparage the character of the person (that you don't really know) that you're responding to. There's no reason to hide behind a new account other than to be an asshole.
empath75 10y ago
Yeah it made me not consider using them for anything other than toy projects.
- balls187 10y ago
  Why? Everyone has their own copy of the gitlab remote.
  - BHSPitMonkey 10y ago
    Not everyone has their own copy of the project's issues/pull requests/settings/webhooks/CI config/etc., which the incident affected (and 6 hours worth were lost). Git repositories themselves do have the happy side effect that at least someone on your team has a full latest clone on their machine, but that's only because the source code is what you're there to use every day. Even if GitLab had a great "checkout your issues/etc. as a git repo" story, not many users would have an up-to-date copy backed up 100% of the time.
    - balls187 10y ago
      Right, there was data loss, and it was not trivial. Does that mean you should only use gitlab for toy projects? I don't think so. I think they'll quickly learn from this.
    - taneq 10y ago
      Maybe a dumb question but couldn't they be included in a hidden folder in the root of your repo or something? That way they'd get replicated automatically every time someone does a pull.
      - Ajedi32 10y ago
        It'd probably have to be on a separate branch, but yeah they technically _could_ do that. Maybe something like this? https://github.com/duplys/git-issues https://github.com/duplys/git-issues
  - 10y ago
djsumdog 10y ago
I don't think this affects to many of the larger shops, since they tend to host their own GitLab instances (typically for security, compliance, legal, etc.) My current shop runs GitLab community edition on an internal cluster.
jchook 10y ago
All press is good press.
Kiro 10y ago
> I believe among other developers I don't know. I'm very hesitant trying out GitLab now whereas I was interested before.
synicalx 10y ago
Getting better at ops? Checking backups regularly to make sure they work is not a new thing.
praneshp 10y ago
HN can be funny sometimes. GitHub got a lot of hate about a year ago just for not releasing new features. GitLab cost everyone a day because their backup/ops practices were silly, and everyone loves them more. I've screwed up before, and I sympathize/empathize with their ops folks, but this should make us think about plan B in case something like this happens again.
- vollmond 10y ago
  There's another reasonable takeaway: GitLab will probably now prepare for this sort of thing better than the average competitor who hasn't been bitten by this.
  - ryanplant-au 10y ago
    Borges: "In Alexandria, it has been said that the only persons incapable of a sin are those who have already committed it and repented."
  - sytse 10y ago
    We'll certainly learn from this, post-mortem write-up issue is in https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108 https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108
- jopsen 10y ago
  Nah, no need to think about backup now. Stuff like this only happens once :) haha, To be fair, I suspect gitlab will a adopt a 'never-again' policy towards making sure their backups work.
- deckar01 10y ago
  > GitLab cost everyone a day GitLab isn't popular because of the stability of its cloud platform. It's popular because you can install your own instance for free practically anywhere with minimal effort. I run GitLab CE on a box in my server closet for projects that involve livelihoods.
  - 10y ago
mercwear 10y ago
I am sick of the "They are transparent so it's ok" argument. The fact is this type of thing should not happen anymore.. I get it, stuff fails but it's not acceptable to lose production data at this scale and save face by being transparent. That was ok back in 2003.. Not anymore.
- seanp2k2 10y ago
  Compare with Github and their 98.935% uptime over the past month: https://status.github.com/graphs/past_month https://status.github.com/graphs/past_month Neither should be exclusively relied upon for business-critical services. Ask Github about their production backups and DR plans some time.
  - mst 10y ago
    I still remember the outage where github ran their tests against production - which started by TRUNACTEing everything before loading fixtures. I think "everybody learns the hard way" is just one of those things with operations.
- anonymous_green 10y ago
  Probably they cut cost by hiring a group of really incompetent set of developers because they are a "startup" and "transparent" and "its ok". Nothing could make state of the company more "transparent" than that. xD
manigandham 10y ago
Transparency is not a substitute for competency. Both are important, but I'll take the latter every time.
rdslw 10y ago
Just to stress one thing, while mentioned in their report, most people don't comprehend: Gitlab was _one_ (last!) final step from complete data loss of everything. One. At that night, there was quite a long moment there had only one copy (and 6 hours old). Every other backup was missing/notworking/deleted. This is scary.
- crististm 10y ago
  This is the fact that deserves more attention. The operator did a manual backup before work which was the only available one. I find this to be an eye opener. The real problem was dodged by doing the right thing in the proverbial last minute (6 hours).
  - s_kilk 10y ago
    In other words, there is an alternate universe in which Gitlab is now categorically dead, all because that employee didn't make that single backup.
- garaetjjte 10y ago
  Didn't they had two copies? (staging server and LVM snapshot)
anonymous_green 10y ago
Transparency? Gitlab is the company that interviews people and rejects them on basis of salary, they've been doing this for awhile, every few months they call a group of people, waste their time and then deny based on the salary. In my case, I've confronted them that USD 100-120k is the market average but they had this stupid startup argument that doesn't make any sense to me.
- hobarrera 10y ago
  > USD 100-120k is the market average In the US maybe, but given that they're a remote company with people all over the world, they don't necessarily have to live by US standard (especially when it comes to spending!). 40k USD is a VERY good developer salary in Argentina, and I'd bet that in other countries a lower figure might make the cut too. I can definitely understand why they'd hesitate to pay thrice that amount. I believe it's over 5 times the average salary.
  - anonymous_green 10y ago
    Yes, but if you look at the average here its not really outrageous or "5 times the average salary" as you said, here: https://www.indeed.com/salaries/Gitlab-Salaries https://www.indeed.com/salaries/Gitlab-Salaries And its not just me, they've done this in the past as well: https://news.ycombinator.com/item?id=10924957 https://news.ycombinator.com/item?id=10924957 Also when you work for a remote company, they are cutting cost in terms of office and all which should reflect back in the salaries, the whole idea of working remotely was to mutually benefit both the employeer and employee and not just gitlab using their whole startup argument to cop out when they want on that "truly remote" so-called transparent company.
    - hobarrera 10y ago
      Like I said, they're a remote company, and therefore can hire people in cheaper regions, and don't have to adapt to more expensive markets (like your own). I fail to see how this disproves it, it merely proves that you're not a good choice for them (because you live in a, relatively, expensive region).
spinlock 10y ago
For me it underscores one simple rule: you backups are useless if you don't regularly practice restoring from them.
- xenithorb 10y ago
  Or I don't know, at least checking they exist

dentemple 10y ago

Any publicity is good publicity, right? :D

testUser69 10y ago

I think the "business people" are people who hide their technological inadequacy behind supposed business expertise. A businesses success is not determined by whether or not they use tools that adhere to business micro-cultural "values". If "too much transparency" is a turn off for you, you're probably just an authoritarian trying to scheme and scam your way into profit, and you probably lack the confidence required to put whatever skill you think you have on display.

GavinMcG 10y ago
I suspect the comment you're replying to meant that the business people might feel negatively about trusting the company that lost data, regardless of transparency.
tyre 10y ago
This demonstrates a remarkable lack of empathy. Try understanding why people on the business side do what they do—it's almost never out of malice or desired authoritarianism. Working with people is hard, specifically because you have to get outside of your head and think about how they see the world.
leesalminen 10y ago
At my company we always heir on the side of transparency and liberally post to our status. subdomain whenever an issue is identified. Recently, our Redis cluster failed and both master & slave host machines rebooted. This caused a latency spike in our app, from roughly a 70ms to 400ms response time for less than 10 minutes. We posted to status within 60 seconds and posted 3 updates within those 10 minutes. The next day, a new customer (who hadn't gone live with the app yet) cancelled their subscription because the app was "not reliable". I guess my point is that there is a balance to strike. Our customers are not tech-savvy in any way and treat any small issue as the end of the world. Maybe there's no need to freak people out for a minutes-long latency spike.
- jasonm23 10y ago
  FYI https://en.oxforddictionaries.com/definition/err_on_the_side_of_caution https://en.oxforddictionaries.com/definition/err_on_the_side... Just in case you ever use that phrase in critical correspondence. Better to err on the side of caution.
  - leesalminen 10y ago
    Ah, thanks! It didn't feel right but was too lazy to check. Will definitely remember this for next time.
debaserab2 10y ago
It really wouldn't have been that hard to write your opinion in a way that doesn't disparage the character of the person (that you don't really know) that you're responding to. There's no reason to hide behind a new account other than to be an asshole.

empath75 10y ago

Yeah it made me not consider using them for anything other than toy projects.

balls187 10y ago
Why? Everyone has their own copy of the gitlab remote.
- BHSPitMonkey 10y ago
  Not everyone has their own copy of the project's issues/pull requests/settings/webhooks/CI config/etc., which the incident affected (and 6 hours worth were lost). Git repositories themselves do have the happy side effect that at least someone on your team has a full latest clone on their machine, but that's only because the source code is what you're there to use every day. Even if GitLab had a great "checkout your issues/etc. as a git repo" story, not many users would have an up-to-date copy backed up 100% of the time.
  - balls187 10y ago
    Right, there was data loss, and it was not trivial. Does that mean you should only use gitlab for toy projects? I don't think so. I think they'll quickly learn from this.
  - taneq 10y ago
    Maybe a dumb question but couldn't they be included in a hidden folder in the root of your repo or something? That way they'd get replicated automatically every time someone does a pull.
    - Ajedi32 10y ago
      It'd probably have to be on a separate branch, but yeah they technically _could_ do that. Maybe something like this? https://github.com/duplys/git-issues https://github.com/duplys/git-issues
- 10y ago

djsumdog 10y ago

I don't think this affects to many of the larger shops, since they tend to host their own GitLab instances (typically for security, compliance, legal, etc.) My current shop runs GitLab community edition on an internal cluster.

jchook 10y ago

All press is good press.

Kiro 10y ago

> I believe among other developers I don't know. I'm very hesitant trying out GitLab now whereas I was interested before.

synicalx 10y ago

Getting better at ops? Checking backups regularly to make sure they work is not a new thing.

praneshp 10y ago

HN can be funny sometimes. GitHub got a lot of hate about a year ago just for not releasing new features. GitLab cost everyone a day because their backup/ops practices were silly, and everyone loves them more. I've screwed up before, and I sympathize/empathize with their ops folks, but this should make us think about plan B in case something like this happens again.

vollmond 10y ago
There's another reasonable takeaway: GitLab will probably now prepare for this sort of thing better than the average competitor who hasn't been bitten by this.
- ryanplant-au 10y ago
  Borges: "In Alexandria, it has been said that the only persons incapable of a sin are those who have already committed it and repented."
- sytse 10y ago
  We'll certainly learn from this, post-mortem write-up issue is in https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108 https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108
jopsen 10y ago
Nah, no need to think about backup now. Stuff like this only happens once :) haha, To be fair, I suspect gitlab will a adopt a 'never-again' policy towards making sure their backups work.
deckar01 10y ago
> GitLab cost everyone a day GitLab isn't popular because of the stability of its cloud platform. It's popular because you can install your own instance for free practically anywhere with minimal effort. I run GitLab CE on a box in my server closet for projects that involve livelihoods.
- 10y ago

mercwear 10y ago

I am sick of the "They are transparent so it's ok" argument. The fact is this type of thing should not happen anymore.. I get it, stuff fails but it's not acceptable to lose production data at this scale and save face by being transparent. That was ok back in 2003.. Not anymore.

seanp2k2 10y ago
Compare with Github and their 98.935% uptime over the past month: https://status.github.com/graphs/past_month https://status.github.com/graphs/past_month Neither should be exclusively relied upon for business-critical services. Ask Github about their production backups and DR plans some time.
- mst 10y ago
  I still remember the outage where github ran their tests against production - which started by TRUNACTEing everything before loading fixtures. I think "everybody learns the hard way" is just one of those things with operations.
anonymous_green 10y ago
Probably they cut cost by hiring a group of really incompetent set of developers because they are a "startup" and "transparent" and "its ok". Nothing could make state of the company more "transparent" than that. xD

manigandham 10y ago

Transparency is not a substitute for competency. Both are important, but I'll take the latter every time.

rdslw 10y ago

Just to stress one thing, while mentioned in their report, most people don't comprehend: Gitlab was _one_ (last!) final step from complete data loss of everything. One. At that night, there was quite a long moment there had only one copy (and 6 hours old). Every other backup was missing/notworking/deleted. This is scary.

crististm 10y ago
This is the fact that deserves more attention. The operator did a manual backup before work which was the only available one. I find this to be an eye opener. The real problem was dodged by doing the right thing in the proverbial last minute (6 hours).
- s_kilk 10y ago
  In other words, there is an alternate universe in which Gitlab is now categorically dead, all because that employee didn't make that single backup.
garaetjjte 10y ago
Didn't they had two copies? (staging server and LVM snapshot)

anonymous_green 10y ago

Transparency? Gitlab is the company that interviews people and rejects them on basis of salary, they've been doing this for awhile, every few months they call a group of people, waste their time and then deny based on the salary. In my case, I've confronted them that USD 100-120k is the market average but they had this stupid startup argument that doesn't make any sense to me.

hobarrera 10y ago
> USD 100-120k is the market average In the US maybe, but given that they're a remote company with people all over the world, they don't necessarily have to live by US standard (especially when it comes to spending!). 40k USD is a VERY good developer salary in Argentina, and I'd bet that in other countries a lower figure might make the cut too. I can definitely understand why they'd hesitate to pay thrice that amount. I believe it's over 5 times the average salary.
- anonymous_green 10y ago
  Yes, but if you look at the average here its not really outrageous or "5 times the average salary" as you said, here: https://www.indeed.com/salaries/Gitlab-Salaries https://www.indeed.com/salaries/Gitlab-Salaries And its not just me, they've done this in the past as well: https://news.ycombinator.com/item?id=10924957 https://news.ycombinator.com/item?id=10924957 Also when you work for a remote company, they are cutting cost in terms of office and all which should reflect back in the salaries, the whole idea of working remotely was to mutually benefit both the employeer and employee and not just gitlab using their whole startup argument to cop out when they want on that "truly remote" so-called transparent company.
  - hobarrera 10y ago
    Like I said, they're a remote company, and therefore can hire people in cheaper regions, and don't have to adapt to more expensive markets (like your own). I fail to see how this disproves it, it merely proves that you're not a good choice for them (because you live in a, relatively, expensive region).

spinlock 10y ago

For me it underscores one simple rule: you backups are useless if you don't regularly practice restoring from them.

xenithorb 10y ago
Or I don't know, at least checking they exist