7 ms·

HN can be funny sometimes. GitHub got a lot of hate about a year ago just for not releasing new features. GitLab cost everyone a day because their backup/ops pr

HN can be funny sometimes. GitHub got a lot of hate about a year ago just for not releasing new features. GitLab cost everyone a day because their backup/ops practices were silly, and everyone loves them more. I've screwed up before, and I sympathize/empathize with their ops folks, but this should make us think about plan B in case something like this happens again.

vollmond 10y ago
There's another reasonable takeaway: GitLab will probably now prepare for this sort of thing better than the average competitor who hasn't been bitten by this.
- ryanplant-au 10y ago
  Borges: "In Alexandria, it has been said that the only persons incapable of a sin are those who have already committed it and repented."
- sytse 10y ago
  We'll certainly learn from this, post-mortem write-up issue is in https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108 https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108
jopsen 10y ago
Nah, no need to think about backup now. Stuff like this only happens once :) haha, To be fair, I suspect gitlab will a adopt a 'never-again' policy towards making sure their backups work.
deckar01 10y ago
> GitLab cost everyone a day GitLab isn't popular because of the stability of its cloud platform. It's popular because you can install your own instance for free practically anywhere with minimal effort. I run GitLab CE on a box in my server closet for projects that involve livelihoods.
- praneshp 10y ago
  I know (because I use it and also host it at home, and like it). The point of my comment was HN's reaction.
  - Spare_account 10y ago
    Remember that HN is not a single-minded mob, during each incident a particular group was vocal on a given subject but that group may not have been the same individuals who were vocal each time.
    - praneshp 10y ago
      Fair point, thank you.
- thewhitetulip 10y ago
  >It's popular because you can install your own instance for free practically anywhere with minimal effort. If that is the reason why you use Gitlab, then why not try gitea or gogs? gogs is written in Go and provides a docker image or a drop in binary
  - pritambarhate 10y ago
    I have been running Gitlab CE for around three years now and I am very happy with it. The interface it good. The access management works very good. And the best part? Keeping it updated is very easy on a ubuntu box. Just a couple of very easy commands. I am so happy with Gitlab that I didn't even install Gogs even once to give it a try, though I know about it since it's initial days. For me Gitlab CE just works. Unless Gitea shows a 10x feature I am very unlikely to shift away from Gitlab CE.
  - deleted 10y ago
    [deleted]
  - E6300 10y ago
    Gogs is pretty sweet. After hearing about it here I installed it on my old home server with constrained memory because I couldn't meet Gitlab's (IMO) obscene memory requirements. It's perfect for my needs as a hobbyist and I even use it as a secondary Git server from work. It's also way, way faster than a self-hosted Gitlab or any public Git server, so even though my current server has more than enough memory to run Gitlab, I still use Gogs. The only (minor) issue I've had with it was when I tried to push the OpenCV repository to it, just for the hell of it, on a heavily constrained VM (Debian with 256 MiB of RAM). The poor thing just couldn't handle it without crashing until I upped the memory to a couple gigabytes.
juskrey 10y ago
The one who erred (and recovered) once is more valuable than the one who never made mistakes.
- bbcbasic 10y ago
  There is no basis for that assertion.
  - juskrey 10y ago
    This is basic risk management. What matters is error survival, not absence of errors. Moreover, total absence of errors means accumulating dangerous risk. See NN Taleb, for example.
    - bbcbasic 10y ago
      Sounds defeatist to me. Especially when we are not talking about literal moonshots, but backup and restore procedures. The IT equivalent of seatbelts.
      - throwaway2048 10y ago
        I think at this point, recognising that software and software deployments invariably have bugs/issues is just realistic. The only setups that are free of them, are setups where the problems havent been discovered yet.
  - shakna 10y ago
    A well known library like curl, has had CVEs. A not known library that also addresses the same circumstances as curl might have none listed. Which would you prefer to use?
    - bbcbasic 10y ago
      That's a cross purpose argument
    - 10y ago
marklgr 10y ago
That's Internet culture for you: being lovable trumps pretty much anything else.
fictioncircle 10y ago
The "Plan B" is to have a local Gitlab instance and use the mirror feature to mirror it to Gitlab.com and/or vice versa. You should always have 2+ production nodes in case one goes down.
yAnonymous 10y ago
>GitLab cost everyone a day That's just nonsense. In most cases, the local repo should be more than enough to continue work.
- seanp2k2 10y ago
  Yeah, kind of defeats the whole D part of DVCS when you centrally-host everything. GitHub is great for creating an open-source watering hole and becoming a de-facto standard for open source project hosting, but bad for mission-critical stuff if you're relying on something like that to be 100% available, especially when using a technology which is designed to be distributed.
  - Floegipoky 10y ago
    It would have been really cool if GitHub had fully bought into the distributed model and built the wiki/issue tracker/etc such that they were included in the repository, so that they were also decentralized.
- mattcoles 10y ago
  The actual problem was with their DB, not the repo storage so I'm not sure what the actual fallout was, but people lost things like issues/merge requests, so I can see how workflow would be interrupted.
- praneshp 10y ago
  Only for development right? If you were someone that handled issues for the team, or depended on webhooks triggering builds, you were still blocked. I'm not trying to skewer Gitlab, I host an instance at home. I've also nuked 30% of the ports on our openstack cluster and screwed up everyone's day. I admire the transparency. I just wanted to call out HN's reaction, vs way smaller (technical) issues involving GitHub. But someone did point out downthread that HN's not a hivemind, so there is that.
antocv 10y ago
Every corporation ever has screwed up like GitLab if not more. They are just better at hiding it, smoothing it over and lying.

vollmond 10y ago

There's another reasonable takeaway: GitLab will probably now prepare for this sort of thing better than the average competitor who hasn't been bitten by this.

ryanplant-au 10y ago
Borges: "In Alexandria, it has been said that the only persons incapable of a sin are those who have already committed it and repented."
sytse 10y ago
We'll certainly learn from this, post-mortem write-up issue is in https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108 https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108

jopsen 10y ago

Nah, no need to think about backup now. Stuff like this only happens once :) haha, To be fair, I suspect gitlab will a adopt a 'never-again' policy towards making sure their backups work.

deckar01 10y ago

> GitLab cost everyone a day GitLab isn't popular because of the stability of its cloud platform. It's popular because you can install your own instance for free practically anywhere with minimal effort. I run GitLab CE on a box in my server closet for projects that involve livelihoods.

praneshp 10y ago
I know (because I use it and also host it at home, and like it). The point of my comment was HN's reaction.
- Spare_account 10y ago
  Remember that HN is not a single-minded mob, during each incident a particular group was vocal on a given subject but that group may not have been the same individuals who were vocal each time.
  - praneshp 10y ago
    Fair point, thank you.
thewhitetulip 10y ago
>It's popular because you can install your own instance for free practically anywhere with minimal effort. If that is the reason why you use Gitlab, then why not try gitea or gogs? gogs is written in Go and provides a docker image or a drop in binary
- pritambarhate 10y ago
  I have been running Gitlab CE for around three years now and I am very happy with it. The interface it good. The access management works very good. And the best part? Keeping it updated is very easy on a ubuntu box. Just a couple of very easy commands. I am so happy with Gitlab that I didn't even install Gogs even once to give it a try, though I know about it since it's initial days. For me Gitlab CE just works. Unless Gitea shows a 10x feature I am very unlikely to shift away from Gitlab CE.
- deleted 10y ago
  [deleted]
- E6300 10y ago
  Gogs is pretty sweet. After hearing about it here I installed it on my old home server with constrained memory because I couldn't meet Gitlab's (IMO) obscene memory requirements. It's perfect for my needs as a hobbyist and I even use it as a secondary Git server from work. It's also way, way faster than a self-hosted Gitlab or any public Git server, so even though my current server has more than enough memory to run Gitlab, I still use Gogs. The only (minor) issue I've had with it was when I tried to push the OpenCV repository to it, just for the hell of it, on a heavily constrained VM (Debian with 256 MiB of RAM). The poor thing just couldn't handle it without crashing until I upped the memory to a couple gigabytes.

juskrey 10y ago

The one who erred (and recovered) once is more valuable than the one who never made mistakes.

bbcbasic 10y ago
There is no basis for that assertion.
- juskrey 10y ago
  This is basic risk management. What matters is error survival, not absence of errors. Moreover, total absence of errors means accumulating dangerous risk. See NN Taleb, for example.
  - bbcbasic 10y ago
    Sounds defeatist to me. Especially when we are not talking about literal moonshots, but backup and restore procedures. The IT equivalent of seatbelts.
    - throwaway2048 10y ago
      I think at this point, recognising that software and software deployments invariably have bugs/issues is just realistic. The only setups that are free of them, are setups where the problems havent been discovered yet.
- shakna 10y ago
  A well known library like curl, has had CVEs. A not known library that also addresses the same circumstances as curl might have none listed. Which would you prefer to use?
  - bbcbasic 10y ago
    That's a cross purpose argument
  - 10y ago

marklgr 10y ago

That's Internet culture for you: being lovable trumps pretty much anything else.

fictioncircle 10y ago

The "Plan B" is to have a local Gitlab instance and use the mirror feature to mirror it to Gitlab.com and/or vice versa. You should always have 2+ production nodes in case one goes down.

yAnonymous 10y ago

>GitLab cost everyone a day That's just nonsense. In most cases, the local repo should be more than enough to continue work.

seanp2k2 10y ago
Yeah, kind of defeats the whole D part of DVCS when you centrally-host everything. GitHub is great for creating an open-source watering hole and becoming a de-facto standard for open source project hosting, but bad for mission-critical stuff if you're relying on something like that to be 100% available, especially when using a technology which is designed to be distributed.
- Floegipoky 10y ago
  It would have been really cool if GitHub had fully bought into the distributed model and built the wiki/issue tracker/etc such that they were included in the repository, so that they were also decentralized.
mattcoles 10y ago
The actual problem was with their DB, not the repo storage so I'm not sure what the actual fallout was, but people lost things like issues/merge requests, so I can see how workflow would be interrupted.
praneshp 10y ago
Only for development right? If you were someone that handled issues for the team, or depended on webhooks triggering builds, you were still blocked. I'm not trying to skewer Gitlab, I host an instance at home. I've also nuked 30% of the ports on our openstack cluster and screwed up everyone's day. I admire the transparency. I just wanted to call out HN's reaction, vs way smaller (technical) issues involving GitHub. But someone did point out downthread that HN's not a hivemind, so there is that.

antocv 10y ago

Every corporation ever has screwed up like GitLab if not more. They are just better at hiding it, smoothing it over and lying.