4 ms·

From their blog post, > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. We ended up

From their blog post, > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. We ended up restoring a 6 hours old backup. That must be _terrifying_ to realize. I mean, thank goodness they had a 6 hour old back up or they'd be in such an awful spot.

cptskippy 10y ago
I would counter that they're still in an awful spot because this announcement reeks of incompetence and isn't something you want to hear from the guys you're entrusting with keeping your code safe. It would be like Boeing or Airbus announcing all the safety features on their airliners were non functioning.
- mynameisvlad 10y ago
  I don't think those two scenarios are comparable. The safety features are in use on an airplane constantly. Backups are only needed when a disaster happens. It'd be more akin to B/A announcing that some emergency system like the air masks isn't functioning. Still incredibly troubling, especially if there was a cabin depressurization scenario, but not to the level of all safety features being broken.
  - andyana 10y ago
    Not all the safety features are in use all the time. Like, you don't use the inflatable slide unless your evacuating.
    - mynameisvlad 10y ago
      Sure, and I never said all of them are used all the time, but parent comment specifically did say "all the safety features". I wanted to make the distinction clear that safety features in planes can and are used actively during flights, whereas backups are only used when there's a catastrophe.
- testUser69 10y ago
  The fact that they were upfront and honest about it and that they even live streamed themselves fixing the problems makes me want to use gitlab even more. If anything I have even more confidence in them. You didn't hear a peep from Microsoft when the forced windows 10 upgrade bricked thousands of laptops. Perhaps that's why such a huge portion of developers prefer OSX/Linux to Windows? I've run six businesses over the past decade and when windows 10 started rolling out I was in a Houston office that lost nearly a hundred terabytes of client data. We did everything by the books, paid for the business and enterprise editions of Windows, used their servers, used their proprietary software stack, used their support service, and they still fucked us and didn't really seem to care or think they did anything wrong. I have another company that runs on a completely open stack, where pretty much nothing is integrated by a specific vendor. We have hiccups, but we've never had the OS get hijacked and upgraded. I've noticed most start-ups run by devs run on a more open stack and hack their way through problems on the cheap, and the ones run by corporate executives try to keep things as closed as possible, but end up spending millions to solve problems that they could have had some people solve for fun on the internet. I prefer to use the right tool for the right job, but I wish companies like Microsoft would be more open when they cause huge issues that end up causing monetary loss. I make sure all my critical infrastructure is open source these days.
  - FireBeyond 10y ago
    "You didn't hear a peep from Microsoft when the forced windows 10 upgrade bricked thousands of laptops. Perhaps that's why such a huge portion of developers prefer OSX/Linux to Windows? " Weird that you group OSX in there. The only company in mainstream tech more secretive than Microsoft when it comes to problems is... Apple.
  - owenmarshall 10y ago
    Being able to own up to a problem doesn't imply an ability to fix the problem going forward. "Test-recover backups" is ops 101. "Monitor your backup process to be sure your backup store isn't empty" is ops 101. "Script your rollouts so you don't have an ops person doing SSH on boxes" is... ok, that one might be ops 102. This points to a company with almost no understanding of how to operationalize software. There are certain to be far more landmines, possibly even bigger ones. Hiring an ops person to fix these problems is definitely possible - and I sincerely wish GitLab luck getting a competent ops team in place before the next crisis.
- holman 10y ago
  > I would counter that they're still in an awful spot because this announcement reeks of incompetence and isn't something you want to hear from the guys you're entrusting with keeping your code safe. This might be a good time to note that GitHub dropped their database awhile back (twice!). Bad Things always happen; you recover, learn, and hopefully fewer Bad Things happen in the future. Nature of the beast, unfortunately.
- Vinnl 10y ago
  > the guys you're entrusting with keeping your code safe. To add to the other replies: I'm not trusting them with keeping my code safe (everybody has copies on their own computers). I'm trusting them with facilitating my workflow, helping me collaborate, and also to keep my issues etc. safe. So yes, this can have a relatively large impact in terms of not being able to work as efficiently for a day and potentially losing some issues, but nowhere near as disastrous as losing all my code.
  - djsumdog 10y ago
    Exactly, that's the whole point of git. Your entire code base is distributed (or at least everything you push out to a remote). Unlike CVS/SVN, you have that whole copy and collective history of everything that was pushed in that repo. If you want to move it, in most cases you can simply create a new bare repo on another remote and push yours to it. It's probably the easiest system out there to simply pick up your data and go, unlike the walled gardens of social networking, video (YouTube isn't video, it's video + annotations + another ecosystem of tools that's not easy to export), and other services. Even blog software isn't as resilient. You still have to export your Wordpress or Ghost blog when you want to move it. With git, when you work on it, you already have a full copy (with a few exceptions of course, like remote branches people prune without merging or local branches people never push).
- gukov 10y ago
  They are in an awful spot. It can be argued that once a company messes up this bad it will make sure nothing similar happens ever again. However, it can also be argued that if a tech company has all five of its back up procedures fail it's borderline criminally negligent.
- dredmorbius 10y ago
  BOAC Comet.

cptskippy 10y ago

I would counter that they're still in an awful spot because this announcement reeks of incompetence and isn't something you want to hear from the guys you're entrusting with keeping your code safe. It would be like Boeing or Airbus announcing all the safety features on their airliners were non functioning.

mynameisvlad 10y ago
I don't think those two scenarios are comparable. The safety features are in use on an airplane constantly. Backups are only needed when a disaster happens. It'd be more akin to B/A announcing that some emergency system like the air masks isn't functioning. Still incredibly troubling, especially if there was a cabin depressurization scenario, but not to the level of all safety features being broken.
- andyana 10y ago
  Not all the safety features are in use all the time. Like, you don't use the inflatable slide unless your evacuating.
  - mynameisvlad 10y ago
    Sure, and I never said all of them are used all the time, but parent comment specifically did say "all the safety features". I wanted to make the distinction clear that safety features in planes can and are used actively during flights, whereas backups are only used when there's a catastrophe.
testUser69 10y ago
The fact that they were upfront and honest about it and that they even live streamed themselves fixing the problems makes me want to use gitlab even more. If anything I have even more confidence in them. You didn't hear a peep from Microsoft when the forced windows 10 upgrade bricked thousands of laptops. Perhaps that's why such a huge portion of developers prefer OSX/Linux to Windows? I've run six businesses over the past decade and when windows 10 started rolling out I was in a Houston office that lost nearly a hundred terabytes of client data. We did everything by the books, paid for the business and enterprise editions of Windows, used their servers, used their proprietary software stack, used their support service, and they still fucked us and didn't really seem to care or think they did anything wrong. I have another company that runs on a completely open stack, where pretty much nothing is integrated by a specific vendor. We have hiccups, but we've never had the OS get hijacked and upgraded. I've noticed most start-ups run by devs run on a more open stack and hack their way through problems on the cheap, and the ones run by corporate executives try to keep things as closed as possible, but end up spending millions to solve problems that they could have had some people solve for fun on the internet. I prefer to use the right tool for the right job, but I wish companies like Microsoft would be more open when they cause huge issues that end up causing monetary loss. I make sure all my critical infrastructure is open source these days.
- FireBeyond 10y ago
  "You didn't hear a peep from Microsoft when the forced windows 10 upgrade bricked thousands of laptops. Perhaps that's why such a huge portion of developers prefer OSX/Linux to Windows? " Weird that you group OSX in there. The only company in mainstream tech more secretive than Microsoft when it comes to problems is... Apple.
- owenmarshall 10y ago
  Being able to own up to a problem doesn't imply an ability to fix the problem going forward. "Test-recover backups" is ops 101. "Monitor your backup process to be sure your backup store isn't empty" is ops 101. "Script your rollouts so you don't have an ops person doing SSH on boxes" is... ok, that one might be ops 102. This points to a company with almost no understanding of how to operationalize software. There are certain to be far more landmines, possibly even bigger ones. Hiring an ops person to fix these problems is definitely possible - and I sincerely wish GitLab luck getting a competent ops team in place before the next crisis.
holman 10y ago
> I would counter that they're still in an awful spot because this announcement reeks of incompetence and isn't something you want to hear from the guys you're entrusting with keeping your code safe. This might be a good time to note that GitHub dropped their database awhile back (twice!). Bad Things always happen; you recover, learn, and hopefully fewer Bad Things happen in the future. Nature of the beast, unfortunately.
Vinnl 10y ago
> the guys you're entrusting with keeping your code safe. To add to the other replies: I'm not trusting them with keeping my code safe (everybody has copies on their own computers). I'm trusting them with facilitating my workflow, helping me collaborate, and also to keep my issues etc. safe. So yes, this can have a relatively large impact in terms of not being able to work as efficiently for a day and potentially losing some issues, but nowhere near as disastrous as losing all my code.
- djsumdog 10y ago
  Exactly, that's the whole point of git. Your entire code base is distributed (or at least everything you push out to a remote). Unlike CVS/SVN, you have that whole copy and collective history of everything that was pushed in that repo. If you want to move it, in most cases you can simply create a new bare repo on another remote and push yours to it. It's probably the easiest system out there to simply pick up your data and go, unlike the walled gardens of social networking, video (YouTube isn't video, it's video + annotations + another ecosystem of tools that's not easy to export), and other services. Even blog software isn't as resilient. You still have to export your Wordpress or Ghost blog when you want to move it. With git, when you work on it, you already have a full copy (with a few exceptions of course, like remote branches people prune without merging or local branches people never push).
gukov 10y ago
They are in an awful spot. It can be argued that once a company messes up this bad it will make sure nothing similar happens ever again. However, it can also be argued that if a tech company has all five of its back up procedures fail it's borderline criminally negligent.
dredmorbius 10y ago
BOAC Comet.