21 ms·

Data Loss at GitLab

476 points by umairshahid 10y ago

wheelerwj 10y ago
i wish everything was discussed/handled as publicly and transparently as this whole scenario. i really hope this becomes a thing.
- cornedor 10y ago
  I totally agree, the live stream [1] is amazing, discussing steps in the open like that and if there is some time left answer questions from the YouTube chat. [1] https://www.youtube.com/c/Gitlab/live https://www.youtube.com/c/Gitlab/live
  - jobvandervoort 10y ago
    Glad to hear this was appreciated. It was an experiment -one we only hope to repeat in other scenarios.
    - throwaway7767 10y ago
      As a sysadmin, I'd find it incredibly distracting to be on a livestream while trying to fix a critical issue. For your employees sake I hope you don't do this again. Have a single point of contact that provides information about the recovery process. Being transparent and providing technical info is good, but that task should not be handled directly by the admins at the same time they are focusing on the drop-everything-shit-is-broke emergency.
  - mandlar 10y ago
    Is there an archive of the stream?
    - jobvandervoort 10y ago
      Part of the stream is here: https://www.youtube.com/watch?v=nc0hPGerSd4 https://www.youtube.com/watch?v=nc0hPGerSd4 Not sure if we'll be able to get the full 8+ hours up.
      - Vinnl 10y ago
        ...and keep it up (sorry, couldn't resist :P)
chaosfox 10y ago
on related news, gitlab.com seems back up now.
jwilk 10y ago
https://news.ycombinator.com/item?id=13537052 https://news.ycombinator.com/item?id=13537052
- willemmali 10y ago
  ^ Related HN discussion on the live report @ Google Docs
overcast 10y ago
So the DeathStar is operational once again?
- cabargas 10y ago
  it is a fully operational battle station once again :)
neals 10y ago
I don't get to work on databases this size and today has been an incredible lesson and a journey. I've been reading all the comments and blogs, watching the stream and Googling what I didn't know or understand. I feel like the next step for me is scaling my business so that we have an actual usage for my newly found interests :)
- sulam 10y ago
  Then you end up having deep expertise in a topic that's only important for larger companies. Sometimes this works out well, sometimes it leaves you a little stuck. :)
  - skywhopper 10y ago
    Oh, there's plenty of demand for skilled DBAs with deep expertise with handling large volumes of data, even at small-to-medium-sized companies. GitLab being one of those companies! In fact, demand for this sort of thing is likely to increase. If it interests you, dig in. You'll never stop learning.
  - stusmall 10y ago
    Meh. Just learn, learn, learn. If it isn't completely applicable to your life that's okay. Sometimes there are pearls of wisdom in best practices in field completely unrelated to your own. Learning something new is always a good thing.
    - haggy 10y ago
      THIS ^^^ So much this. I can't tell you how many times I picked up a book or paper thinking "There's no way i'm going to get anything new out of this". As I start reading, I start to find little tidbits of information that make me think in ways I didn't before.
  - hyperpape 10y ago
    I'm not sure if it's really company size, though there's probably a correlation. My company is only marginally bigger than GitLab in terms of employees (just recently 200+), and we have several dozen databases bigger than this.
hyperpape 10y ago
I'm surprised by the statement that 4 gigs of replication lag is normal. However, I don't manage backups for anything larger than personal pet projects, so I don't have a sense of scale.
- aexaey 10y ago
  > don't have a sense of scale From [1], complete db is ~300GB and from some iffy pixel measurement of the graph at the very bottom of that page, copying speed between otherwise idle db hosts was about 22.8 GB/hour (in-production replication is probably slower than that). From that, 4GB of replication lag would represent 1.3% of db by size, or 10+ minutes of lag (as measured by time required to catch up under ideal circumstances). [1] https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/ https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-...
  - stock_toaster 10y ago
    Thanks for the context, in my personal experience at jobs dealing with large databases, we always used "minutes behind" to determine how well our replication was keeping up. This was the first I had heard of someone using data size for that same metric.
    - johannes1234321 10y ago
      Both have different uses. Seconds behind is useful to see how critical this is to the user (they see outdated data), data behind tells you about load and let's you guess how much time is required for catching up
  - hyperpape 10y ago
    I didn't think to eyeball the graph to guesstimate how long the 4GB translated to, so thanks. However, scale was the wrong word for what I was wondering about. My question should've been whether 1% of your total DB/10 minutes of replication lag seems reasonable/nothing to worry about, like the article suggested.
marricks 10y ago
From their blog post, > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. We ended up restoring a 6 hours old backup. That must be _terrifying_ to realize. I mean, thank goodness they had a 6 hour old back up or they'd be in such an awful spot.
- cptskippy 10y ago
  I would counter that they're still in an awful spot because this announcement reeks of incompetence and isn't something you want to hear from the guys you're entrusting with keeping your code safe. It would be like Boeing or Airbus announcing all the safety features on their airliners were non functioning.
  - mynameisvlad 10y ago
    I don't think those two scenarios are comparable. The safety features are in use on an airplane constantly. Backups are only needed when a disaster happens. It'd be more akin to B/A announcing that some emergency system like the air masks isn't functioning. Still incredibly troubling, especially if there was a cabin depressurization scenario, but not to the level of all safety features being broken.
    - andyana 10y ago
      Not all the safety features are in use all the time. Like, you don't use the inflatable slide unless your evacuating.
      - mynameisvlad 10y ago
        Sure, and I never said all of them are used all the time, but parent comment specifically did say "all the safety features". I wanted to make the distinction clear that safety features in planes can and are used actively during flights, whereas backups are only used when there's a catastrophe.
  - 10y ago
activatedgeek 10y ago
While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.
- FlorianRappl 10y ago
  Couldn't agree more. Also transparency is key here. Everyone makes mistakes, but being open about it does not only let customers (and employees!) know what's going on, but also gives other professionals the ability to learn from it.
  - neurotech1 10y ago
    There are two types of web application providers. Those who have already had database mishaps causing data loss, and those that WILL. No matter how good, or our how large and 'professional' the company is, mistakes can happen. Being open about it inspires confidence that they will improve in the future, instead of being quiet and having repeat incidents.
- provost 10y ago
  As someone who hasn't had the opportunity to use Gitlab, could you expand on why you "think Gitlab is the best thing happened to OSS"?
  - mi100hael 10y ago
    It provides a lot of the same features as GitHub, only you can self-host for free, contribute to development, and all the other niceties of a free project. A lot of people (myself included, and OP I suspect) believe it's also a particularly good project because GitHub has too much market share and influence in the free software ecosystem for a for-profit company.
    - Vinnl 10y ago
      Note that GitLab is a for-profit company as well. It's just that it aligns better with the open source values.
aamederen 10y ago
With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.
- dentemple 10y ago
  Any publicity is good publicity, right? :D
- testUser69 10y ago
  I think the "business people" are people who hide their technological inadequacy behind supposed business expertise. A businesses success is not determined by whether or not they use tools that adhere to business micro-cultural "values". If "too much transparency" is a turn off for you, you're probably just an authoritarian trying to scheme and scam your way into profit, and you probably lack the confidence required to put whatever skill you think you have on display.
  - GavinMcG 10y ago
    I suspect the comment you're replying to meant that the business people might feel negatively about trusting the company that lost data, regardless of transparency.
  - tyre 10y ago
    This demonstrates a remarkable lack of empathy. Try understanding why people on the business side do what they do—it's almost never out of malice or desired authoritarianism. Working with people is hard, specifically because you have to get outside of your head and think about how they see the world.
  - leesalminen 10y ago
    At my company we always heir on the side of transparency and liberally post to our status. subdomain whenever an issue is identified. Recently, our Redis cluster failed and both master & slave host machines rebooted. This caused a latency spike in our app, from roughly a 70ms to 400ms response time for less than 10 minutes. We posted to status within 60 seconds and posted 3 updates within those 10 minutes. The next day, a new customer (who hadn't gone live with the app yet) cancelled their subscription because the app was "not reliable". I guess my point is that there is a balance to strike. Our customers are not tech-savvy in any way and treat any small issue as the end of the world. Maybe there's no need to freak people out for a minutes-long latency spike.
intsunny 10y ago
The gitlab situation and Uber's article speak to the level of immaturity of PGSQL's native replication feature, and more importantly: how not widely google-able nor documented/adopted the replication strategies are.
- djsumdog 10y ago
  I didn't think replication was one of the issues Uber noted, but then I looked back at the blog posts: https://eng.uber.com/mysql-migration/ https://eng.uber.com/mysql-migration/ ..and yep, it's listed. In my own experience, I'd still take PostgreSQL over MySQL. MySQL doesn't allow for DDL modifications within a transaction, which makes database migration with tools like Flyway a little less resilient. On the other hand, you can use one connection for multiple databases with MySQL, MSSQL and others, which you can't with Postgres. I mean really, they all have trade-offs. It really just depends on your specific use case.
- tshannon 10y ago
  I believe gitlab used slony, not the native replication. I'm not well versed in postgres, but that's what I gleaned from reading their event log.
  - jessaustin 10y ago
    The decision not to use native replication might be attributed to its purported immaturity?
  - detaro 10y ago
    The google doc says at the bottom that slony only was used for a migration once, otherwise they use the replication features built into postgres.
  - YorickPeterse 10y ago
    As mentioned below we only used Slony to upgrade from 9.2.something to 9.6.1. For regular replication we use PostgreSQL's streaming replication.
- 10y ago
pdog 10y ago
It feels like PostgreSQL is at the center of every terrible story about data loss[1] or poor performance[2]. I think companies prefer other databases like MySQL because they "just work." [1]: https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/ https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-... [2]: https://eng.uber.com/mysql-migration/ https://eng.uber.com/mysql-migration/
- sgarman 10y ago
  Seems like confirmation bias. I always think of nosql databases when I'm thinking about terrible data loss stories, I especially remember the couchDB one. Postgres has been nothing but amazing for my uses.
  - patmcguire 10y ago
    If you delete your data directory, you're going to have a bad time no matter what your stack is.
    - VintageCool 10y ago
      Yes, it's probably an outage event, but then you flip over to your slave MySQL instance and keep going on your merry way.
- AlisdairO 10y ago
  You gave one example of each. That's a pretty painful definition of the word 'every'. On performance: Every database will have some areas where it performs better than others - Uber just happened to hit a case that MySQL is well optimised for. On data loss: MySQL's binlog replication is hardly a 'just works' solution. It's definitely got a lot better over time (particularly in 5.7), but it's not like a replica falling behind is some incredibly unusual event. Go back a fair while and it was really broken - much more so than Postgres' solution has ever been. It's fair to say it's more mature right now, and one reason to prefer MySQL for replication is that there's certainly more help out there for it.
- 10y ago
jbverschoor 10y ago
Only pull requests and issues are gone. And even if the repo's were gone, wasn't git supposed to be a distributed vcs anyway? ;)
bfrog 10y ago
Testing backups or at least monitoring them for correctness is a huge deal and it's problem I myself fudged up on one occasion, right around the holiday season which was terrible. I've since setup wal-e with daily base backups deleting things older than a week along with the nightly pg_dump's along with a hot stanby. Maybe thats overkill, but after having lost data once. Never again! The nice part about doing wal archiving like barman or wal-e do is you can do more than just backup/restore. You can do it with some time target in mind as well. Someone somehow do a massive update or delete or insert millions+ in garbage? No worries, stop, destroy, restore to a previous point in time, continue onwards. A bug in postgres or the kernel or the filesystem or any other multi-million line codebase in that stack screw up? Most likely the WAL segments are good up to a point still. Hot standby gets you potentially sub-minute failovers if you automate them, or short enough to be ok even with manual failovers. WAL archiving gets you another whole level of safety net that is hard to beat.
- sgs1370 10y ago
  One thing I could never figure out, and it is probably because I didn't study the docs enough, is when I could delete the archive files. What is the best reference on this. OR, maybe I was trying to reinvent the wheel, I didn't really look at 3rd party libs - only postgresql options (plus my glue on top) - is the only right way to use the archive (file shipping) solution to combine it with a 3rd party?
thisisadumb 10y ago
it's a good thing our data is always backed up by your friends at the nsa
- PhantomGremlin 10y ago
  They'll back it up for you, but they won't give you access to your own data: https://memegenerator.net/instance/68192872 https://memegenerator.net/instance/68192872
synicalx 10y ago
Honestly, I'm completely flabbergasted by this. Five backups, and NONE worked properly? Who made this? The S3 bucket was EMPTY? Has no one ever tested any of these backups? It's not just the impact, which is fairly sizeable in it's own right, but it's the HUGE oversight on their part and the fact they tried to pin part of this on PostreSQL? Credit where it's due; their report/transparency were good if a little unprofessional, and something I'd like to see more of from other companies. Putting on my BOFH hat, this is what happens when you let Devs do operational stuff.
- developer2 10y ago
  I agree. Their catastrophe is not the kind of thing that represents mild hiccups in operations that will be quickly resolved by the same employees who allowed this scenario to occur the first time around. This isn't the result of a single oversight. It screams of a systemic problem with the way the business operates - period. Maybe every project is rushed out with the deadline being the only metric that counts, quality be damned. Or maybe they're missing talent in areas that matter - like experienced postgres DBAs managing the database rather than having the developers and/or someone who has only ever used MySQL trying to wrangle it. What they went through is what you'd expect from a first-time pet project, not a professional business out to make money. I suspect they prioritized trying to scale with enough features to catch up with the competition above all else, to the point where sustainability is an afterthought. When desperately trying to gain market share is more important than the quality of the product, this is what you can expect.
  - sytse 10y ago
    I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/ https://about.gitlab.com/jobs/specialist/database/
    - BrentOzar 10y ago
      The word "backup" doesn't appear anywhere in the job description. Might wanna revise that today.
lsh123 10y ago
Many years ago in 96 or 97, my colleague tried to upgrade Oracle DB by running "sudo /path/install.sh" from the root folder. Little did he know that the script did "rm -rf *" on one of the first lines :) The day was saved by the fact that Oracle stored data on a block device directly. There were no data loss and we just had to restore the machine itself. Since that day, I never run any scripts in /, /etc/, ...
- dredmorbius 10y ago
  I've always despised the practice of vendor-provided installation scripts for much this reason. Use local installers. Or tarballs.
aidenn0 10y ago
We have a saying at my work: "If it's not tested it doesn't work" while this was originally talking about software, it's amazing how many other places it applies. Do you require code reviews before commit? Periodically sample a few random commits and see if one was done for each. Heck, if you don't test your system for running automated tests, it may be that you aren't even testing what you think you're testing.
pzh 10y ago
Does nobody else find the report cringeworthy? Apparently, there are some junior engineers fumbling around and committing serious errors, but where are the senior ones and the process/failsafes to prevent all this?
- jjirsa 10y ago
  Yes. It makes me angry. It's not that backups failed, it's more like broad incompetence at all the wrong places.
- davzie 10y ago
  From what I know of the dev who caused this, he is certainly no junior engineer. One of the smartest devs I know.
esseti 10y ago
Aside from the incident, this is a great opportunity to learn something new (at least for me). Said so, does anyone of you know how they plot all the charts of postgres and co? such as this: http://monitor.gitlab.net/dashboard/db/postgres-queries http://monitor.gitlab.net/dashboard/db/postgres-queries I know that the chart is made in grafana, but: how do they collect the data?
- sytse 10y ago
  We use Prometheus prometheus.io We have a team of Prometheus engineers and have a vacancy for more.
carmate383 10y ago
How can a "company" with more than 150 employees, (That is essentially, a data storage company) let this happen? GitLab is surely losing subscribers faster than land in Crimea after this...
- cs02rm0 10y ago
  Incident upon incident followed by a mistake under pressure late at night. --it happens. I suspect the the guy responsible for the final straw is feeling pretty bad. I know I've come close to doing similar things on production environments I really didn't want to be touching while they were falling apart. But they've been honest about it. If they learn from it and six hours of database data is the worst data loss they ever experience I think it'll be a credit to them that they've been promptly transparent.
deleted 10y ago
[deleted]
da4c30ff 10y ago
Git really should have issue and merge/pull request data shipped with the repository. Does anyone know if this has been planned or not?
- Ajedi32 10y ago
  https://gitlab.com/gitlab-org/gitlab-ce/issues/14924 https://gitlab.com/gitlab-org/gitlab-ce/issues/14924
giancarlostoro 10y ago
On the other hand, even if there was some data loss, shouldn't most people have their entire repositories on their drives (at least the ones they're actively working on) so in theory much can be recovered by end-users who are active. The only true worry is inactive users. Not sure if this was discussed much.
- yoavm 10y ago
  AFAIK the problem was with the database and not the with the repos. So yeah, I have all my files on my machines, but I don't have any copy of the issues, merge requests, wiki pages etc'.
slowhand09 10y ago
TL;DR I'm sure its been said already. Doing backups is great. But testing RECOVERY is critical and should be top priority. A data company is scary. When you must back up your data because your data company can't be trusted to backup your data...

476 points by umairshahid 10y ago

i wish everything was discussed/handled as publicly and transparently as this whole scenario. i really hope this becomes a thing.

cornedor 10y ago
I totally agree, the live stream [1] is amazing, discussing steps in the open like that and if there is some time left answer questions from the YouTube chat. [1] https://www.youtube.com/c/Gitlab/live https://www.youtube.com/c/Gitlab/live
- jobvandervoort 10y ago
  Glad to hear this was appreciated. It was an experiment -one we only hope to repeat in other scenarios.
  - throwaway7767 10y ago
    As a sysadmin, I'd find it incredibly distracting to be on a livestream while trying to fix a critical issue. For your employees sake I hope you don't do this again. Have a single point of contact that provides information about the recovery process. Being transparent and providing technical info is good, but that task should not be handled directly by the admins at the same time they are focusing on the drop-everything-shit-is-broke emergency.
- mandlar 10y ago
  Is there an archive of the stream?
  - jobvandervoort 10y ago
    Part of the stream is here: https://www.youtube.com/watch?v=nc0hPGerSd4 https://www.youtube.com/watch?v=nc0hPGerSd4 Not sure if we'll be able to get the full 8+ hours up.
    - Vinnl 10y ago
      ...and keep it up (sorry, couldn't resist :P)

chaosfox 10y ago

on related news, gitlab.com seems back up now.

jwilk 10y ago

https://news.ycombinator.com/item?id=13537052 https://news.ycombinator.com/item?id=13537052

willemmali 10y ago
^ Related HN discussion on the live report @ Google Docs

overcast 10y ago

So the DeathStar is operational once again?

cabargas 10y ago
it is a fully operational battle station once again :)

neals 10y ago

I don't get to work on databases this size and today has been an incredible lesson and a journey. I've been reading all the comments and blogs, watching the stream and Googling what I didn't know or understand. I feel like the next step for me is scaling my business so that we have an actual usage for my newly found interests :)

sulam 10y ago
Then you end up having deep expertise in a topic that's only important for larger companies. Sometimes this works out well, sometimes it leaves you a little stuck. :)
- skywhopper 10y ago
  Oh, there's plenty of demand for skilled DBAs with deep expertise with handling large volumes of data, even at small-to-medium-sized companies. GitLab being one of those companies! In fact, demand for this sort of thing is likely to increase. If it interests you, dig in. You'll never stop learning.
- stusmall 10y ago
  Meh. Just learn, learn, learn. If it isn't completely applicable to your life that's okay. Sometimes there are pearls of wisdom in best practices in field completely unrelated to your own. Learning something new is always a good thing.
  - haggy 10y ago
    THIS ^^^ So much this. I can't tell you how many times I picked up a book or paper thinking "There's no way i'm going to get anything new out of this". As I start reading, I start to find little tidbits of information that make me think in ways I didn't before.
- hyperpape 10y ago
  I'm not sure if it's really company size, though there's probably a correlation. My company is only marginally bigger than GitLab in terms of employees (just recently 200+), and we have several dozen databases bigger than this.

hyperpape 10y ago

I'm surprised by the statement that 4 gigs of replication lag is normal. However, I don't manage backups for anything larger than personal pet projects, so I don't have a sense of scale.

aexaey 10y ago
> don't have a sense of scale From [1], complete db is ~300GB and from some iffy pixel measurement of the graph at the very bottom of that page, copying speed between otherwise idle db hosts was about 22.8 GB/hour (in-production replication is probably slower than that). From that, 4GB of replication lag would represent 1.3% of db by size, or 10+ minutes of lag (as measured by time required to catch up under ideal circumstances). [1] https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/ https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-...
- stock_toaster 10y ago
  Thanks for the context, in my personal experience at jobs dealing with large databases, we always used "minutes behind" to determine how well our replication was keeping up. This was the first I had heard of someone using data size for that same metric.
  - johannes1234321 10y ago
    Both have different uses. Seconds behind is useful to see how critical this is to the user (they see outdated data), data behind tells you about load and let's you guess how much time is required for catching up
- hyperpape 10y ago
  I didn't think to eyeball the graph to guesstimate how long the 4GB translated to, so thanks. However, scale was the wrong word for what I was wondering about. My question should've been whether 1% of your total DB/10 minutes of replication lag seems reasonable/nothing to worry about, like the article suggested.

marricks 10y ago

From their blog post, > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. We ended up restoring a 6 hours old backup. That must be _terrifying_ to realize. I mean, thank goodness they had a 6 hour old back up or they'd be in such an awful spot.

cptskippy 10y ago
I would counter that they're still in an awful spot because this announcement reeks of incompetence and isn't something you want to hear from the guys you're entrusting with keeping your code safe. It would be like Boeing or Airbus announcing all the safety features on their airliners were non functioning.
- mynameisvlad 10y ago
  I don't think those two scenarios are comparable. The safety features are in use on an airplane constantly. Backups are only needed when a disaster happens. It'd be more akin to B/A announcing that some emergency system like the air masks isn't functioning. Still incredibly troubling, especially if there was a cabin depressurization scenario, but not to the level of all safety features being broken.
  - andyana 10y ago
    Not all the safety features are in use all the time. Like, you don't use the inflatable slide unless your evacuating.
    - mynameisvlad 10y ago
      Sure, and I never said all of them are used all the time, but parent comment specifically did say "all the safety features". I wanted to make the distinction clear that safety features in planes can and are used actively during flights, whereas backups are only used when there's a catastrophe.
- 10y ago

activatedgeek 10y ago

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

FlorianRappl 10y ago
Couldn't agree more. Also transparency is key here. Everyone makes mistakes, but being open about it does not only let customers (and employees!) know what's going on, but also gives other professionals the ability to learn from it.
- neurotech1 10y ago
  There are two types of web application providers. Those who have already had database mishaps causing data loss, and those that WILL. No matter how good, or our how large and 'professional' the company is, mistakes can happen. Being open about it inspires confidence that they will improve in the future, instead of being quiet and having repeat incidents.
provost 10y ago
As someone who hasn't had the opportunity to use Gitlab, could you expand on why you "think Gitlab is the best thing happened to OSS"?
- mi100hael 10y ago
  It provides a lot of the same features as GitHub, only you can self-host for free, contribute to development, and all the other niceties of a free project. A lot of people (myself included, and OP I suspect) believe it's also a particularly good project because GitHub has too much market share and influence in the free software ecosystem for a for-profit company.
  - Vinnl 10y ago
    Note that GitLab is a for-profit company as well. It's just that it aligns better with the open source values.

aamederen 10y ago

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

dentemple 10y ago
Any publicity is good publicity, right? :D
testUser69 10y ago
I think the "business people" are people who hide their technological inadequacy behind supposed business expertise. A businesses success is not determined by whether or not they use tools that adhere to business micro-cultural "values". If "too much transparency" is a turn off for you, you're probably just an authoritarian trying to scheme and scam your way into profit, and you probably lack the confidence required to put whatever skill you think you have on display.
- GavinMcG 10y ago
  I suspect the comment you're replying to meant that the business people might feel negatively about trusting the company that lost data, regardless of transparency.
- tyre 10y ago
  This demonstrates a remarkable lack of empathy. Try understanding why people on the business side do what they do—it's almost never out of malice or desired authoritarianism. Working with people is hard, specifically because you have to get outside of your head and think about how they see the world.
- leesalminen 10y ago
  At my company we always heir on the side of transparency and liberally post to our status. subdomain whenever an issue is identified. Recently, our Redis cluster failed and both master & slave host machines rebooted. This caused a latency spike in our app, from roughly a 70ms to 400ms response time for less than 10 minutes. We posted to status within 60 seconds and posted 3 updates within those 10 minutes. The next day, a new customer (who hadn't gone live with the app yet) cancelled their subscription because the app was "not reliable". I guess my point is that there is a balance to strike. Our customers are not tech-savvy in any way and treat any small issue as the end of the world. Maybe there's no need to freak people out for a minutes-long latency spike.

intsunny 10y ago

The gitlab situation and Uber's article speak to the level of immaturity of PGSQL's native replication feature, and more importantly: how not widely google-able nor documented/adopted the replication strategies are.

djsumdog 10y ago
I didn't think replication was one of the issues Uber noted, but then I looked back at the blog posts: https://eng.uber.com/mysql-migration/ https://eng.uber.com/mysql-migration/ ..and yep, it's listed. In my own experience, I'd still take PostgreSQL over MySQL. MySQL doesn't allow for DDL modifications within a transaction, which makes database migration with tools like Flyway a little less resilient. On the other hand, you can use one connection for multiple databases with MySQL, MSSQL and others, which you can't with Postgres. I mean really, they all have trade-offs. It really just depends on your specific use case.
tshannon 10y ago
I believe gitlab used slony, not the native replication. I'm not well versed in postgres, but that's what I gleaned from reading their event log.
- jessaustin 10y ago
  The decision not to use native replication might be attributed to its purported immaturity?
- detaro 10y ago
  The google doc says at the bottom that slony only was used for a migration once, otherwise they use the replication features built into postgres.
- YorickPeterse 10y ago
  As mentioned below we only used Slony to upgrade from 9.2.something to 9.6.1. For regular replication we use PostgreSQL's streaming replication.
10y ago

pdog 10y ago

It feels like PostgreSQL is at the center of every terrible story about data loss[1] or poor performance[2]. I think companies prefer other databases like MySQL because they "just work." [1]: https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/ https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-... [2]: https://eng.uber.com/mysql-migration/ https://eng.uber.com/mysql-migration/

sgarman 10y ago
Seems like confirmation bias. I always think of nosql databases when I'm thinking about terrible data loss stories, I especially remember the couchDB one. Postgres has been nothing but amazing for my uses.
- patmcguire 10y ago
  If you delete your data directory, you're going to have a bad time no matter what your stack is.
  - VintageCool 10y ago
    Yes, it's probably an outage event, but then you flip over to your slave MySQL instance and keep going on your merry way.
AlisdairO 10y ago
You gave one example of each. That's a pretty painful definition of the word 'every'. On performance: Every database will have some areas where it performs better than others - Uber just happened to hit a case that MySQL is well optimised for. On data loss: MySQL's binlog replication is hardly a 'just works' solution. It's definitely got a lot better over time (particularly in 5.7), but it's not like a replica falling behind is some incredibly unusual event. Go back a fair while and it was really broken - much more so than Postgres' solution has ever been. It's fair to say it's more mature right now, and one reason to prefer MySQL for replication is that there's certainly more help out there for it.
10y ago

jbverschoor 10y ago

Only pull requests and issues are gone. And even if the repo's were gone, wasn't git supposed to be a distributed vcs anyway? ;)

bfrog 10y ago

Testing backups or at least monitoring them for correctness is a huge deal and it's problem I myself fudged up on one occasion, right around the holiday season which was terrible. I've since setup wal-e with daily base backups deleting things older than a week along with the nightly pg_dump's along with a hot stanby. Maybe thats overkill, but after having lost data once. Never again! The nice part about doing wal archiving like barman or wal-e do is you can do more than just backup/restore. You can do it with some time target in mind as well. Someone somehow do a massive update or delete or insert millions+ in garbage? No worries, stop, destroy, restore to a previous point in time, continue onwards. A bug in postgres or the kernel or the filesystem or any other multi-million line codebase in that stack screw up? Most likely the WAL segments are good up to a point still. Hot standby gets you potentially sub-minute failovers if you automate them, or short enough to be ok even with manual failovers. WAL archiving gets you another whole level of safety net that is hard to beat.

sgs1370 10y ago
One thing I could never figure out, and it is probably because I didn't study the docs enough, is when I could delete the archive files. What is the best reference on this. OR, maybe I was trying to reinvent the wheel, I didn't really look at 3rd party libs - only postgresql options (plus my glue on top) - is the only right way to use the archive (file shipping) solution to combine it with a 3rd party?

thisisadumb 10y ago

it's a good thing our data is always backed up by your friends at the nsa

PhantomGremlin 10y ago
They'll back it up for you, but they won't give you access to your own data: https://memegenerator.net/instance/68192872 https://memegenerator.net/instance/68192872

synicalx 10y ago

Honestly, I'm completely flabbergasted by this. Five backups, and NONE worked properly? Who made this? The S3 bucket was EMPTY? Has no one ever tested any of these backups? It's not just the impact, which is fairly sizeable in it's own right, but it's the HUGE oversight on their part and the fact they tried to pin part of this on PostreSQL? Credit where it's due; their report/transparency were good if a little unprofessional, and something I'd like to see more of from other companies. Putting on my BOFH hat, this is what happens when you let Devs do operational stuff.

developer2 10y ago
I agree. Their catastrophe is not the kind of thing that represents mild hiccups in operations that will be quickly resolved by the same employees who allowed this scenario to occur the first time around. This isn't the result of a single oversight. It screams of a systemic problem with the way the business operates - period. Maybe every project is rushed out with the deadline being the only metric that counts, quality be damned. Or maybe they're missing talent in areas that matter - like experienced postgres DBAs managing the database rather than having the developers and/or someone who has only ever used MySQL trying to wrangle it. What they went through is what you'd expect from a first-time pet project, not a professional business out to make money. I suspect they prioritized trying to scale with enough features to catch up with the competition above all else, to the point where sustainability is an afterthought. When desperately trying to gain market share is more important than the quality of the product, this is what you can expect.
- sytse 10y ago
  I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/ https://about.gitlab.com/jobs/specialist/database/
  - BrentOzar 10y ago
    The word "backup" doesn't appear anywhere in the job description. Might wanna revise that today.

lsh123 10y ago

Many years ago in 96 or 97, my colleague tried to upgrade Oracle DB by running "sudo /path/install.sh" from the root folder. Little did he know that the script did "rm -rf *" on one of the first lines :) The day was saved by the fact that Oracle stored data on a block device directly. There were no data loss and we just had to restore the machine itself. Since that day, I never run any scripts in /, /etc/, ...

dredmorbius 10y ago
I've always despised the practice of vendor-provided installation scripts for much this reason. Use local installers. Or tarballs.

aidenn0 10y ago

We have a saying at my work: "If it's not tested it doesn't work" while this was originally talking about software, it's amazing how many other places it applies. Do you require code reviews before commit? Periodically sample a few random commits and see if one was done for each. Heck, if you don't test your system for running automated tests, it may be that you aren't even testing what you think you're testing.

pzh 10y ago

Does nobody else find the report cringeworthy? Apparently, there are some junior engineers fumbling around and committing serious errors, but where are the senior ones and the process/failsafes to prevent all this?

jjirsa 10y ago
Yes. It makes me angry. It's not that backups failed, it's more like broad incompetence at all the wrong places.
davzie 10y ago
From what I know of the dev who caused this, he is certainly no junior engineer. One of the smartest devs I know.

esseti 10y ago

Aside from the incident, this is a great opportunity to learn something new (at least for me). Said so, does anyone of you know how they plot all the charts of postgres and co? such as this: http://monitor.gitlab.net/dashboard/db/postgres-queries http://monitor.gitlab.net/dashboard/db/postgres-queries I know that the chart is made in grafana, but: how do they collect the data?

sytse 10y ago
We use Prometheus prometheus.io We have a team of Prometheus engineers and have a vacancy for more.

carmate383 10y ago

How can a "company" with more than 150 employees, (That is essentially, a data storage company) let this happen? GitLab is surely losing subscribers faster than land in Crimea after this...

cs02rm0 10y ago
Incident upon incident followed by a mistake under pressure late at night. --it happens. I suspect the the guy responsible for the final straw is feeling pretty bad. I know I've come close to doing similar things on production environments I really didn't want to be touching while they were falling apart. But they've been honest about it. If they learn from it and six hours of database data is the worst data loss they ever experience I think it'll be a credit to them that they've been promptly transparent.

deleted 10y ago

[deleted]

da4c30ff 10y ago

Git really should have issue and merge/pull request data shipped with the repository. Does anyone know if this has been planned or not?

Ajedi32 10y ago
https://gitlab.com/gitlab-org/gitlab-ce/issues/14924 https://gitlab.com/gitlab-org/gitlab-ce/issues/14924

giancarlostoro 10y ago

On the other hand, even if there was some data loss, shouldn't most people have their entire repositories on their drives (at least the ones they're actively working on) so in theory much can be recovered by end-users who are active. The only true worry is inactive users. Not sure if this was discussed much.

yoavm 10y ago
AFAIK the problem was with the database and not the with the repos. So yeah, I have all my files on my machines, but I don't have any copy of the issues, merge requests, wiki pages etc'.

slowhand09 10y ago

TL;DR I'm sure its been said already. Doing backups is great. But testing RECOVERY is critical and should be top priority. A data company is scary. When you must back up your data because your data company can't be trusted to backup your data...