4 ms·

>Get to work making your infrastructure resilient to a simple accidental deletion I'm sure they've been working on that since the deletion

by jeron 10y ago

aquabib 10y ago
But where are the stories on this work? What improvements have been made? Detailed posts on that is how you begin to restore confidence. No one is just going to take their word that "stuff are in place now".
- YorickPeterse 10y ago
  This is mostly spread across different issues in our issue tracker (https://gitlab.com/gitlab-com/infrastructure https://gitlab.com/gitlab-com/infrastructure). I suspect we'll write up a blog post once all the moving parts are in place, have been tested/used for a while, etc.
- a3_nm 10y ago
  There is a list of issues in https://about.gitlab.com/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/2017/02/10/postmortem-of-database-o... -- also in https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCx..., see Recovery, 3, l. I think it's great that they are being completely transparent about this. That said, it's true that it's been almost two months and it seems that the some important issues there are still open and don't look especially active.
  - sytse 10y ago
    The follow up was pretty extensive and we'll be working on it for months to come. Some issues that have been done: 1. Update PS1 across all hosts to more clearly differentiate between hosts and environments https://gitlab.com/gitlab-com/infrastructure/issues/1094 https://gitlab.com/gitlab-com/infrastructure/issues/1094 2. Set PostgreSQL's max_connections to a sane value https://gitlab.com/gitlab-com/infrastructure/issues/1096 https://gitlab.com/gitlab-com/infrastructure/issues/1096 3. Move staging to the ARM environment https://gitlab.com/gitlab-com/infrastructure/issues/1100 https://gitlab.com/gitlab-com/infrastructure/issues/1100 4. Improve PostgreSQL replication documentation/runbooks https://gitlab.com/gitlab-com/infrastructure/issues/1103 https://gitlab.com/gitlab-com/infrastructure/issues/1103 5. Build Streaming Database Backup https://gitlab.com/gitlab-com/infrastructure/issues/1152 https://gitlab.com/gitlab-com/infrastructure/issues/1152 6. Assign an owner for data durability https://gitlab.com/gitlab-com/infrastructure/issues/1163 https://gitlab.com/gitlab-com/infrastructure/issues/1163
hartator 10y ago
Their reaction has been at least very un-profesional. I wouldn't trust them with my data. Specifically when you have a github in front of them.
- LyndsySimon 10y ago
  My impression is the opposite of yours - I've moved personal projects from GitHub since their incident.
  - hartator 10y ago
    I respect that but why? Not the incident itself, but the architecture thwy pick point to that they have no idea about what they are doing.
    - LyndsySimon 10y ago
      Their incident response gives me confidence that this sort of thing is much less likely to happen in the future. It also brought to my attention how much they've progressed as a platform since I used them last.
      - winteriscoming 10y ago
        >> Their incident response gives me confidence that this sort of thing is much less likely to happen in the future That doesn't sound logical to me. The chances of such incidents happening isn't related to how they announce that incident.
        marcus_holmes 10y ago
        yes it totally is. A company culture of openly admitting mistakes, knowing that your team-mates will not play the blame game, means that problems will be reported quickly. In a "conventional" culture, an engineer who makes a mistake (and who will be fired for making that mistake) is incentivised to cover up the mistake and hope that the blame falls elsewhere. In an open culture, the engineer who makes a mistake is incentivised to immediately raise the alarm. So while the chances of a mistake happening are the same (we're all human), the chances of it being caught quickly and dealt with quickly are better in an open culture.

>Get to work making your infrastructure resilient to a simple accidental deletion I'm sure they've been working on that since the deletion

by jeron 10y ago

>Get to work making your infrastructure resilient to a simple accidental deletion I'm sure they've been working on that since the deletion

aquabib 10y ago

But where are the stories on this work? What improvements have been made? Detailed posts on that is how you begin to restore confidence. No one is just going to take their word that "stuff are in place now".

YorickPeterse 10y ago
This is mostly spread across different issues in our issue tracker (https://gitlab.com/gitlab-com/infrastructure https://gitlab.com/gitlab-com/infrastructure). I suspect we'll write up a blog post once all the moving parts are in place, have been tested/used for a while, etc.
a3_nm 10y ago
There is a list of issues in https://about.gitlab.com/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/2017/02/10/postmortem-of-database-o... -- also in https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCx..., see Recovery, 3, l. I think it's great that they are being completely transparent about this. That said, it's true that it's been almost two months and it seems that the some important issues there are still open and don't look especially active.
- sytse 10y ago
  The follow up was pretty extensive and we'll be working on it for months to come. Some issues that have been done: 1. Update PS1 across all hosts to more clearly differentiate between hosts and environments https://gitlab.com/gitlab-com/infrastructure/issues/1094 https://gitlab.com/gitlab-com/infrastructure/issues/1094 2. Set PostgreSQL's max_connections to a sane value https://gitlab.com/gitlab-com/infrastructure/issues/1096 https://gitlab.com/gitlab-com/infrastructure/issues/1096 3. Move staging to the ARM environment https://gitlab.com/gitlab-com/infrastructure/issues/1100 https://gitlab.com/gitlab-com/infrastructure/issues/1100 4. Improve PostgreSQL replication documentation/runbooks https://gitlab.com/gitlab-com/infrastructure/issues/1103 https://gitlab.com/gitlab-com/infrastructure/issues/1103 5. Build Streaming Database Backup https://gitlab.com/gitlab-com/infrastructure/issues/1152 https://gitlab.com/gitlab-com/infrastructure/issues/1152 6. Assign an owner for data durability https://gitlab.com/gitlab-com/infrastructure/issues/1163 https://gitlab.com/gitlab-com/infrastructure/issues/1163

hartator 10y ago

Their reaction has been at least very un-profesional. I wouldn't trust them with my data. Specifically when you have a github in front of them.

LyndsySimon 10y ago
My impression is the opposite of yours - I've moved personal projects from GitHub since their incident.
- hartator 10y ago
  I respect that but why? Not the incident itself, but the architecture thwy pick point to that they have no idea about what they are doing.
  - LyndsySimon 10y ago
    Their incident response gives me confidence that this sort of thing is much less likely to happen in the future. It also brought to my attention how much they've progressed as a platform since I used them last.
    - winteriscoming 10y ago
      >> Their incident response gives me confidence that this sort of thing is much less likely to happen in the future That doesn't sound logical to me. The chances of such incidents happening isn't related to how they announce that incident.
      - marcus_holmes 10y ago
        yes it totally is. A company culture of openly admitting mistakes, knowing that your team-mates will not play the blame game, means that problems will be reported quickly. In a "conventional" culture, an engineer who makes a mistake (and who will be fired for making that mistake) is incentivised to cover up the mistake and hope that the blame falls elsewhere. In an open culture, the engineer who makes a mistake is incentivised to immediately raise the alarm. So while the chances of a mistake happening are the same (we're all human), the chances of it being caught quickly and dealt with quickly are better in an open culture.