5 ms·

But where are the stories on this work? What improvements have been made? Detailed posts on that is how you begin to restore confidence. No one is just going

by aquabib 10y ago

But where are the stories on this work? What improvements have been made? Detailed posts on that is how you begin to restore confidence. No one is just going to take their word that "stuff are in place now".

YorickPeterse 10y ago
This is mostly spread across different issues in our issue tracker (https://gitlab.com/gitlab-com/infrastructure https://gitlab.com/gitlab-com/infrastructure). I suspect we'll write up a blog post once all the moving parts are in place, have been tested/used for a while, etc.
a3_nm 10y ago
There is a list of issues in https://about.gitlab.com/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/2017/02/10/postmortem-of-database-o... -- also in https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCx..., see Recovery, 3, l. I think it's great that they are being completely transparent about this. That said, it's true that it's been almost two months and it seems that the some important issues there are still open and don't look especially active.
- sytse 10y ago
  The follow up was pretty extensive and we'll be working on it for months to come. Some issues that have been done: 1. Update PS1 across all hosts to more clearly differentiate between hosts and environments https://gitlab.com/gitlab-com/infrastructure/issues/1094 https://gitlab.com/gitlab-com/infrastructure/issues/1094 2. Set PostgreSQL's max_connections to a sane value https://gitlab.com/gitlab-com/infrastructure/issues/1096 https://gitlab.com/gitlab-com/infrastructure/issues/1096 3. Move staging to the ARM environment https://gitlab.com/gitlab-com/infrastructure/issues/1100 https://gitlab.com/gitlab-com/infrastructure/issues/1100 4. Improve PostgreSQL replication documentation/runbooks https://gitlab.com/gitlab-com/infrastructure/issues/1103 https://gitlab.com/gitlab-com/infrastructure/issues/1103 5. Build Streaming Database Backup https://gitlab.com/gitlab-com/infrastructure/issues/1152 https://gitlab.com/gitlab-com/infrastructure/issues/1152 6. Assign an owner for data durability https://gitlab.com/gitlab-com/infrastructure/issues/1163 https://gitlab.com/gitlab-com/infrastructure/issues/1163

by aquabib 10y ago

YorickPeterse 10y ago

This is mostly spread across different issues in our issue tracker (https://gitlab.com/gitlab-com/infrastructure https://gitlab.com/gitlab-com/infrastructure). I suspect we'll write up a blog post once all the moving parts are in place, have been tested/used for a while, etc.

a3_nm 10y ago

There is a list of issues in https://about.gitlab.com/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/2017/02/10/postmortem-of-database-o... -- also in https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCx..., see Recovery, 3, l. I think it's great that they are being completely transparent about this. That said, it's true that it's been almost two months and it seems that the some important issues there are still open and don't look especially active.

sytse 10y ago
The follow up was pretty extensive and we'll be working on it for months to come. Some issues that have been done: 1. Update PS1 across all hosts to more clearly differentiate between hosts and environments https://gitlab.com/gitlab-com/infrastructure/issues/1094 https://gitlab.com/gitlab-com/infrastructure/issues/1094 2. Set PostgreSQL's max_connections to a sane value https://gitlab.com/gitlab-com/infrastructure/issues/1096 https://gitlab.com/gitlab-com/infrastructure/issues/1096 3. Move staging to the ARM environment https://gitlab.com/gitlab-com/infrastructure/issues/1100 https://gitlab.com/gitlab-com/infrastructure/issues/1100 4. Improve PostgreSQL replication documentation/runbooks https://gitlab.com/gitlab-com/infrastructure/issues/1103 https://gitlab.com/gitlab-com/infrastructure/issues/1103 5. Build Streaming Database Backup https://gitlab.com/gitlab-com/infrastructure/issues/1152 https://gitlab.com/gitlab-com/infrastructure/issues/1152 6. Assign an owner for data durability https://gitlab.com/gitlab-com/infrastructure/issues/1163 https://gitlab.com/gitlab-com/infrastructure/issues/1163