2 ms·

> Speaking of postmortems, reading through the Google Doc of their outage timeline, my first postmortem question would be why is the website dependent on the da

> Speaking of postmortems, reading through the Google Doc of their outage timeline, my first postmortem question would be why is the website dependent on the database being up? You can't do anything with GitLab without the database being available. > Shouldn't there at least be a fallback that says "We're offline right now sorry" when the database is unavailable? Or in the name of extreme transparency, have it frame or show a link to the google doc of the outage? We have a deploy page, but it requires us to explicitly enable it, which can be easy to forget. I do think adding a link to our Twitter status to the error pages could be useful, or maybe a widget of some kind. > My second question would be how often do we test failures of a read slave, of the master, and of pgbouncer? In the past we tested certain scenarios whenever necessary. For example, when introducing new parts of the database load balancer we'd test what happens if we terminate X or Y. However, we do not yet do periodic chaos monkey style testing, which has been on our todo list for quite a while. > Do we at least have a practiced procedure for manual failover if not an automatic failover? Failovers are actively worked on (https://gitlab.com/gitlab-com/database/issues/91 https://gitlab.com/gitlab-com/database/issues/91, https://gitlab.com/gitlab-com/database/issues/86 https://gitlab.com/gitlab-com/database/issues/86, https://gitlab.com/gitlab-com/database/issues/40 https://gitlab.com/gitlab-com/database/issues/40). Right now our failover procedure is still quite painful, and it's easy to make mistakes. We held off using this approach for our maintenance because of these reasons, and also because a failover usually results in 2-3 hours of slow loading times. Ironically we ended up with just that. > My third question would be what sort of alerts do we have, if any, on replication lag? Plenty: https://gitlab.com/gitlab-com/runbooks/blob/810824765ec455344bdae35300a352be8c147dbb/alerts/postgresqls.yml https://gitlab.com/gitlab-com/runbooks/blob/810824765ec45534... In this particular instance they go to a Slack channel called "#database", and this channel has been on fire with alerts since the problem started.

jedberg 8y ago
Wow sweet, I didn't actually expect a response! Nice work. > We have a deploy page, but it requires us to explicitly enable it, which can be easy to forget. I do think adding a link to our Twitter status to the error pages could be useful, or maybe a widget of some kind. Or make it automatic when the database isn't available? > In this particular instance they go to a Slack channel called "#database", and this channel has been on fire with alerts since the problem started. Good to know. My next question would be "were there any signs of impending doom before the event happened that could have warned us, and do we have alerts on that?"
- YorickPeterse 8y ago
  > Or make it automatic when the database isn't available? I agree this is probably a good idea, so I created https://gitlab.com/gitlab-com/infrastructure/issues/4382 https://gitlab.com/gitlab-com/infrastructure/issues/4382 to keep track of things. > were there any signs of impending doom before the event happened that could have warned us, and do we have alerts on that? I don't think you can detect corrupt WAL until it is actually corrupt. In those case the host won't start up, for which we already have various errors that cover that case (e.g. there's an error for when too few transactions are happening). There also wasn't really any impending doom before we started things. Two issues that triggered this were: 1. A lack of `set -e` 2. Apparently when we stop a command using `gitlab-ctl`, it will time out if the process doesn't stop fast enough. I'm not sure what exit status we report in that case, but in this instance it resulted in the script continuing to run.
merb 8y ago
you should probably start using zalando/patroni on a k8s cluster. and scale the db in k8s and maybe use a lb on the outside of k8s with ingress/lb. it's way simplier than repmgr, etc.

jedberg 8y ago

Wow sweet, I didn't actually expect a response! Nice work. > We have a deploy page, but it requires us to explicitly enable it, which can be easy to forget. I do think adding a link to our Twitter status to the error pages could be useful, or maybe a widget of some kind. Or make it automatic when the database isn't available? > In this particular instance they go to a Slack channel called "#database", and this channel has been on fire with alerts since the problem started. Good to know. My next question would be "were there any signs of impending doom before the event happened that could have warned us, and do we have alerts on that?"

YorickPeterse 8y ago
> Or make it automatic when the database isn't available? I agree this is probably a good idea, so I created https://gitlab.com/gitlab-com/infrastructure/issues/4382 https://gitlab.com/gitlab-com/infrastructure/issues/4382 to keep track of things. > were there any signs of impending doom before the event happened that could have warned us, and do we have alerts on that? I don't think you can detect corrupt WAL until it is actually corrupt. In those case the host won't start up, for which we already have various errors that cover that case (e.g. there's an error for when too few transactions are happening). There also wasn't really any impending doom before we started things. Two issues that triggered this were: 1. A lack of `set -e` 2. Apparently when we stop a command using `gitlab-ctl`, it will time out if the process doesn't stop fast enough. I'm not sure what exit status we report in that case, but in this instance it resulted in the script continuing to run.

merb 8y ago

you should probably start using zalando/patroni on a k8s cluster. and scale the db in k8s and maybe use a lb on the outside of k8s with ingress/lb. it's way simplier than repmgr, etc.