3 ms·
Wow sweet, I didn't actually expect a response! Nice work. > We have a deploy page, but it requires us to explicitly enable it, which can be easy to forget. I
by jedberg 8y ago
Wow sweet, I didn't actually expect a response! Nice work.
> We have a deploy page, but it requires us to explicitly enable it, which can be easy to forget. I do think adding a link to our Twitter status to the error pages could be useful, or maybe a widget of some kind.
Or make it automatic when the database isn't available?
> In this particular instance they go to a Slack channel called "#database", and this channel has been on fire with alerts since the problem started.
Good to know. My next question would be "were there any signs of impending doom before the event happened that could have warned us, and do we have alerts on that?"
- YorickPeterse 8y ago> Or make it automatic when the database isn't available? I agree this is probably a good idea, so I created https://gitlab.com/gitlab-com/infrastructure/issues/4382 https://gitlab.com/gitlab-com/infrastructure/issues/4382 to keep track of things. > were there any signs of impending doom before the event happened that could have warned us, and do we have alerts on that? I don't think you can detect corrupt WAL until it is actually corrupt. In those case the host won't start up, for which we already have various errors that cover that case (e.g. there's an error for when too few transactions are happening). There also wasn't really any impending doom before we started things. Two issues that triggered this were: 1. A lack of `set -e` 2. Apparently when we stop a command using `gitlab-ctl`, it will time out if the process doesn't stop fast enough. I'm not sure what exit status we report in that case, but in this instance it resulted in the script continuing to run.