> Or make it automatic when the database isn't available? I agree this is probably a good idea, so I created https://gitlab.com/gitlab-com/infrastructure/issue

> Or make it automatic when the database isn't available? I agree this is probably a good idea, so I created https://gitlab.com/gitlab-com/infrastructure/issues/4382 https://gitlab.com/gitlab-com/infrastructure/issues/4382 to keep track of things. > were there any signs of impending doom before the event happened that could have warned us, and do we have alerts on that? I don't think you can detect corrupt WAL until it is actually corrupt. In those case the host won't start up, for which we already have various errors that cover that case (e.g. there's an error for when too few transactions are happening). There also wasn't really any impending doom before we started things. Two issues that triggered this were: 1. A lack of `set -e` 2. Apparently when we stop a command using `gitlab-ctl`, it will time out if the process doesn't stop fast enough. I'm not sure what exit status we report in that case, but in this instance it resulted in the script continuing to run.