For those curious: the root cause for the incident is, ultimately, degraded statistics on a table. There are a set of high frequency (very frequently run queri

For those curious: the root cause for the incident is, ultimately, degraded statistics on a table. There are a set of high frequency (very frequently run queries) that are quite sensible to plan flipping (they normally run with a given execution plan; if the plan changes to a worse one, effects can be dramatic, given their frequency). That leads to a lot of query timeouts (GitLab's database limit query execution time to prevent further damage), which are visualized as database errors. This in turn leads to partial or complete downtime (effectively, if the database is running few effective --non timed out-- traffic). Some of these queries depend on a particular Postgres planner way of working that is less than ideal (in PG11, it is improved in PG12), and may lead to plan flipping. Which in turn, happens when table statistics degrade. In GitLab's case, they are normally close to the threshold, and slight statistics degradation caused a plan change, which in turn lead to many queries time out and excessive load on the fleet. At the end of the day, the fix is quite simple, however: update the table statistics (running ANALYZE). For more information: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3962#note_529314106 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...