8 ms·

A bunch of thoughts from a guy running postgres on a large site: A restore rate of 100Gb/hour is way too low. Over a 10GigE interface with a semi-decent SSD R

A bunch of thoughts from a guy running postgres on a large site: A restore rate of 100Gb/hour is way too low. Over a 10GigE interface with a semi-decent SSD RAID you should be pushing 1Gb/second easily. Also, pgbouncer is never the problem. If it’s complaining it’s either configured wrong or the database is having a bad time. I’d be scared out of my mind if this was my service and I was running it all on a single master with no usable backups. Then making it worse by letting production traffic hit it thereby moving the database further and further from when the isssue occurred. Those snapshots are going to be unusable unlsss the underlying FS is frozen before the snapshot gets taken. Honestly reading the notes (and not knowing anything about the team) they need less DevOps and more SysAdmin/DBA.

jfroma 9y ago
> Honestly reading the notes (and not knowing anything about the team) they need less DevOps and more SysAdmin/DBA. Having had problems in the past with Postgres and managing their database[1] I wonder why they haven't switch to a Postgres-as-a-service solution like Google Cloud SQL or Amazon RDS. It seems they are even running the site in their own hardware [2]. [1]: https://about.gitlab.com/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/2017/02/10/postmortem-of-database-o... [2]: https://about.gitlab.com/2016/12/11/proposed-server-purchase-for-gitlab-com/ https://about.gitlab.com/2016/12/11/proposed-server-purchase...
- sytse 9y ago
  GitLab.com is on Azure https://about.gitlab.com/2017/03/02/why-we-are-not-leaving-the-cloud/ https://about.gitlab.com/2017/03/02/why-we-are-not-leaving-t... and in the process of moving to Google. After the move we'll look into Cloud SQL for PostgreSQL.
- edjboston 9y ago
  GitLab VPE here. We apologize for the site performance degradation today. We do have a dedicated DB team of three people. We also have an open vacancy for the team's manager and and individual contributor. We are in the midst of a move from Azure to GCP. That means more work than usual is going on currently because we're replicating data across infrastructure-as-a-service providers and keeping the site running. This has been the case for a couple months now and it's shouldn't impact site performance. Today's site slowness was unfortunately due to a manual mistake related to a Postgres upgrade task. [2] This article is old. A move to metal never happened, and is not in our future plans. In fact, after we move to GCP we are likely to set up a CloudSQL replica and evaluate it's performance

jfroma 9y ago

> Honestly reading the notes (and not knowing anything about the team) they need less DevOps and more SysAdmin/DBA. Having had problems in the past with Postgres and managing their database[1] I wonder why they haven't switch to a Postgres-as-a-service solution like Google Cloud SQL or Amazon RDS. It seems they are even running the site in their own hardware [2]. [1]: https://about.gitlab.com/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/2017/02/10/postmortem-of-database-o... [2]: https://about.gitlab.com/2016/12/11/proposed-server-purchase-for-gitlab-com/ https://about.gitlab.com/2016/12/11/proposed-server-purchase...

sytse 9y ago
GitLab.com is on Azure https://about.gitlab.com/2017/03/02/why-we-are-not-leaving-the-cloud/ https://about.gitlab.com/2017/03/02/why-we-are-not-leaving-t... and in the process of moving to Google. After the move we'll look into Cloud SQL for PostgreSQL.
edjboston 9y ago
GitLab VPE here. We apologize for the site performance degradation today. We do have a dedicated DB team of three people. We also have an open vacancy for the team's manager and and individual contributor. We are in the midst of a move from Azure to GCP. That means more work than usual is going on currently because we're replicating data across infrastructure-as-a-service providers and keeping the site running. This has been the case for a couple months now and it's shouldn't impact site performance. Today's site slowness was unfortunately due to a manual mistake related to a Postgres upgrade task. [2] This article is old. A move to metal never happened, and is not in our future plans. In fact, after we move to GCP we are likely to set up a CloudSQL replica and evaluate it's performance