5 ms·

One thing that's sorely needed in the official documentation is a "best practice" for backup/restore from "cold and dark" where you lose your main db in a fire

One thing that's sorely needed in the official documentation is a "best practice" for backup/restore from "cold and dark" where you lose your main db in a fire and are now restoring from offsite backups for business continuity. Particularly in the 100-2TB range where probably most businesses lie, and backup/restore can take anywhere from 6 to 72 hours, often in less than ideal conditions. Like many things with SQL there's many ways to do it, but an official roadmap for order of operations would be very useful for backup/restore of roles/permissions, schema etc. You will figure it out eventually, but in my experience the dev and prod db size delta is so large many things that "just work" in the sub-1gb scale really trip you up over 200-500gb. Finding out you did one step out of order (manually, or badly written script) halfway through the restore process can mean hours and hours of rework. Heaven help you if you didn't start a screen session on your EC2 instance when you logged in.

forinti 1y ago
If you can have a secondary database (at another site or on the cloud) being updated with streaming replication, you can switch over very quickly and with little fuss.
- SoftTalker 1y ago
  Which is what you must do if minimizing downtime is critical. And, of course, your disaster recovery plan is incomplete until you've tested it (at scale). You don't want to be looking up Postgres documentation when you need to restore from a cold backup, you want to be following the checklist you have in your recovery plan and already verified.
- zie 1y ago
  Sure, but there are lots of failure modes where the failure goes with the streaming replication and all instances are trashed.
- bityard 1y ago
  There needs to be a DBA version of the saying, "RAID is not a backup"
  - lmz 1y ago
    Just expand RAID to "Replicas At Independent Datacenters".
  - CoolCold 1y ago
    While I totally agree here, replication/raid vs backups, I must say that having some weak (in terms of HW resources) replica somewhere in the closet is much much better than system with just single master.
  - forinti 1y ago
    It's not, but they complement each other nicely.
nijave 1y ago
Ideally off-site replica you fail over too and don't need to restore. pg_restore will handle roles, indexes, etc assuming you didn't switch the flags around to disable them If you're on EC2, hopefully you're using disk snapshots and WAL archiving.
- pgwhalen 1y ago
  Of course that’s preferable, but OP is specifically asking about the cold restore case, which tends to pose different problems, and is just as important to maintain and test.
- Arbortheus 1y ago
  Offsite replica is only applicable if the cause is a failure of the primary. What if I’m restoring a backup because someone accidentally dropped the wrong table?
  - ants_everywhere 1y ago
    I would hope dropping a table on a production database is something that is code reviewed
    - benreesman 1y ago
      nah, on a long enough timeline everything will go wrong. blaming the person who managed to drop the table finally is dumb: if you can't fix literally everything that could happen to it, it's not done.
    - anonymars 1y ago
      Isn't the entirety of disaster recovery about situations that aren't supposed to happen? High availability is different from disaster recovery
  - nijave 1y ago
    You can PITR on a replica which would be much faster than restoring a full backup of a large DB
WJW 1y ago
> in the 100-2TB range where probably most businesses lie Assuming you mean that range to start at 100GB, I've worked with databases that size multiple times but as a freelancer it's definitely not been "most" businesses in that range.
- 8n4vidtmkvmk 1y ago
  What then? My 10 year old SaaS is only at about 200MB compressed.
zie 1y ago
What we do, is automated restores. We have a _hourly and an _daily restore that just happens via shell script. We encourage staff to play with both, and they can play with impunity since it's a copy that will get replaced soon-ish. This makes it important that both work reliably, which means we know when our backups stop working. We haven't had a disaster recovery situation yet(hopefully never), but I feel fairly confident that getting the DB back shouldn't be a big deal.
- hadlock 1y ago
  Yes but did you have to write your own, or did you pull it from an official repo? I'm all for customizing things but we're a long, long ways from pg8.0, something besides the bare bones official pgdump and pgrestore binaries with their very agnostic and vanilla man pages would be tremendously useful.
  - zie 1y ago
    Agreed. We use barman[0] and some shell. 0: https://pgbarman.org https://pgbarman.org
vira28 1y ago
It’s one of the area where Postgres docs are light. I don’t remember they have a similar doc for setting up HA.

forinti 1y ago

If you can have a secondary database (at another site or on the cloud) being updated with streaming replication, you can switch over very quickly and with little fuss.

SoftTalker 1y ago
Which is what you must do if minimizing downtime is critical. And, of course, your disaster recovery plan is incomplete until you've tested it (at scale). You don't want to be looking up Postgres documentation when you need to restore from a cold backup, you want to be following the checklist you have in your recovery plan and already verified.
zie 1y ago
Sure, but there are lots of failure modes where the failure goes with the streaming replication and all instances are trashed.
bityard 1y ago
There needs to be a DBA version of the saying, "RAID is not a backup"
- lmz 1y ago
  Just expand RAID to "Replicas At Independent Datacenters".
- CoolCold 1y ago
  While I totally agree here, replication/raid vs backups, I must say that having some weak (in terms of HW resources) replica somewhere in the closet is much much better than system with just single master.
- forinti 1y ago
  It's not, but they complement each other nicely.

nijave 1y ago

Ideally off-site replica you fail over too and don't need to restore. pg_restore will handle roles, indexes, etc assuming you didn't switch the flags around to disable them If you're on EC2, hopefully you're using disk snapshots and WAL archiving.

pgwhalen 1y ago
Of course that’s preferable, but OP is specifically asking about the cold restore case, which tends to pose different problems, and is just as important to maintain and test.
Arbortheus 1y ago
Offsite replica is only applicable if the cause is a failure of the primary. What if I’m restoring a backup because someone accidentally dropped the wrong table?
- ants_everywhere 1y ago
  I would hope dropping a table on a production database is something that is code reviewed
  - benreesman 1y ago
    nah, on a long enough timeline everything will go wrong. blaming the person who managed to drop the table finally is dumb: if you can't fix literally everything that could happen to it, it's not done.
  - anonymars 1y ago
    Isn't the entirety of disaster recovery about situations that aren't supposed to happen? High availability is different from disaster recovery
- nijave 1y ago
  You can PITR on a replica which would be much faster than restoring a full backup of a large DB

WJW 1y ago

> in the 100-2TB range where probably most businesses lie Assuming you mean that range to start at 100GB, I've worked with databases that size multiple times but as a freelancer it's definitely not been "most" businesses in that range.

8n4vidtmkvmk 1y ago
What then? My 10 year old SaaS is only at about 200MB compressed.

zie 1y ago

What we do, is automated restores. We have a _hourly and an _daily restore that just happens via shell script. We encourage staff to play with both, and they can play with impunity since it's a copy that will get replaced soon-ish. This makes it important that both work reliably, which means we know when our backups stop working. We haven't had a disaster recovery situation yet(hopefully never), but I feel fairly confident that getting the DB back shouldn't be a big deal.

hadlock 1y ago
Yes but did you have to write your own, or did you pull it from an official repo? I'm all for customizing things but we're a long, long ways from pg8.0, something besides the bare bones official pgdump and pgrestore binaries with their very agnostic and vanilla man pages would be tremendously useful.
- zie 1y ago
  Agreed. We use barman[0] and some shell. 0: https://pgbarman.org https://pgbarman.org

vira28 1y ago

It’s one of the area where Postgres docs are light. I don’t remember they have a similar doc for setting up HA.