4 ms·

> Google Cloud just killed all our servers > we were taking Postgres base backups (to Google Cloud Storage) Rule #1 of backups - do not host backups in the sa

> Google Cloud just killed all our servers > we were taking Postgres base backups (to Google Cloud Storage) Rule #1 of backups - do not host backups in the same location as the primary

ddorian43 3y ago
The egrees fees will be bigger than your db cost.
- silon42 3y ago
  Yes, maybe (some kind of diff/sync could maybe help), but this means using such a cloud is a bad IT practice.
  - theanirudh 3y ago
    Yes, the egress fees on base backups alone were higher than the cost of the DB VMs. If we replicate the WAL also, it would be way higher. In the post, the example DB was 4.3 GB, but the WAL created was 77 GB.
    - sgarland 3y ago
      The joys of WAL bloat [0]. UUIDv4s? [0]: https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-page-writes/ https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-pa...
- deleted 3y ago
  [deleted]
scoot 3y ago
That's incorrect. You definitely do want backups in the same location as production if possible to enable rapid restore. You just don't want that to be your only copy. The canonical strategy is the 3-2-1 rule: three copies, two different media, one offsite; but there are variations, so I'd consider this the minimum.
- aembleton 3y ago
  What other media should you store backups in? Tape? Paper print out?
  - shiroiuma 3y ago
    Either papyrus or clay tablets if you want it to last. More seriously, perhaps the "2 different media" means don't use, for instance, the same brand and/or model of hard drive for your multiple backups.
    - fatihpense 3y ago
      Papyrus doesn't last :) You want clay tablets, buried in the ground. Looking at Sumerian tablets that would give you 5000 years.
      - shiroiuma 3y ago
        I thought papyrus lasted a really long time, as long as you sealed it in huge stone tombs in the desert. I think we should build a big library in a lava tube on the Moon to store all the most important data humanity has generated (important works of art and literature, Wikipedia, etc.). That's probably our best hope of really preserving so much knowledge.
        tim333 3y ago
        Some lasted at least 3000 years https://www.britannica.com/topic/Ebers-papyrus https://www.britannica.com/topic/Ebers-papyrus
  - scoot 3y ago
    Historically tape, but in practice these days it means "not on the same storage as your production data". For example in addition to a snapshot on your production system (rapid point in time recovery if the data is hosed), a local copy on deduplicated storage (recovery if the production volume is hosed), and an offsite copy derived from replicated deltas (disaster recovery if your site is hosed). The same principle can be applied to cloud hosted workloads.

> Google Cloud just killed all our servers > we were taking Postgres base backups (to Google Cloud Storage) Rule #1 of backups - do not host backups in the same location as the primary

ddorian43 3y ago

The egrees fees will be bigger than your db cost.

silon42 3y ago
Yes, maybe (some kind of diff/sync could maybe help), but this means using such a cloud is a bad IT practice.
- theanirudh 3y ago
  Yes, the egress fees on base backups alone were higher than the cost of the DB VMs. If we replicate the WAL also, it would be way higher. In the post, the example DB was 4.3 GB, but the WAL created was 77 GB.
  - sgarland 3y ago
    The joys of WAL bloat [0]. UUIDv4s? [0]: https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-page-writes/ https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-pa...
deleted 3y ago
[deleted]

scoot 3y ago

That's incorrect. You definitely do want backups in the same location as production if possible to enable rapid restore. You just don't want that to be your only copy. The canonical strategy is the 3-2-1 rule: three copies, two different media, one offsite; but there are variations, so I'd consider this the minimum.

aembleton 3y ago
What other media should you store backups in? Tape? Paper print out?
- shiroiuma 3y ago
  Either papyrus or clay tablets if you want it to last. More seriously, perhaps the "2 different media" means don't use, for instance, the same brand and/or model of hard drive for your multiple backups.
  - fatihpense 3y ago
    Papyrus doesn't last :) You want clay tablets, buried in the ground. Looking at Sumerian tablets that would give you 5000 years.
    - shiroiuma 3y ago
      I thought papyrus lasted a really long time, as long as you sealed it in huge stone tombs in the desert. I think we should build a big library in a lava tube on the Moon to store all the most important data humanity has generated (important works of art and literature, Wikipedia, etc.). That's probably our best hope of really preserving so much knowledge.
      - tim333 3y ago
        Some lasted at least 3000 years https://www.britannica.com/topic/Ebers-papyrus https://www.britannica.com/topic/Ebers-papyrus
- scoot 3y ago
  Historically tape, but in practice these days it means "not on the same storage as your production data". For example in addition to a snapshot on your production system (rapid point in time recovery if the data is hosed), a local copy on deduplicated storage (recovery if the production volume is hosed), and an offsite copy derived from replicated deltas (disaster recovery if your site is hosed). The same principle can be applied to cloud hosted workloads.