5 ms·

How does this e.g. compare to Barman mentioned in the 2ndquadrant Gitlab data loss reply http://blog.2ndquadrant.com/dataloss-at-gitlab/ http://blog.2ndquadran

How does this e.g. compare to Barman mentioned in the 2ndquadrant Gitlab data loss reply http://blog.2ndquadrant.com/dataloss-at-gitlab/ http://blog.2ndquadrant.com/dataloss-at-gitlab/

craigkerstiens 10y ago
At a high-level they're very similar in the problem they solve, both are focused on giving you reliable disaster recovery. At the time Wal-E was written barman didn't exist and S3 was one of the few reliable options to backup to (this was over 5 years ago). Since then Wal-E has expanded to include just about every object store you could want, and at the same time 2Q introduced barman as their take on it. Wal-E has been used for a number of years to provide disaster recovery for Heroku Postgres, for over a million databases. It enables their follow and fork functionality, and we're using it at Citus as well for Citus Cloud given we have the person that authored it. As for exact differences I'm less familiar as I've not seriously run barman in production so perhaps someone that's run both can chime in.
- _Codemonkeyism 10y ago
  Thanks for your insight, I have no clue but will need a backup to pg in the next months, so this helped.
  - icebraining 10y ago
    Just came from a nice presentation on the PG backup fundamentals, I highly recommend watching it when the video comes online: https://fosdem.org/2017/schedule/event/postgresql_backup/ https://fosdem.org/2017/schedule/event/postgresql_backup/
    - TwistedWave 10y ago
      The video is online now.
      - _Codemonkeyism 10y ago
        Thanks!
- icebraining 10y ago
  It enables their follow and fork functionality Doesn't WAL replication only let you restore a full cluster, not a single DB? How do they get around that?
  - craigkerstiens 10y ago
    Correct, fork and follow isn't enabled on the multi-tenant level. Well, sort of, for some of the production level plans that are multi-tenant there is still a single Postgres cluster running but multiple of those on a node. Wal-E in those cases running for each one.
    - icebraining 10y ago
      It's a shame, though. Having to do WAL replication for DR plus logical backups to enable per-db restores is such a waste :|
      - user5994461 10y ago
        The real waste: Wal-e is constantly writing the transaction logs, doesn't matter if there are writes or not. (Read: a new 10MB file to S3 every 10sec, even when the DB is 100% idle).
        icebraining 10y ago
        That's weird, what's your checkpoint_timeout? The default is 5 minutes, so you certainly shouldn't be pushing to S3 every 10s if the DB is idle. Apparently PG 10 will improve your use case, though: http://paquier.xyz/postgresql-2/postgres-10-checkpoint-skip/ http://paquier.xyz/postgresql-2/postgres-10-checkpoint-skip/ EDIT: Also, wal-e compresses by default, so even those regular WAL files should be much smaller than 10MB. Are you sure the DB is really idle?
        user5994461 10y ago
        There is a setting for the time interval. That's not the point. I'm not gonna delay replication and backup by minutes just because the replication system sucks. I'd rather pay the storage for my use case. Compression is enabled indeed. Surprisingly, the compression ratio for "nothing going on" is terrible. (still multiple MBytes). The next version of postgre will redo the transaction log to have dynamic adaptive sizing, with new settings to control it. Not there yet.
fabian2k 10y ago
From what I read, Wal-E is meant to run on the database server and backups directly to S3 or equivalent while Barman typically runs on a separate server and can backup one or more remote Postgres servers.
deleted 10y ago
[deleted]
stubish 10y ago
wal-e stuffs data in 'cloud' object storage, such as S3 and compatible, WABS, Swift, and GCE's object storage. barman stores data on a filesystem. I'm using wal-e, for me a no brainer since I don't have elastic filesystems available but do have scalable object storage. Other sites will have the opposite problem.

How does this e.g. compare to Barman mentioned in the 2ndquadrant Gitlab data loss reply http://blog.2ndquadrant.com/dataloss-at-gitlab/ http://blog.2ndquadrant.com/dataloss-at-gitlab/

craigkerstiens 10y ago

At a high-level they're very similar in the problem they solve, both are focused on giving you reliable disaster recovery. At the time Wal-E was written barman didn't exist and S3 was one of the few reliable options to backup to (this was over 5 years ago). Since then Wal-E has expanded to include just about every object store you could want, and at the same time 2Q introduced barman as their take on it. Wal-E has been used for a number of years to provide disaster recovery for Heroku Postgres, for over a million databases. It enables their follow and fork functionality, and we're using it at Citus as well for Citus Cloud given we have the person that authored it. As for exact differences I'm less familiar as I've not seriously run barman in production so perhaps someone that's run both can chime in.

_Codemonkeyism 10y ago
Thanks for your insight, I have no clue but will need a backup to pg in the next months, so this helped.
- icebraining 10y ago
  Just came from a nice presentation on the PG backup fundamentals, I highly recommend watching it when the video comes online: https://fosdem.org/2017/schedule/event/postgresql_backup/ https://fosdem.org/2017/schedule/event/postgresql_backup/
  - TwistedWave 10y ago
    The video is online now.
    - _Codemonkeyism 10y ago
      Thanks!
icebraining 10y ago
It enables their follow and fork functionality Doesn't WAL replication only let you restore a full cluster, not a single DB? How do they get around that?
- craigkerstiens 10y ago
  Correct, fork and follow isn't enabled on the multi-tenant level. Well, sort of, for some of the production level plans that are multi-tenant there is still a single Postgres cluster running but multiple of those on a node. Wal-E in those cases running for each one.
  - icebraining 10y ago
    It's a shame, though. Having to do WAL replication for DR plus logical backups to enable per-db restores is such a waste :|
    - user5994461 10y ago
      The real waste: Wal-e is constantly writing the transaction logs, doesn't matter if there are writes or not. (Read: a new 10MB file to S3 every 10sec, even when the DB is 100% idle).
      - icebraining 10y ago
        That's weird, what's your checkpoint_timeout? The default is 5 minutes, so you certainly shouldn't be pushing to S3 every 10s if the DB is idle. Apparently PG 10 will improve your use case, though: http://paquier.xyz/postgresql-2/postgres-10-checkpoint-skip/ http://paquier.xyz/postgresql-2/postgres-10-checkpoint-skip/ EDIT: Also, wal-e compresses by default, so even those regular WAL files should be much smaller than 10MB. Are you sure the DB is really idle?
        user5994461 10y ago
        There is a setting for the time interval. That's not the point. I'm not gonna delay replication and backup by minutes just because the replication system sucks. I'd rather pay the storage for my use case. Compression is enabled indeed. Surprisingly, the compression ratio for "nothing going on" is terrible. (still multiple MBytes). The next version of postgre will redo the transaction log to have dynamic adaptive sizing, with new settings to control it. Not there yet.

fabian2k 10y ago

From what I read, Wal-E is meant to run on the database server and backups directly to S3 or equivalent while Barman typically runs on a separate server and can backup one or more remote Postgres servers.

deleted 10y ago

[deleted]

stubish 10y ago

wal-e stuffs data in 'cloud' object storage, such as S3 and compatible, WABS, Swift, and GCE's object storage. barman stores data on a filesystem. I'm using wal-e, for me a no brainer since I don't have elastic filesystems available but do have scalable object storage. Other sites will have the opposite problem.