4 ms·

Correct, fork and follow isn't enabled on the multi-tenant level. Well, sort of, for some of the production level plans that are multi-tenant there is still a s

Correct, fork and follow isn't enabled on the multi-tenant level. Well, sort of, for some of the production level plans that are multi-tenant there is still a single Postgres cluster running but multiple of those on a node. Wal-E in those cases running for each one.

icebraining 10y ago
It's a shame, though. Having to do WAL replication for DR plus logical backups to enable per-db restores is such a waste :|
- user5994461 10y ago
  The real waste: Wal-e is constantly writing the transaction logs, doesn't matter if there are writes or not. (Read: a new 10MB file to S3 every 10sec, even when the DB is 100% idle).
  - icebraining 10y ago
    That's weird, what's your checkpoint_timeout? The default is 5 minutes, so you certainly shouldn't be pushing to S3 every 10s if the DB is idle. Apparently PG 10 will improve your use case, though: http://paquier.xyz/postgresql-2/postgres-10-checkpoint-skip/ http://paquier.xyz/postgresql-2/postgres-10-checkpoint-skip/ EDIT: Also, wal-e compresses by default, so even those regular WAL files should be much smaller than 10MB. Are you sure the DB is really idle?
    - user5994461 10y ago
      There is a setting for the time interval. That's not the point. I'm not gonna delay replication and backup by minutes just because the replication system sucks. I'd rather pay the storage for my use case. Compression is enabled indeed. Surprisingly, the compression ratio for "nothing going on" is terrible. (still multiple MBytes). The next version of postgre will redo the transaction log to have dynamic adaptive sizing, with new settings to control it. Not there yet.
      - icebraining 10y ago
        AFAIK the dynamic WAL sizing has already landed in 9.5: http://www.databasesoup.com/2016/01/configuration-changes-in-95-transaction.html http://www.databasesoup.com/2016/01/configuration-changes-in... Or is this something else you're talking about?
        user5994461 10y ago
        Yes, I'm talking about that and it's in 9.5 (which was released recently)
  - wiredfool 10y ago
    That sounds like your archive_timeout is set to 10 seconds. (or close, as WAL segments are 16MB pre-compression). That's the maximum age of a WAL segment before rotation, and Postgres will force the log rotation even if there's no data in the WAL segment. If you're worried about keeping the last n seconds of data in replication, streaming replication is a far better tactic than log-shipping. (Though log shipping is useful for longer term storage)

icebraining 10y ago

It's a shame, though. Having to do WAL replication for DR plus logical backups to enable per-db restores is such a waste :|

user5994461 10y ago
The real waste: Wal-e is constantly writing the transaction logs, doesn't matter if there are writes or not. (Read: a new 10MB file to S3 every 10sec, even when the DB is 100% idle).
- icebraining 10y ago
  That's weird, what's your checkpoint_timeout? The default is 5 minutes, so you certainly shouldn't be pushing to S3 every 10s if the DB is idle. Apparently PG 10 will improve your use case, though: http://paquier.xyz/postgresql-2/postgres-10-checkpoint-skip/ http://paquier.xyz/postgresql-2/postgres-10-checkpoint-skip/ EDIT: Also, wal-e compresses by default, so even those regular WAL files should be much smaller than 10MB. Are you sure the DB is really idle?
  - user5994461 10y ago
    There is a setting for the time interval. That's not the point. I'm not gonna delay replication and backup by minutes just because the replication system sucks. I'd rather pay the storage for my use case. Compression is enabled indeed. Surprisingly, the compression ratio for "nothing going on" is terrible. (still multiple MBytes). The next version of postgre will redo the transaction log to have dynamic adaptive sizing, with new settings to control it. Not there yet.
    - icebraining 10y ago
      AFAIK the dynamic WAL sizing has already landed in 9.5: http://www.databasesoup.com/2016/01/configuration-changes-in-95-transaction.html http://www.databasesoup.com/2016/01/configuration-changes-in... Or is this something else you're talking about?
      - user5994461 10y ago
        Yes, I'm talking about that and it's in 9.5 (which was released recently)
- wiredfool 10y ago
  That sounds like your archive_timeout is set to 10 seconds. (or close, as WAL segments are 16MB pre-compression). That's the maximum age of a WAL segment before rotation, and Postgres will force the log rotation even if there's no data in the WAL segment. If you're worried about keeping the last n seconds of data in replication, streaming replication is a far better tactic than log-shipping. (Though log shipping is useful for longer term storage)