3 ms·

Exactly. There is no longer any point in backing up an entire "server" or a "disk". Servers and disks are created and destroyed automatically these days. It's t

by kijin 1y ago

Exactly. There is no longer any point in backing up an entire "server" or a "disk". Servers and disks are created and destroyed automatically these days. It's the database that matters, and each type of database has its own tooling for creating "application consistent backups".

mekster 1y ago
For regular DB like MySQL/PostgreSQL, just snapshot on zfs without thinking.
- binwiederhier 1y ago
  Databases these days are pretty resilient to restoring from crash consistent backups like that, so yes, you'll likely be fine. It's a good enough approach for many cases. But you can't be sure that it really recovers. However, ZFS snapshots alone are not a good enough backup if you don't off-site them somewhere else. A server/backplane/storage controller could die or corrupt your entire zpool, or the place could burn down. Lots of ways to fail. You gotta at least zfs send the snapshots somewhere.
  - mekster 1y ago
    How do you mean can’t be sure if it recovers? It’s not hoping for inconsistent states to be recovered by the db but they’re supposed to be in good state with file system snapshotting. https://serverfault.com/a/806305 https://serverfault.com/a/806305 https://zrepl.github.io/v0.2.1/configuration/snapshotting.html#postgres-checkpoint-hook https://zrepl.github.io/v0.2.1/configuration/snapshotting.ht...
    - binwiederhier 1y ago
      Ha! I did not expect a reference to `innodb_flush_log_at_trx_commit` here. I wrote a blog post a few years ago about MySQL lossless semi-sync replication [1] and I've had quite enough of innodb_flush_log_at_trx_commit for a lifetime :-) Depending on the database you're using, and on your configuration, they may NOT recover, or require manual intervention to recover. There is a reason that MSSQL has a VSS writer in Windows, and that PostgreSQL and MySQL have their own "dump programs" that do clean backups. Pulling the plug (= file system snapshotting) without involving the database/app is risky business. Databases these days are really resilient, so I'm not saying that $yourfavoriteapp will never recover. But unless you involve the application or a VSS writer (which does that for you), you cannot be sure that it'll come back up. [1] https://blog.heckel.io/2021/10/19/lossless-mysql-semi-sync-replication-and-automated-failover/ https://blog.heckel.io/2021/10/19/lossless-mysql-semi-sync-r...
binwiederhier 1y ago
This strongly depends on your environment and on your RTO/RPO. Sure, there are environments that have automatically deployed, largely stateless servers. Why back them up if you can recreate them in an hour or two ;-) Even then, though, if we're talking about important production systems with an RTO of only a few minutes, then having a BCDR solution with instant virtualization is worth your weight in gold. I may be biased though, given that I professionally write BCDR software, hehe. However, many environments are not like that: There are lots of stateful servers out there with bespoke configurations, lots of "the customer needed this to be that way and it doesn't fit our automation". Having all servers backed up the same way gives you peace of mind if you manage servers for a living. Being able to just spin up a virtual machine of a server and run things from a backup while you restore or repair the original system is truly magical.

by kijin 1y ago

mekster 1y ago

For regular DB like MySQL/PostgreSQL, just snapshot on zfs without thinking.

binwiederhier 1y ago
Databases these days are pretty resilient to restoring from crash consistent backups like that, so yes, you'll likely be fine. It's a good enough approach for many cases. But you can't be sure that it really recovers. However, ZFS snapshots alone are not a good enough backup if you don't off-site them somewhere else. A server/backplane/storage controller could die or corrupt your entire zpool, or the place could burn down. Lots of ways to fail. You gotta at least zfs send the snapshots somewhere.
- mekster 1y ago
  How do you mean can’t be sure if it recovers? It’s not hoping for inconsistent states to be recovered by the db but they’re supposed to be in good state with file system snapshotting. https://serverfault.com/a/806305 https://serverfault.com/a/806305 https://zrepl.github.io/v0.2.1/configuration/snapshotting.html#postgres-checkpoint-hook https://zrepl.github.io/v0.2.1/configuration/snapshotting.ht...
  - binwiederhier 1y ago
    Ha! I did not expect a reference to `innodb_flush_log_at_trx_commit` here. I wrote a blog post a few years ago about MySQL lossless semi-sync replication [1] and I've had quite enough of innodb_flush_log_at_trx_commit for a lifetime :-) Depending on the database you're using, and on your configuration, they may NOT recover, or require manual intervention to recover. There is a reason that MSSQL has a VSS writer in Windows, and that PostgreSQL and MySQL have their own "dump programs" that do clean backups. Pulling the plug (= file system snapshotting) without involving the database/app is risky business. Databases these days are really resilient, so I'm not saying that $yourfavoriteapp will never recover. But unless you involve the application or a VSS writer (which does that for you), you cannot be sure that it'll come back up. [1] https://blog.heckel.io/2021/10/19/lossless-mysql-semi-sync-replication-and-automated-failover/ https://blog.heckel.io/2021/10/19/lossless-mysql-semi-sync-r...

binwiederhier 1y ago

This strongly depends on your environment and on your RTO/RPO. Sure, there are environments that have automatically deployed, largely stateless servers. Why back them up if you can recreate them in an hour or two ;-) Even then, though, if we're talking about important production systems with an RTO of only a few minutes, then having a BCDR solution with instant virtualization is worth your weight in gold. I may be biased though, given that I professionally write BCDR software, hehe. However, many environments are not like that: There are lots of stateful servers out there with bespoke configurations, lots of "the customer needed this to be that way and it doesn't fit our automation". Having all servers backed up the same way gives you peace of mind if you manage servers for a living. Being able to just spin up a virtual machine of a server and run things from a backup while you restore or repair the original system is truly magical.