6 ms·
It's a shame that Linux doesn't have APIs as comprehensive as Windows VSS. One of the several features is something called Volume Shadow Copy Service which let
by rwmj 5y ago
It's a shame that Linux doesn't have APIs as comprehensive as Windows VSS. One of the several features is something called Volume Shadow Copy Service which lets you take backups of block devices which are in use. It's kind of similar to this but more supported and I think it also interacts with user processes like databases, quiescing them so you can take a consistent snapshot.
[Also if you are playing with custom block devices in Linux -- mounted or otherwise but not /dev/sda -- https://gitlab.com/nbdkit/nbdkit https://gitlab.com/nbdkit/nbdkit https://libguestfs.org/nbdkit.1.html https://libguestfs.org/nbdkit.1.html One filter we have which is kind of similar to blktrace is called the nbdkit-stats-filter https://libguestfs.org/nbdkit-stats-filter.1.html https://libguestfs.org/nbdkit-stats-filter.1.html]
- Rygian 5y agoI've been doing my full disk backups with zfs snapshots for a while now and zfs sending them offsite. Granted it's a freebsd box, but same tools should be available on Linux.
- makr17 5y agoDefinitely possible on Linux. In the not distant past I had a large on-prem PostgreSQL DB that had a hot standby that we used for backups. Since ZFS snapshots are atomic you can just snapshot the dataset under the DB without worrying about quiescing the database. And then zfs send backups to files that are then backed up offsite. If you ever cared it's easy enough to clone one of those snapshots and bring it up as another DB instance to poke around at how things used to be...
- ryanmjacobs 5y agoI've been doing this with several production PostgreSQL instances. PostgreSQL on ZFS is great. I have zstd compression enabled and I average 3.50x compression ratios. (Probably some pretty awful CPU tradeoffs in there, but system load and query times seem fine. My database is 50 GB before zstd compression, so enabling it helps a ton.) I also have ZFS snapshots taken hourly and shipped offsite. It's awesome that I don't need to pause PostgreSQL or anything to take the snapshot. It's atomic and instant.
- donavanm 5y agoWith some quick and dirty `time` style tests zstd has a pretty low overhead. IIRC writing was around off = 100, lz4 = 110, zstd = 115 cpu utilization on my personal data set that resulted in 1x, 1.7x, 2.1x compression. Reading was negligible, single digit percentages, for both lz4 & zstd. For anything on a spindle that's a pretty good trade off of CPU time.
- emj 5y agoThat can't be supported by many databases, it must be insanely difficult to get that right, is it really used?
- namibj 5y agoIt just needs to listen to explicit fsync commands and choose a matching point-in-time. Beyond that, no, it's not hard as a database to tell the OS about moments where a block-level disk image would not need repairing, after the OS asks you. I assume this is to prevent situations that would need time-intensive WAL-replay or such, and there it'd only be opportunistic with the "sudden power loss" recovery as a fall-back path.
- mappu 5y agoOracle, MSSQL and some rarer ones (Actian Zen?) are "VSS-Aware" in that they register a writer driver. This gives them a callback before the snapshot starts - a chance to flush transactions and hold off writing any more, for a second or two until the disk snapshot is created and block tracking has begun. Then they get a second callback saying they can resume. More interesting is e.g. Hyper-V using this to snapshot the guest VMs. It just takes you one step beyond being only crash-consistent, so that everything is actually application-consistent. Implementing a VSS writer is a big 90's-style COM interface, but there is also a lightweight alternative model on top of that that eludes my search right now. Red Hat has some similar convention for freeze/thaw scripts that can make API calls to the DB to ask it to do the same thing.
- pram 5y agoDoes it just buffer to memory during this, or do the databases block writes?
- jodrellblank 5y ago> "More interesting is e.g. Hyper-V using this to snapshot the guest VMs." It can run through VMware tools into virtual machines, too, and down a layer to SAN storage and trigger a SAN snapshot so that a backup system like Veeam can take data from the SAN while knowing the guests two levels higher flushed data before the snapshot was taken. or e.g. here Nimble SAN can talk up to servers through VSS to trigger SQL Server quiesce before they take a SAN snapshot: https://infosight.hpe.com/InfoSight/media/cms/active/public/tmg_HPE_Nimble_Storage_Deployment_Considerations_for_Microsoft_SQL_Server_doc_version_family.whz/oaq1491942206687.html https://infosight.hpe.com/InfoSight/media/cms/active/public/...
- binwiederhier 5y agoThe dattobd [1] block device driver fills this gap. It basically implements point in time like behavior on Linux, with hooks similar to the ones that VSS provides. It is used in the Datto Linux backup agent in combination with the backup appliance [2], but it can be used stand alone as well (see readme). Disclaimer: I'm a software engineer at Datto. [1] https://github.com/datto/dattobd https://github.com/datto/dattobd [2] https://www.datto.com/products/siris https://www.datto.com/products/siris
- mappu 5y agoThis is just the coolest thing in the world, thank you so much for open sourcing it. Have you asked upstream what they think of it, at all?
- Conan_Kudo 5y agoHey, I'm one of the folks that work on maintaining dattobd. We haven't had a chance to talk to the upstream Linux kernel storage folks about a way to upstream it, but we're interested in exploring it, of course!
- amluto 5y agoOnce the world gets more back to normal, you could submit a talk topic to LSF/MM. Or just email the appropriate mailing list.
- Stratoscope 5y agoVSS is pretty great. I use Macrium Reflect on Windows, and it just works. I've never once had a problem restoring a Macrium backup to bare metal, even if it has a bunch of incremental backups.
- xeyownt 5y agoVSS is great indeed. Very useful to eg. P2V a machine live, even saving the P2V image on the same drive. But VSS, like LVM or other snapshoting FS, requires dedicated FS support. You don't have VSS on FAT32. Here, blktrace works at block device level, so does not rely on any FS support.
- rfjakob 5y agoI'm not sure I follow here. Linux does have LVM, what's the problem with it? That's also what the article says: If you have LVM, you don't need any special tool.
- rwmj 5y agoLVM doesn't signal to databases and other user applications so it's definitely not a replacement for VSS. As for whether LVM is a replacement for nbdkit, I guess it depends on if you like writing kernel code, or prefer instead to write nbdkit plugins in userspace in a variety of programming languages (even shell script: https://www.youtube.com/watch?v=9E5A608xJG0 https://www.youtube.com/watch?v=9E5A608xJG0)
- rfjakob 5y agoFor user application it does not signal anything I guess, but I'm not sure if that's important for databases. Databases are usually crash-consistent, which is equivalent to a snapshot. However, it does signal to the filesystem, quoting https://manned.org/fsfreeze.8 https://manned.org/fsfreeze.8 : fsfreeze is unnecessary for device-mapper devices. The device-mapper (and LVM) automatically freezes a filesystem on the device when a snapshot creation is requested. For more details see the dmsetup(8) man page.
- rwmj 5y agoNo one is doubting that databases are crash consistent. But if you read the other replies in this thread you can see there are still advantages to doing the backup in cooperation with the userspace programs, eg in time taken to restore instead of having to do full WAL replay. That's ignoring the non-database cases too.
- rfjakob 5y agoI think I saw Oracle mentioned earlier, this has some info about the VSS integration. It's called 'Oracle VSS Writer': https://docs.oracle.com/database/121/NTQRF/vss.htm#BEIJJCAE https://docs.oracle.com/database/121/NTQRF/vss.htm#BEIJJCAE I guess MS-SQL does something similar, and that does seem useful beyond what LVM does. Also VSS apparently can offload the snapshotting to a SAN ('VSS Hardware Provider'): https://docs.microsoft.com/en-us/windows-server/storage/file-server/volume-shadow-copy-service https://docs.microsoft.com/en-us/windows-server/storage/file...
- pjc50 5y agoIs VSS deprecated? They took out the "previous versions of file" feature in Windows 8.
- rwmj 5y agoApparently the GUI part was removed in Windows 8 and then re-added in Windows 10: https://en.wikipedia.org/wiki/Shadow_Copy#Windows_8_and_Server_2012 https://en.wikipedia.org/wiki/Shadow_Copy#Windows_8_and_Serv...