4 ms·

So you implemented something like Windows's volume snapshots/shadow copies? (Why couldn't you use the existing feature?) EDIT: Thanks for both replies! :)

aseipp 9y ago
Pretty much. But Windows snapshots aren't writeable after you take them, which is a major downer and a major necessary feature. (Having this feature lets people do things like "Take snapshot and delete everything except *.lic files, under C:\Users, on every backup" and you can just issue deletes to the shadow volume after it is consistently created, and intercept the new deletions, like normal. Tracking deletions like this lets you more easily know what you should/should not back up -- only back up the allocated blocks in the shadow copy.) So, there was a thing that added this feature to the Windows kernel in the product to make this work. Aside from the Linux stuff, which was totally separate. But if you don't need the writing capability, Shadow Copies are good enough, sure.. (Source: I used to work with the guy who made the above post)
mbrumlow 9y ago
Yes, and no. Windows Volume Shadow Copy has the advantage of being integrated with he FS a bit closer. So in Windows VSS can avoid some overhead by skipping the 'copy' part and just allocating a new block and updating the block list for the file. For the Linux systems we had the requirement to work with all file systems (including FAT). So we could not simply modify the file system to do some fancy accounting when data in the snapshot was about to be nuked. So that resulted in me writing a module that sits between the FS and the real block driver. From there I can delay a write request long enough to ensure I can submit a read request for the same block (and wait for it to be fulfilled) before allowing the write to pass through. > (Why couldn't you use the existing feature?) We did on Windows, VSS is used with a bit of fancy stuff added on top. For Linux there is no VSS equivalent (other than the one I wrote, and maybe something somebody working on a similar product may have written). And even if one did come about (or is and I am just not aware of) it for sure was not available when I started this project.
- ploxiln 9y ago
  It sounds like you implemented something very similar to the LVM snapshot feature. You didn't want to require LVM but ... is that really worse than requiring your custom module which is roughly equivalent? EDIT: ok so it looks like the management of the snapshot space is a bit different. still, you could probably have wrapped LVM management enough to make it palatable, in less than the time it took to write a custom module
  - mbrumlow 9y ago
    The problem is requiring LVM is a NOGO if they did not already have LVM. We could have standardized on LVM, but then all the people who were not already using LVM at the time would not be able to use the product. At the time many hosting providers -- who we sold too -- just were not using LVM. We to this day still have many fewer people using LVM than raw block devices, or some sort of raid. Also at the time LVM snapshots were super slow. I don't have the numbers but even with the overhead my driver created I was able to have less impact on system performance. I was able to do some fancy stuff to optimize some of the more popular file systems by making inline look-ups to the allocation map (bitmap on ext3). This allowed me to not COW blocks that were not allocated before the snapshot. This was a huge saving because most of the time on ext3 your writes will be to newly allocated blocks. Wrapping LVM would probably not work, and still require a custom module to do, the user space tools don't do much. LVM really is a block management system that needs manage the entire block device, so existing file systems not sitting on top of LVM would get nuked if you attempted to let LVM start managing those blocks, and you still had the issue that reads and writes were coming in on a different block device. Asking people to change mount points was not a option. There were also some other requirements like block change tracking that LVM does not have the concept of doing. This is for incremental snapshots. Without this sort of tracking you will either have to checksum every block after every snapshot if you wish to to only copy the changes. This module also was responsible of reporting back to a user space daemon that keep a map of what blocks changed. So when backup time arrived we could use this list (and a few other list) to create a master list of blocks that we need to send back. This significantly cuts down on incremental backup time. Some companies call this "deduplication" but I feel that is disingenuous -- to me deduplication is on the storage side and would span across all backups. So yes, requiring a module is much easier than telling a customer they can't trial, or use this product until they took their production system off line and reformatted it with LVM. Many people hated LVM at the time, it was considered slow and caused performance problems, this was like 8 years ago... LVM has vastly changed and does not have these type of complaints any more. But I can tell you people still are going to scream bloody murder if we told them they had to redo their production images and redeploy a fleet of 200+ servers just to switch to LVM so they could get a decent backup solution. Also shout out to aseipp! Miss working with you. Have yet to find a bug in the code you wrote :p

aseipp 9y ago

Pretty much. But Windows snapshots aren't writeable after you take them, which is a major downer and a major necessary feature. (Having this feature lets people do things like "Take snapshot and delete everything except *.lic files, under C:\Users, on every backup" and you can just issue deletes to the shadow volume after it is consistently created, and intercept the new deletions, like normal. Tracking deletions like this lets you more easily know what you should/should not back up -- only back up the allocated blocks in the shadow copy.) So, there was a thing that added this feature to the Windows kernel in the product to make this work. Aside from the Linux stuff, which was totally separate. But if you don't need the writing capability, Shadow Copies are good enough, sure.. (Source: I used to work with the guy who made the above post)

mbrumlow 9y ago

Yes, and no. Windows Volume Shadow Copy has the advantage of being integrated with he FS a bit closer. So in Windows VSS can avoid some overhead by skipping the 'copy' part and just allocating a new block and updating the block list for the file. For the Linux systems we had the requirement to work with all file systems (including FAT). So we could not simply modify the file system to do some fancy accounting when data in the snapshot was about to be nuked. So that resulted in me writing a module that sits between the FS and the real block driver. From there I can delay a write request long enough to ensure I can submit a read request for the same block (and wait for it to be fulfilled) before allowing the write to pass through. > (Why couldn't you use the existing feature?) We did on Windows, VSS is used with a bit of fancy stuff added on top. For Linux there is no VSS equivalent (other than the one I wrote, and maybe something somebody working on a similar product may have written). And even if one did come about (or is and I am just not aware of) it for sure was not available when I started this project.

ploxiln 9y ago
It sounds like you implemented something very similar to the LVM snapshot feature. You didn't want to require LVM but ... is that really worse than requiring your custom module which is roughly equivalent? EDIT: ok so it looks like the management of the snapshot space is a bit different. still, you could probably have wrapped LVM management enough to make it palatable, in less than the time it took to write a custom module
- mbrumlow 9y ago
  The problem is requiring LVM is a NOGO if they did not already have LVM. We could have standardized on LVM, but then all the people who were not already using LVM at the time would not be able to use the product. At the time many hosting providers -- who we sold too -- just were not using LVM. We to this day still have many fewer people using LVM than raw block devices, or some sort of raid. Also at the time LVM snapshots were super slow. I don't have the numbers but even with the overhead my driver created I was able to have less impact on system performance. I was able to do some fancy stuff to optimize some of the more popular file systems by making inline look-ups to the allocation map (bitmap on ext3). This allowed me to not COW blocks that were not allocated before the snapshot. This was a huge saving because most of the time on ext3 your writes will be to newly allocated blocks. Wrapping LVM would probably not work, and still require a custom module to do, the user space tools don't do much. LVM really is a block management system that needs manage the entire block device, so existing file systems not sitting on top of LVM would get nuked if you attempted to let LVM start managing those blocks, and you still had the issue that reads and writes were coming in on a different block device. Asking people to change mount points was not a option. There were also some other requirements like block change tracking that LVM does not have the concept of doing. This is for incremental snapshots. Without this sort of tracking you will either have to checksum every block after every snapshot if you wish to to only copy the changes. This module also was responsible of reporting back to a user space daemon that keep a map of what blocks changed. So when backup time arrived we could use this list (and a few other list) to create a master list of blocks that we need to send back. This significantly cuts down on incremental backup time. Some companies call this "deduplication" but I feel that is disingenuous -- to me deduplication is on the storage side and would span across all backups. So yes, requiring a module is much easier than telling a customer they can't trial, or use this product until they took their production system off line and reformatted it with LVM. Many people hated LVM at the time, it was considered slow and caused performance problems, this was like 8 years ago... LVM has vastly changed and does not have these type of complaints any more. But I can tell you people still are going to scream bloody murder if we told them they had to redo their production images and redeploy a fleet of 200+ servers just to switch to LVM so they could get a decent backup solution. Also shout out to aseipp! Miss working with you. Have yet to find a bug in the code you wrote :p