4 ms·

Thanks, I will read up on that. The problem I'm trying to solve, is I tried restoring c:\Users from a tar backup backup, and it took quite a while of clicking

Thanks, I will read up on that. The problem I'm trying to solve, is I tried restoring c:\Users from a tar backup backup, and it took quite a while of clicking to get all the permissions sorted out. As for "permissions were granted to some local user", in the tar file format it stores a UID/GID number, as well as a username/groupname, so when restoring to a different system the username takes precedence over the UID number. Could the same idea work on Windows? (Granted, I'd probably have to store this in the PAX header's name/value pairs, due to potentially longer length of the UID numbers). The other item I've read is that Windows file permissions can be inherited from parent objects, so it may be useful to store those permissions, but not restore them (or give a warning if the effective permissions end up being different due to restoring to a different location).

Const-me 9y ago
GetSecurityInfo() will return you the effective set of permissions/owners. Good enough for a single file, inefficient for the complete hierarchy. To backup the complete C:\Users directory to be restored later on the same PC, I’d use specially-designed APIs for that, BackupRead/BackupWrite. See this article for an overview: https://msdn.microsoft.com/en-us/library/windows/desktop/aa362520(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... That article was written in the assumption that you’ll be using a real hardware tape for backups, but you can backup to any other place. This will handle file data (including alternative NTFS streams) and file security, but you need to handle file names, file attributes, and timestamps somewhere outside of that. And TAR format can’t quite fit all that info. File names in TAR are limited to 100 characters, there’s only 1 timestamp in TAR but NTFS has 3, resolution of TAR timestamp is 1 second but NTFS keeps 100 nanoseconds, finally there’s no place to keep file attributes. I wouldn’t pick TAR format for that kind of backups. Also note that NTFS hard links and EFS encrypted files both need special care to backup.
- derekp7 9y ago
  Thanks! I'll check that one out too. Note, I'm planning on using PAX extensions for tar, so I can store an unlimited amount of data name/value pairs per file, and also handle unlimited length filenames.
  - Const-me 9y ago
    Yeah, PAX will probably work. But IMO it brings questionable value here. You’ll surely be able to create a PAX-based format that your own tools will be able to backup and restore correctly. But it won’t be compatible, i.e. no other tool will be able to restore or read the data. BackupRead API returns zero or more streams, each prefixed with a variable-length WIN32_STREAM_ID structure. For normal files without security descriptors, you could just unwrap the content of the BACKUP_DATA stream to the file content inside the TAR and you’re good. But even if you’ll find a way to store BACKUP_SECURITY_DATA stream in these PAX extended names-values, there’s other stuff to keep somewhere. Alternative data streams, usually (when written by windows explorer) they’re quite small, dozens of bytes, but nothing prevents them from being gigabytes, I’m not sure PAX will be happy about values that large. And also there’re sparse files, BackupRead won’t return you these zeroes like ReadFile, it’ll return you a BACKUP_SPARSE_BLOCK stream instead, with just the non-zero portions of the file. If you won’t bother parsing the backup data and unwrapping the streams, it’ll work for your tool, but your TARs will be incompatible with standard TAR tools which won’t see the content of your backup.
    - derekp7 9y ago
      What about storing each stream as a separate logical file in the tar file? So you have the main file, with a property of "nt_stream_count=2", and the file name for each stream would be the main filename followed by ":stream_id". That way each stream would show up as a separate file upon restore (assuming a standard tar is used for restoring), but the custom tar would know what to do. Also, unrecognized pax data would be ignored by a regular tar. And in my use case, when data is submitted by the client to the Snebu backend, it has its own tar implementation built in for separating file data from metadata, and re-synthesizes a tar file upon restore. (The reason that Snebu uses tar format for transfering, is it was a good way to make it agentless for normal Unix/Linux servers, yet extensible enough to support other use cases. Also I figured that most tar implementations were fairly well optimized, at least more so then I could do in a short amount of time). Thanks for your input, really appreciate it. Been a little bit tough learning the WIN APIs, after a couple decades lost in Unix land.

Const-me 9y ago

GetSecurityInfo() will return you the effective set of permissions/owners. Good enough for a single file, inefficient for the complete hierarchy. To backup the complete C:\Users directory to be restored later on the same PC, I’d use specially-designed APIs for that, BackupRead/BackupWrite. See this article for an overview: https://msdn.microsoft.com/en-us/library/windows/desktop/aa362520(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... That article was written in the assumption that you’ll be using a real hardware tape for backups, but you can backup to any other place. This will handle file data (including alternative NTFS streams) and file security, but you need to handle file names, file attributes, and timestamps somewhere outside of that. And TAR format can’t quite fit all that info. File names in TAR are limited to 100 characters, there’s only 1 timestamp in TAR but NTFS has 3, resolution of TAR timestamp is 1 second but NTFS keeps 100 nanoseconds, finally there’s no place to keep file attributes. I wouldn’t pick TAR format for that kind of backups. Also note that NTFS hard links and EFS encrypted files both need special care to backup.

derekp7 9y ago
Thanks! I'll check that one out too. Note, I'm planning on using PAX extensions for tar, so I can store an unlimited amount of data name/value pairs per file, and also handle unlimited length filenames.
- Const-me 9y ago
  Yeah, PAX will probably work. But IMO it brings questionable value here. You’ll surely be able to create a PAX-based format that your own tools will be able to backup and restore correctly. But it won’t be compatible, i.e. no other tool will be able to restore or read the data. BackupRead API returns zero or more streams, each prefixed with a variable-length WIN32_STREAM_ID structure. For normal files without security descriptors, you could just unwrap the content of the BACKUP_DATA stream to the file content inside the TAR and you’re good. But even if you’ll find a way to store BACKUP_SECURITY_DATA stream in these PAX extended names-values, there’s other stuff to keep somewhere. Alternative data streams, usually (when written by windows explorer) they’re quite small, dozens of bytes, but nothing prevents them from being gigabytes, I’m not sure PAX will be happy about values that large. And also there’re sparse files, BackupRead won’t return you these zeroes like ReadFile, it’ll return you a BACKUP_SPARSE_BLOCK stream instead, with just the non-zero portions of the file. If you won’t bother parsing the backup data and unwrapping the streams, it’ll work for your tool, but your TARs will be incompatible with standard TAR tools which won’t see the content of your backup.
  - derekp7 9y ago
    What about storing each stream as a separate logical file in the tar file? So you have the main file, with a property of "nt_stream_count=2", and the file name for each stream would be the main filename followed by ":stream_id". That way each stream would show up as a separate file upon restore (assuming a standard tar is used for restoring), but the custom tar would know what to do. Also, unrecognized pax data would be ignored by a regular tar. And in my use case, when data is submitted by the client to the Snebu backend, it has its own tar implementation built in for separating file data from metadata, and re-synthesizes a tar file upon restore. (The reason that Snebu uses tar format for transfering, is it was a good way to make it agentless for normal Unix/Linux servers, yet extensible enough to support other use cases. Also I figured that most tar implementations were fairly well optimized, at least more so then I could do in a short amount of time). Thanks for your input, really appreciate it. Been a little bit tough learning the WIN APIs, after a couple decades lost in Unix land.