4 ms·

Thanks! I'll check that one out too. Note, I'm planning on using PAX extensions for tar, so I can store an unlimited amount of data name/value pairs per file,

Thanks! I'll check that one out too. Note, I'm planning on using PAX extensions for tar, so I can store an unlimited amount of data name/value pairs per file, and also handle unlimited length filenames.

Const-me 9y ago
Yeah, PAX will probably work. But IMO it brings questionable value here. You’ll surely be able to create a PAX-based format that your own tools will be able to backup and restore correctly. But it won’t be compatible, i.e. no other tool will be able to restore or read the data. BackupRead API returns zero or more streams, each prefixed with a variable-length WIN32_STREAM_ID structure. For normal files without security descriptors, you could just unwrap the content of the BACKUP_DATA stream to the file content inside the TAR and you’re good. But even if you’ll find a way to store BACKUP_SECURITY_DATA stream in these PAX extended names-values, there’s other stuff to keep somewhere. Alternative data streams, usually (when written by windows explorer) they’re quite small, dozens of bytes, but nothing prevents them from being gigabytes, I’m not sure PAX will be happy about values that large. And also there’re sparse files, BackupRead won’t return you these zeroes like ReadFile, it’ll return you a BACKUP_SPARSE_BLOCK stream instead, with just the non-zero portions of the file. If you won’t bother parsing the backup data and unwrapping the streams, it’ll work for your tool, but your TARs will be incompatible with standard TAR tools which won’t see the content of your backup.
- derekp7 9y ago
  What about storing each stream as a separate logical file in the tar file? So you have the main file, with a property of "nt_stream_count=2", and the file name for each stream would be the main filename followed by ":stream_id". That way each stream would show up as a separate file upon restore (assuming a standard tar is used for restoring), but the custom tar would know what to do. Also, unrecognized pax data would be ignored by a regular tar. And in my use case, when data is submitted by the client to the Snebu backend, it has its own tar implementation built in for separating file data from metadata, and re-synthesizes a tar file upon restore. (The reason that Snebu uses tar format for transfering, is it was a good way to make it agentless for normal Unix/Linux servers, yet extensible enough to support other use cases. Also I figured that most tar implementations were fairly well optimized, at least more so then I could do in a short amount of time). Thanks for your input, really appreciate it. Been a little bit tough learning the WIN APIs, after a couple decades lost in Unix land.
  - Const-me 9y ago
    > What about storing each stream as a separate logical file in the tar file? I think that’ll work. Just be sure to name these separate streams in a way so they never conflict with other files that might be in the same directory or with other streams that may be in the same file. E.g. in Windows, there’re characters forbidden in file names which may work fine in Linux: https://stackoverflow.com/a/31976060/126995 https://stackoverflow.com/a/31976060/126995 > with a property of "nt_stream_count=2" Read this: https://msdn.microsoft.com/en-us/library/windows/desktop/aa362667%28v=vs.85%29.aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... I don’t think just count is enough. Alternate data streams have names. Other data streams, extended attribute, security descriptor, etc., are essentially unnamed. You can implement some naming schemes for them, e.g. “file” for the main data, “file:xxx” for “xxx” alternative stream of the file, “file?sd” for security descriptor of the file, “file?sparse” for sparse file data. This can result in TAR backups that are more or less readable by the standard tools, but still allow your own tool to restore the complete thing. You have two problems to solve. 1. Directories. Quite often, they contain at least security descriptors. But AFAIK directories just aren’t stored in these TARs. 2. File names and their encoding. AFAIK modern Linux is UTF8 all the way down. Probably TAR is OK with that. But if UTF8 representation of the UCS2 name (WinNT isn’t quite UTF16, UCS2 is a subset of that) is longer than 100 bytes, you’ll have multiple files in your TAR with the same name, they’ll only differ by your extended PAX attributes that contain the complete name. If you combine that with the above fake names with security info & alternate streams, it becomes even more complex. P.S. I’ve been programming for windows for decades, only occasionally for linux or other platforms.

Const-me 9y ago

Yeah, PAX will probably work. But IMO it brings questionable value here. You’ll surely be able to create a PAX-based format that your own tools will be able to backup and restore correctly. But it won’t be compatible, i.e. no other tool will be able to restore or read the data. BackupRead API returns zero or more streams, each prefixed with a variable-length WIN32_STREAM_ID structure. For normal files without security descriptors, you could just unwrap the content of the BACKUP_DATA stream to the file content inside the TAR and you’re good. But even if you’ll find a way to store BACKUP_SECURITY_DATA stream in these PAX extended names-values, there’s other stuff to keep somewhere. Alternative data streams, usually (when written by windows explorer) they’re quite small, dozens of bytes, but nothing prevents them from being gigabytes, I’m not sure PAX will be happy about values that large. And also there’re sparse files, BackupRead won’t return you these zeroes like ReadFile, it’ll return you a BACKUP_SPARSE_BLOCK stream instead, with just the non-zero portions of the file. If you won’t bother parsing the backup data and unwrapping the streams, it’ll work for your tool, but your TARs will be incompatible with standard TAR tools which won’t see the content of your backup.

derekp7 9y ago
What about storing each stream as a separate logical file in the tar file? So you have the main file, with a property of "nt_stream_count=2", and the file name for each stream would be the main filename followed by ":stream_id". That way each stream would show up as a separate file upon restore (assuming a standard tar is used for restoring), but the custom tar would know what to do. Also, unrecognized pax data would be ignored by a regular tar. And in my use case, when data is submitted by the client to the Snebu backend, it has its own tar implementation built in for separating file data from metadata, and re-synthesizes a tar file upon restore. (The reason that Snebu uses tar format for transfering, is it was a good way to make it agentless for normal Unix/Linux servers, yet extensible enough to support other use cases. Also I figured that most tar implementations were fairly well optimized, at least more so then I could do in a short amount of time). Thanks for your input, really appreciate it. Been a little bit tough learning the WIN APIs, after a couple decades lost in Unix land.
- Const-me 9y ago
  > What about storing each stream as a separate logical file in the tar file? I think that’ll work. Just be sure to name these separate streams in a way so they never conflict with other files that might be in the same directory or with other streams that may be in the same file. E.g. in Windows, there’re characters forbidden in file names which may work fine in Linux: https://stackoverflow.com/a/31976060/126995 https://stackoverflow.com/a/31976060/126995 > with a property of "nt_stream_count=2" Read this: https://msdn.microsoft.com/en-us/library/windows/desktop/aa362667%28v=vs.85%29.aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... I don’t think just count is enough. Alternate data streams have names. Other data streams, extended attribute, security descriptor, etc., are essentially unnamed. You can implement some naming schemes for them, e.g. “file” for the main data, “file:xxx” for “xxx” alternative stream of the file, “file?sd” for security descriptor of the file, “file?sparse” for sparse file data. This can result in TAR backups that are more or less readable by the standard tools, but still allow your own tool to restore the complete thing. You have two problems to solve. 1. Directories. Quite often, they contain at least security descriptors. But AFAIK directories just aren’t stored in these TARs. 2. File names and their encoding. AFAIK modern Linux is UTF8 all the way down. Probably TAR is OK with that. But if UTF8 representation of the UCS2 name (WinNT isn’t quite UTF16, UCS2 is a subset of that) is longer than 100 bytes, you’ll have multiple files in your TAR with the same name, they’ll only differ by your extended PAX attributes that contain the complete name. If you combine that with the above fake names with security info & alternate streams, it becomes even more complex. P.S. I’ve been programming for windows for decades, only occasionally for linux or other platforms.