3 ms·

Hello, thank you for sharing your work. The documentation is too technical for my level. I wanted to use and understand based on simple examples for daily oper

Hello, thank you for sharing your work. The documentation is too technical for my level. I wanted to use and understand based on simple examples for daily operations but wasn't easy to find. This was eventually found on the test cases as you describe, but even there the level of complexity for the code makes it unusable for me as a lower-level programmer. My question would be for searches based on the hash, for example SHA-256 to reduce duplicates, or more advanced cases like TLSH that permits to find similar files. Would this archive be able to be used that way? The second part is about using .rgz as extension instead of .gz on the archives. If my assumption is correct, we can simply keep using .gz to maintain compatibility and familiarity with other tools, right?

ycuser20241126 2y ago
Hi and thanks for your interest. This is a specification of a format, for implementers, like an RFC would be; it's not meant to be a documentation of the reference implementation. The implementation is mostly opaque, but the public apis have javadoc. They should be as familiar as any other wrapping channel or wrapping stream would be. The extension can be .gz because it is a gzip file. I know it is convenient to test just the extension and that .rgz would be convenient. I'm not going to impose anything on the industry. It's like .jpeg or .jpg which doesn't matter that much obviously. If users end up making .rgz popular, I won't stand in the way. Ragzip detection on a file is trivially done by looking at the footer. Detection as a stream starting at byte 0 would be impractical because the first evidence of ragzip occur with the 1st of the level 1 index, showing the 'RA' subfield. Unfortunately this is after at most 4096 pages of data (assuming 4096 was the index size), far too deep in the stream. For that matter, the .rgz extension may score a point in some contexts. I have no idea what you meant with the hash and searches question. Sorry. Can you reformulate with a more complete use-case? All I can say for now, is that ragzip is meant to be seekable, it's not searching anything but positions. Thanks.

ycuser20241126 2y ago

Hi and thanks for your interest. This is a specification of a format, for implementers, like an RFC would be; it's not meant to be a documentation of the reference implementation. The implementation is mostly opaque, but the public apis have javadoc. They should be as familiar as any other wrapping channel or wrapping stream would be. The extension can be .gz because it is a gzip file. I know it is convenient to test just the extension and that .rgz would be convenient. I'm not going to impose anything on the industry. It's like .jpeg or .jpg which doesn't matter that much obviously. If users end up making .rgz popular, I won't stand in the way. Ragzip detection on a file is trivially done by looking at the footer. Detection as a stream starting at byte 0 would be impractical because the first evidence of ragzip occur with the 1st of the level 1 index, showing the 'RA' subfield. Unfortunately this is after at most 4096 pages of data (assuming 4096 was the index size), far too deep in the stream. For that matter, the .rgz extension may score a point in some contexts. I have no idea what you meant with the hash and searches question. Sorry. Can you reformulate with a more complete use-case? All I can say for now, is that ragzip is meant to be seekable, it's not searching anything but positions. Thanks.