9 ms·

Libbbf: Bound Book Format, A high-performance container for comics and manga

106 points by zdw 9mo ago

its-summertime 9mo ago
https://www.reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_an_argument_on_discord_about_how/ https://www.reddit.com/r/selfhosted/comments/1qi64pr/i_got_i...
- wernsey 9mo ago
  Maybe you should quote the full title of that post: "I got into an argument on Discord about how inefficient CBR/CBZ is, so I wrote a new file format. It's 100x faster than CBZ." It has some charts, notes and comments Here's the old.reddit link: https://old.reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_an_argument_on_discord_about_how/ https://old.reddit.com/r/selfhosted/comments/1qi64pr/i_got_i...
remix2000 9mo ago
I thought zips already support random access?
dfajgljsldkjag 9mo ago
The feature matrix says cbz/zip doesn't have random page access, but it definitely does. Zip also supports appending more files without too much overhead. Certainly there's a complexity argument to be made, because you don't actually need compression just to hold a bundle of files. But these days zip just works. The perf measurement charts also make no sense. What exactly are they measuring? Edit: This reddit post seems to go into more depth on performance: old.reddit.com/r/selfhosted/comments/1qi64pr/comment/o0pqaeo/
- creata 9mo ago
  Zip also has per-asset checksums, contrary to the comparison table. And what's the point of aligning the files to be "DirectStorage-ready" if they're going to be JPEGs, a format that, as far as I know, DirectStorage doesn't understand? And the author says it's a problem that "Metadata isn't native to CBZ, you have to use a ComicInfo.xml file.", but... that's not a problem at all? The whole thing makes no sense.
  - gwern 9mo ago
    It makes no sense because it's some degree of AI slop: https://reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_an_argument_on_discord_about_how/o0p68hb/ https://reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_... Note that he doesn't quite say, when asked pointblank how much AI he used in his erroneous microbenchmarking, that he didn't use AI: https://reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_an_argument_on_discord_about_how/o0rlpl0/ https://reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_... Which explains all of it. Kudos to /u/teraflop, for having infinitely more patience with this than I would.
    - snailmailman 9mo ago
      That whole subreddit has unfortunately become inundated with AI slop. It used to be a decent resource to learn about what services people were self hosting. But now, many posts are variations of, “I’ve made this huge complicated app in an afternoon please install it on your server”. I’ve even seen a vibe-coded password manager posted there. Reputable alternatives to the software posted there exist a a huge amount of the time. Not to mention audited alternatives in the case of password managers, or even just actively maintained alternatives.
      - Semaphor 9mo ago
        3 days ago the rules changed that vibe coded stuff is only allowed on Fridays. https://old.reddit.com/r/selfhosted/comments/1qfp2t0/mod_announcement_introducing_vibe_code_friday/ https://old.reddit.com/r/selfhosted/comments/1qfp2t0/mod_ann...
    - Aransentin 9mo ago
      I'm a moderator for a decently large programming subreddit, and I'd estimate about half the project submissions now being obvious slop. You get a very good nose for sniffing that stuff out after a while, though it can be frustrating when you can't really convince other people beyond going "trust me, it's slop".
- usefulposter 9mo ago
  Bullshit asymmetry by way of impulsive LLM slop strikes again. Every new readme, announcement post, and codebase is tailored to achieve maximum bloviation. No substance, no credibility———just vibes.
  - panja 9mo ago
    If you read the reddit thread, it was coded by hand then only bug checked with ai.
    - toyg 9mo ago
      It was benchmarked with AI. Benchmarks being the main reason for this thing existing...
      - Imustaskforhelp 9mo ago
        I mean, its open source so people can create benchmark and independently verify if the AI was wrong and then have the claims be passed to the author. I haven't read the reddit thread or anything but If the author coded it by hand or is passionate about this project, he will probably understand what we are talking about. But I don't believe its such a big deal to have a benchmark be written by AI though? no?
        CyberDildonics 9mo ago
        Why would someone spend their time checking someone else's AI slop when that person couldn't even be bothered to write the basic checks that prove their project was worthwhile?
        gwern 9mo ago
        > I mean, its open source so people can create benchmark and independently verify if the AI was wrong and then have the claims be passed to the author. Thank you for volunteering. I look forward to your results.
aidenn0 9mo ago
I assume the comparison table is supposed to have something other than footnotes (e.g. check-marks or X's)? That's not showing for me on Firefox
- QuantumNomad_ 9mo ago
  There are emojis in the table for green check marks, red crosses, and yellow warning signs. Do the emojis not show for you?
  - aidenn0 9mo ago
    They do not. [edit] If I download the README I can see them in every program on my system except Firefox. I previously had issues with CJK only not displaying in Firefox, so there's probably some workaround specific to it... [Edit 2] If Firefox uses "Noto Color Emoji" (which Firefox seems to use as fallback for any font that doesn't have Emoji characters; fc-match shows a different result for e.g. :charset=2705) then I get nothing, but if I force a font that has the emoji in it (e.g. FreeSerif) then it renders. Weird.
- leosanchez 9mo ago
  They are just below the table.
riffraff 9mo ago
At a glance this looks like an obviously nicer format that a zip of jpegs, but I struggle to think of a time I thought "wow CBZ is a problem here". I didn't even realize random access is not possible, presumably because readers just support it by linear scanning or putting everything in memory at once, and comic size is peanuts compared to modern memory size. I suppose this becomes more useful if you have multiple issues/volumes in a single archive.
- aidenn0 9mo ago
  Random access is completely possible within a zip, to the degree that it's needed for cbz; you might not be able to randomly access within a file, if for some reason the cbz was stored with deflate on a jpeg, but you can always access individual files independently of each other, so seeking to a random page is O(1).
  - formerly_proven 9mo ago
    ZIP literally has a central directory. I don’t understand what’s the point of any of this over a minimal subset of PDF (one image per page).
yonisto 9mo ago
Honest question, something I don't understand, if you use DirectStorage to move images directly to the GPU (I assume into the VRAM) where the decoding take place? directly on the GPU? Can GPU decode PNG? it is very unfriendly format for GPU as far as I know
- PufPufPuf 9mo ago
  From the readme: > Note: DirectStorage isn't avaliable for images yet (as far as I know), but I've made sure to accomodate such a thing in the future with this format. So the whole DirectStorage thing is just a nothingburger. The author glosses over the fact that decoding images on GPU is not possible (or at least very impractical).
  - zigzag312 9mo ago
    It seems that at least JPEG can be decoded on the GPU [1] [2] [1] https://docs.nvidia.com/cuda/nvjpeg/index.html https://docs.nvidia.com/cuda/nvjpeg/index.html [2] https://github.com/CESNET/GPUJPEG https://github.com/CESNET/GPUJPEG
  - fc417fc802 9mo ago
    The entire thing makes no sense. How many images per second do you need to decode here? How big is the archive even? It would be one thing if you were designing a format to optimize feeding data to an ML model during training but that's not even remotely what this is supposed to be.
  - yonisto 9mo ago
    The note was added after I posted the question. It really didn't make any sense to me
chromehearts 9mo ago
But with which library are you able to host these? And which scraper currently finds manga with chapters in that file format? does anybody have experience hosting their own manga server & downloading them?
sedatk 9mo ago
> Footer indexed So, like ZIP? > Uses XXH3 for integrity checks I don’t think XXH3 is suitable for that purpose. It’s not cryptographically secure and designed mostly for stuff like hash tables (e.g. relatively small data).
- MallocVoidstar 9mo ago
  > It’s not cryptographically secure Neither is CRC32. I'm pretty sure xxhash is a straight upgrade compared to CRC32.
  - myrmidon 9mo ago
    > I'm pretty sure xxhash is a straight upgrade compared to CRC32. Unclear; performance should be pretty similar to CRC32 (depending on implementation), and since integrity checking can basically be done at RAM read speeds this should not matter either way.
- zigzag312 9mo ago
  Why would it need to be cryptographically secure for this use case?
  - rovr138 9mo ago
    If the data is big enough, collisions. Right?
    - zigzag312 9mo ago
      Then you just need a bigger hash.
jmillikin 9mo ago
I use CBZ to archive both physical and digital comic books so I was interested in the idea of an improved container format, but the claimed improvements here don't make sense. --- For example they make a big deal about each archive entry being aligned to a 4 KiB boundary "allowing for DirectStorage transfers directly from disk to GPU memory", but the pages within a CBZ are going to be encoded (JPEG/PNG/etc) rather than just being bitmaps. They need to be decoded first, the GPU isn't going to let you create a texture directly from JPEG data. Furthermore the README says "While folders allow memory mapping, individual images within them are rarely sector-aligned for optimized DirectStorage throughput" which ... what? If an image file needs to be sector-aligned (!?) then a BBF file would also need to be, else the 4 KiB alignment within the file doesn't work, so what is special about the format that causes the OS to place its files differently on disk? Also in the official DirectStorage docs (https://github.com/microsoft/DirectStorage/blob/main/Docs/DeveloperGuidance.md https://github.com/microsoft/DirectStorage/blob/main/Docs/De...) it says this: > Don't worry about 4-KiB alignment restrictions > * Win32 has a restriction that asynchronous requests be aligned on a > 4-KiB boundary and be a multiple of 4-KiB in size. > * DirectStorage does not have a 4-KiB alignment or size restriction. This > means you don't need to pad your data which just adds extra size to your > package and internal buffers. Where is the supposed 4 KiB alignment restriction even coming from? There are zip-based formats that align files so they can be mmap'd as executable pages, but that's not what's happening here, and I've never heard of a JPEG/PNG/etc image decoder that requires aligned buffers for the input data. Is the entire 4 KiB alignment requirement fictitious? --- The README also talks about using xxhash instead of CRC32 for integrity checking (the OP calls it "verification"), claiming this is more performant for large collections, but this is insane: > ZIP/RAR use CRC32, which is aging, collision-prone, and significantly slower > to verify than XXH3 for large archival collections. > [...] > On multi-core systems, the verifier splits the asset table into chunks and > validates multiple pages simultaneously. This makes BBF verification up to > 10x faster than ZIP/RAR CRC checks. CRC32 is limited by memory bandwidth if you're using a normal (i.e. SIMD) implementation. Assuming 100 GiB/s throughput, a typical comic book page (a few megabytes) will take like ... a millisecond? And there's no data dependency between file content checksums in the zip format, so for a CBZ you can run the CRC32 calculations in parallel for each page just like BBF says it does. But that doesn't matter because to actually check the integrity of archived files you want to use something like sha256, not CRC32 or xxhash. Checksum each archive (not each page), store that checksum as a `.sha256` file (or whatever), and now you can (1) use normal tools to check that your archives are intact, and (2) record those checksums as metadata in the blob storage service you're using. --- The Reddit thread has more comments from people who have noticed other sorts of discrepancies, and the author is having a really difficult time responding to them in a coherent way. The most charitable interpretation is that this whole project (supposed problems with CBZ, the readme, the code) is the output of an LLM.
PufPufPuf 9mo ago
"Native Data Deduplication" not supported in CBZ/CBR? But those are just ZIP/RAR, which are compression formats, deduplication is their whole deal...?
- greysonp 9mo ago
  They may be referring to the fact that ZIP compresses each file individually. It can't compress across files. I think RAR does compress across files though.
its-summertime 9mo ago
Thinking more about this: ZIP files can be set up to have the data on whatever alignment of one's choosing (as noted in the reddit thread). Integrity checks can be done in parallel by doing them in parallel. mmap is possible just by not using zip compression. The aspect of integrity checking speed in a saturated context (N workers, regardless if its multiple workers per file, or a worker per file), CRC32(C) seems to be nearly twice as fast https://btrfs.readthedocs.io/en/latest/Checksumming.html https://btrfs.readthedocs.io/en/latest/Checksumming.html ZIP can also support arbitrary metadata. I think this could have all been backported to ZIP files themselves
lsbehe 9mo ago
Why are the metadata blocks the way they are? I see you used pack directives but there already are plenty of padding and reserved bits. A 19 byte header just seems wrong. https://github.com/ef1500/libbbf/blob/b3ff5cb83d5ef1d841eca1e93f32a6b1baea0b87/src/libbbf.h#L27-L107 https://github.com/ef1500/libbbf/blob/b3ff5cb83d5ef1d841eca1...
grumbel 9mo ago
This feels like the wrong end to optimize. Zip is plenty of fast, especially when it comes to a few hundred pages of a comic. Meanwhile the image decoding can take a while when you want to have a quick thumbnail overview showing all those hundred pages at once. No comic/ebook software I have ever touched as managed to match the responsiveness of an actual book where you can flip through those hundreds of pages in a second with zero loading time, despite it being somewhat trivial to implement when you generate the necessary thumbnail/image-pyramid data first. A multi-resolution image format would make more sense than optimizing the archive format. There would also be room for additional features like multi-language support, searchable text, … that the current "jpg in a zip" doesn't handle (though one might end up reinventing DJVU here).
- fc417fc802 9mo ago
  > A multi-resolution image format There are already quite a few cbz archives in the wild that contain jxl encoded images. That's a multi-resolution format at least to the extent that it supports progressive decoding at fixed levels that range from 1:8 to as high as 1:4096. I think it might also support other arbitrary ratios subject to certain encoding constraints but I'm less clear on that. Readers might need to be updated to make use of the feature in an intelligent manner though. The jxl cbzs I've encountered either didn't make use of progressive encoding or else the software I used failed to take advantage of it - I'm not sure which.
Am4TIfIsER0ppos 9mo ago
What's wrong with every page being a separate image file on your disk?

106 points by zdw 9mo ago

https://www.reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_an_argument_on_discord_about_how/ https://www.reddit.com/r/selfhosted/comments/1qi64pr/i_got_i...

wernsey 9mo ago
Maybe you should quote the full title of that post: "I got into an argument on Discord about how inefficient CBR/CBZ is, so I wrote a new file format. It's 100x faster than CBZ." It has some charts, notes and comments Here's the old.reddit link: https://old.reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_an_argument_on_discord_about_how/ https://old.reddit.com/r/selfhosted/comments/1qi64pr/i_got_i...

remix2000 9mo ago

I thought zips already support random access?

dfajgljsldkjag 9mo ago

The feature matrix says cbz/zip doesn't have random page access, but it definitely does. Zip also supports appending more files without too much overhead. Certainly there's a complexity argument to be made, because you don't actually need compression just to hold a bundle of files. But these days zip just works. The perf measurement charts also make no sense. What exactly are they measuring? Edit: This reddit post seems to go into more depth on performance: old.reddit.com/r/selfhosted/comments/1qi64pr/comment/o0pqaeo/

creata 9mo ago
Zip also has per-asset checksums, contrary to the comparison table. And what's the point of aligning the files to be "DirectStorage-ready" if they're going to be JPEGs, a format that, as far as I know, DirectStorage doesn't understand? And the author says it's a problem that "Metadata isn't native to CBZ, you have to use a ComicInfo.xml file.", but... that's not a problem at all? The whole thing makes no sense.
- gwern 9mo ago
  It makes no sense because it's some degree of AI slop: https://reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_an_argument_on_discord_about_how/o0p68hb/ https://reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_... Note that he doesn't quite say, when asked pointblank how much AI he used in his erroneous microbenchmarking, that he didn't use AI: https://reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_an_argument_on_discord_about_how/o0rlpl0/ https://reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_... Which explains all of it. Kudos to /u/teraflop, for having infinitely more patience with this than I would.
  - snailmailman 9mo ago
    That whole subreddit has unfortunately become inundated with AI slop. It used to be a decent resource to learn about what services people were self hosting. But now, many posts are variations of, “I’ve made this huge complicated app in an afternoon please install it on your server”. I’ve even seen a vibe-coded password manager posted there. Reputable alternatives to the software posted there exist a a huge amount of the time. Not to mention audited alternatives in the case of password managers, or even just actively maintained alternatives.
    - Semaphor 9mo ago
      3 days ago the rules changed that vibe coded stuff is only allowed on Fridays. https://old.reddit.com/r/selfhosted/comments/1qfp2t0/mod_announcement_introducing_vibe_code_friday/ https://old.reddit.com/r/selfhosted/comments/1qfp2t0/mod_ann...
  - Aransentin 9mo ago
    I'm a moderator for a decently large programming subreddit, and I'd estimate about half the project submissions now being obvious slop. You get a very good nose for sniffing that stuff out after a while, though it can be frustrating when you can't really convince other people beyond going "trust me, it's slop".
usefulposter 9mo ago
Bullshit asymmetry by way of impulsive LLM slop strikes again. Every new readme, announcement post, and codebase is tailored to achieve maximum bloviation. No substance, no credibility———just vibes.
- panja 9mo ago
  If you read the reddit thread, it was coded by hand then only bug checked with ai.
  - toyg 9mo ago
    It was benchmarked with AI. Benchmarks being the main reason for this thing existing...
    - Imustaskforhelp 9mo ago
      I mean, its open source so people can create benchmark and independently verify if the AI was wrong and then have the claims be passed to the author. I haven't read the reddit thread or anything but If the author coded it by hand or is passionate about this project, he will probably understand what we are talking about. But I don't believe its such a big deal to have a benchmark be written by AI though? no?
      - CyberDildonics 9mo ago
        Why would someone spend their time checking someone else's AI slop when that person couldn't even be bothered to write the basic checks that prove their project was worthwhile?
      - gwern 9mo ago
        > I mean, its open source so people can create benchmark and independently verify if the AI was wrong and then have the claims be passed to the author. Thank you for volunteering. I look forward to your results.

aidenn0 9mo ago

I assume the comparison table is supposed to have something other than footnotes (e.g. check-marks or X's)? That's not showing for me on Firefox

QuantumNomad_ 9mo ago
There are emojis in the table for green check marks, red crosses, and yellow warning signs. Do the emojis not show for you?
- aidenn0 9mo ago
  They do not. [edit] If I download the README I can see them in every program on my system except Firefox. I previously had issues with CJK only not displaying in Firefox, so there's probably some workaround specific to it... [Edit 2] If Firefox uses "Noto Color Emoji" (which Firefox seems to use as fallback for any font that doesn't have Emoji characters; fc-match shows a different result for e.g. :charset=2705) then I get nothing, but if I force a font that has the emoji in it (e.g. FreeSerif) then it renders. Weird.
leosanchez 9mo ago
They are just below the table.

riffraff 9mo ago

At a glance this looks like an obviously nicer format that a zip of jpegs, but I struggle to think of a time I thought "wow CBZ is a problem here". I didn't even realize random access is not possible, presumably because readers just support it by linear scanning or putting everything in memory at once, and comic size is peanuts compared to modern memory size. I suppose this becomes more useful if you have multiple issues/volumes in a single archive.

aidenn0 9mo ago
Random access is completely possible within a zip, to the degree that it's needed for cbz; you might not be able to randomly access within a file, if for some reason the cbz was stored with deflate on a jpeg, but you can always access individual files independently of each other, so seeking to a random page is O(1).
- formerly_proven 9mo ago
  ZIP literally has a central directory. I don’t understand what’s the point of any of this over a minimal subset of PDF (one image per page).

yonisto 9mo ago

Honest question, something I don't understand, if you use DirectStorage to move images directly to the GPU (I assume into the VRAM) where the decoding take place? directly on the GPU? Can GPU decode PNG? it is very unfriendly format for GPU as far as I know

PufPufPuf 9mo ago
From the readme: > Note: DirectStorage isn't avaliable for images yet (as far as I know), but I've made sure to accomodate such a thing in the future with this format. So the whole DirectStorage thing is just a nothingburger. The author glosses over the fact that decoding images on GPU is not possible (or at least very impractical).
- zigzag312 9mo ago
  It seems that at least JPEG can be decoded on the GPU [1] [2] [1] https://docs.nvidia.com/cuda/nvjpeg/index.html https://docs.nvidia.com/cuda/nvjpeg/index.html [2] https://github.com/CESNET/GPUJPEG https://github.com/CESNET/GPUJPEG
- fc417fc802 9mo ago
  The entire thing makes no sense. How many images per second do you need to decode here? How big is the archive even? It would be one thing if you were designing a format to optimize feeding data to an ML model during training but that's not even remotely what this is supposed to be.
- yonisto 9mo ago
  The note was added after I posted the question. It really didn't make any sense to me

chromehearts 9mo ago

But with which library are you able to host these? And which scraper currently finds manga with chapters in that file format? does anybody have experience hosting their own manga server & downloading them?

sedatk 9mo ago

> Footer indexed So, like ZIP? > Uses XXH3 for integrity checks I don’t think XXH3 is suitable for that purpose. It’s not cryptographically secure and designed mostly for stuff like hash tables (e.g. relatively small data).

MallocVoidstar 9mo ago
> It’s not cryptographically secure Neither is CRC32. I'm pretty sure xxhash is a straight upgrade compared to CRC32.
- myrmidon 9mo ago
  > I'm pretty sure xxhash is a straight upgrade compared to CRC32. Unclear; performance should be pretty similar to CRC32 (depending on implementation), and since integrity checking can basically be done at RAM read speeds this should not matter either way.
zigzag312 9mo ago
Why would it need to be cryptographically secure for this use case?
- rovr138 9mo ago
  If the data is big enough, collisions. Right?
  - zigzag312 9mo ago
    Then you just need a bigger hash.

jmillikin 9mo ago

I use CBZ to archive both physical and digital comic books so I was interested in the idea of an improved container format, but the claimed improvements here don't make sense. --- For example they make a big deal about each archive entry being aligned to a 4 KiB boundary "allowing for DirectStorage transfers directly from disk to GPU memory", but the pages within a CBZ are going to be encoded (JPEG/PNG/etc) rather than just being bitmaps. They need to be decoded first, the GPU isn't going to let you create a texture directly from JPEG data. Furthermore the README says "While folders allow memory mapping, individual images within them are rarely sector-aligned for optimized DirectStorage throughput" which ... what? If an image file needs to be sector-aligned (!?) then a BBF file would also need to be, else the 4 KiB alignment within the file doesn't work, so what is special about the format that causes the OS to place its files differently on disk? Also in the official DirectStorage docs (https://github.com/microsoft/DirectStorage/blob/main/Docs/DeveloperGuidance.md https://github.com/microsoft/DirectStorage/blob/main/Docs/De...) it says this: > Don't worry about 4-KiB alignment restrictions > * Win32 has a restriction that asynchronous requests be aligned on a > 4-KiB boundary and be a multiple of 4-KiB in size. > * DirectStorage does not have a 4-KiB alignment or size restriction. This > means you don't need to pad your data which just adds extra size to your > package and internal buffers. Where is the supposed 4 KiB alignment restriction even coming from? There are zip-based formats that align files so they can be mmap'd as executable pages, but that's not what's happening here, and I've never heard of a JPEG/PNG/etc image decoder that requires aligned buffers for the input data. Is the entire 4 KiB alignment requirement fictitious? --- The README also talks about using xxhash instead of CRC32 for integrity checking (the OP calls it "verification"), claiming this is more performant for large collections, but this is insane: > ZIP/RAR use CRC32, which is aging, collision-prone, and significantly slower > to verify than XXH3 for large archival collections. > [...] > On multi-core systems, the verifier splits the asset table into chunks and > validates multiple pages simultaneously. This makes BBF verification up to > 10x faster than ZIP/RAR CRC checks. CRC32 is limited by memory bandwidth if you're using a normal (i.e. SIMD) implementation. Assuming 100 GiB/s throughput, a typical comic book page (a few megabytes) will take like ... a millisecond? And there's no data dependency between file content checksums in the zip format, so for a CBZ you can run the CRC32 calculations in parallel for each page just like BBF says it does. But that doesn't matter because to actually check the integrity of archived files you want to use something like sha256, not CRC32 or xxhash. Checksum each archive (not each page), store that checksum as a `.sha256` file (or whatever), and now you can (1) use normal tools to check that your archives are intact, and (2) record those checksums as metadata in the blob storage service you're using. --- The Reddit thread has more comments from people who have noticed other sorts of discrepancies, and the author is having a really difficult time responding to them in a coherent way. The most charitable interpretation is that this whole project (supposed problems with CBZ, the readme, the code) is the output of an LLM.

PufPufPuf 9mo ago

"Native Data Deduplication" not supported in CBZ/CBR? But those are just ZIP/RAR, which are compression formats, deduplication is their whole deal...?

greysonp 9mo ago
They may be referring to the fact that ZIP compresses each file individually. It can't compress across files. I think RAR does compress across files though.

its-summertime 9mo ago

Thinking more about this: ZIP files can be set up to have the data on whatever alignment of one's choosing (as noted in the reddit thread). Integrity checks can be done in parallel by doing them in parallel. mmap is possible just by not using zip compression. The aspect of integrity checking speed in a saturated context (N workers, regardless if its multiple workers per file, or a worker per file), CRC32(C) seems to be nearly twice as fast https://btrfs.readthedocs.io/en/latest/Checksumming.html https://btrfs.readthedocs.io/en/latest/Checksumming.html ZIP can also support arbitrary metadata. I think this could have all been backported to ZIP files themselves

lsbehe 9mo ago

Why are the metadata blocks the way they are? I see you used pack directives but there already are plenty of padding and reserved bits. A 19 byte header just seems wrong. https://github.com/ef1500/libbbf/blob/b3ff5cb83d5ef1d841eca1e93f32a6b1baea0b87/src/libbbf.h#L27-L107 https://github.com/ef1500/libbbf/blob/b3ff5cb83d5ef1d841eca1...

grumbel 9mo ago

This feels like the wrong end to optimize. Zip is plenty of fast, especially when it comes to a few hundred pages of a comic. Meanwhile the image decoding can take a while when you want to have a quick thumbnail overview showing all those hundred pages at once. No comic/ebook software I have ever touched as managed to match the responsiveness of an actual book where you can flip through those hundreds of pages in a second with zero loading time, despite it being somewhat trivial to implement when you generate the necessary thumbnail/image-pyramid data first. A multi-resolution image format would make more sense than optimizing the archive format. There would also be room for additional features like multi-language support, searchable text, … that the current "jpg in a zip" doesn't handle (though one might end up reinventing DJVU here).

fc417fc802 9mo ago
> A multi-resolution image format There are already quite a few cbz archives in the wild that contain jxl encoded images. That's a multi-resolution format at least to the extent that it supports progressive decoding at fixed levels that range from 1:8 to as high as 1:4096. I think it might also support other arbitrary ratios subject to certain encoding constraints but I'm less clear on that. Readers might need to be updated to make use of the feature in an intelligent manner though. The jxl cbzs I've encountered either didn't make use of progressive encoding or else the software I used failed to take advantage of it - I'm not sure which.

Am4TIfIsER0ppos 9mo ago

What's wrong with every page being a separate image file on your disk?