mhx77

3 ms·

1.
by mhx77 2y ago
FWIW, I've updated the comparison using the latest versions of EROFS and DwarFS. There definitely have been improvements to EROFS in the meantime. DwarFS is still orders of magnitude faster in creating the file system and can achieve s
2.
by mhx77 2y ago
Thanks for the feedback! I was probably a little too excited when writing the documentation. Reading it again with some distance, I can certainly see why this might be a bit off-putting. I'll keep this in mind for the future!
3.
by mhx77 2y ago
https://en.wikipedia.org/wiki/Locality-sensitive_hashing https://en.wikipedia.org/wiki/Nilsimsa_Hash https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.m
4.
by mhx77 2y ago
Correct, there are no images in the data except for 68 PNGs. It's just HTML files.
5.
by mhx77 2y ago
It's ever so slightly smaller than a .tar.xz of the same data. The main difference being that you don't have to fully extract it in order to access the data.
6.
by mhx77 2y ago
That's definitely a fair point! DwarFS actually works perfectly fine on 64-bit ARM and would likely work fine on a smartphone as well. Still (and I might be completely wrong about this), I think the primary goal of EROFS is to consume
7.
by mhx77 2y ago
I've got a (relatively old) snapshot of the English Wikipedia that I'm using for testing. The snapshot is around 200 GiB in 14,000,000 files and compresses down to an 11 GiB DwarFS image.
8.
by mhx77 2y ago
https://github.com/mhx/dwarfs?tab=readme-ov-file#with-erofs It's been a while since I did that comparison, so the results could be significantly different now.
9.
by mhx77 5y ago
Somewhat related (and definitely born out of a very similar use case): https://github.com/mhx/dwarfs I initially built this for having access to 1000+ Perl installations (spanning decades of Perl releases). The compres
10.
by mhx77 6y ago
> mkdwarfs crashed with recursive links (1-level, just pointing to itself) That's odd, it shouldn't crash with links at all, as it doesn't actively follow links. Can you please file a bug if you can reproduce this? > an
11.
by mhx77 6y ago
Yes, and I've documented this now: https://github.com/mhx/dwarfs/blob/main/doc/dwarfs.md#settin...
12.
by mhx77 6y ago
> have you investigated why this might be the case? Very briefly. It looks like clang has a different strategy breaking up the code (which is mostly C++ templates) into actual functions vs. inlining it, and the hot code ultimately perfor
13.
by mhx77 6y ago
> If you upload a module to CPAN, you automatically get it tested against a huge matrix of configurations. Very true, and it's definitely a great service! However, the set of versions/configurations is still limited, and it can
14.
by mhx77 6y ago
This is really cool, I'll give squashfs-tools-ng a try! > Does the Raspbian image comparison compare XZ compression against SquashFS with Zstd? That's correct. It's not an exhaustive matrix of comparisons. > Also, is th
15.
by mhx77 6y ago
Precisely.
16.
by mhx77 6y ago
Files that you've accessed will be kept in the kernel's cache. The cache I was talking about is a cache for decompressed blocks. Single files can stretch across multiple blocks, so you need to be able to keep more than one in memo
17.
by mhx77 6y ago
> @OP: Can you please explain why you keep 50 gigs of perl around? :-) Sure. I've been the maintainer of a perl portability module (Devel::PPPort) for a long time and every release was tested against basically every possible version
18.
by mhx77 6y ago
> How opposed would you be to this being reworked to being able to be mainline kernel support too? I don't see any way of getting this anywhere near the kernel without a full rewrite. It's C++ and it depends on libraries that a
19.
by mhx77 6y ago
Author here :) I'm not sure low-spec hardware is necessarily the best use case for DwarFS. It doesn't necessarily use less resources than SquashFS, although it can create file systems that are smaller with much less CPU resources.
20.
by mhx77 6y ago
Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the whole thing a few weeks back.
21.
by mhx77 6y ago
Read-only media, for example. Or in general, stuff that doesn't really change. In my case: https://github.com/mhx/dwarfs#history
22.
by mhx77 6y ago
Author here :) I'll add more benchmarks, this is still WIP and so far I've mainly tried to satisfy my own needs. My intention with DwarFS wasn't to write "a better SquashFS", but to make it better in certain scenari