mxmlnkn

10 ms·

61.
by mxmlnkn 3y ago
> it seems that rejecting valid trees with unused symbols were the defining factor You are correct. This assumption helps a lot to make the block finder faster. I tested with lots of compressors and I have never found one that creates &q
62.
by mxmlnkn 3y ago
Interesting, I never thought about looking for the EOB symbol like that but it makes sense. I would guess that some blocks you wouldn't be finding with this are, e.g., compressed zeros, because there would basically be only two Huffman
63.
by mxmlnkn 3y ago
That sounds more than I, for some reason, expected. I'm not sure why, but I was expecting "only" 2x speedup at maximum, while the CRC32 adds "only" 5% overhead. But, 5-8x speedup would be really nice. With that, the
64.
by mxmlnkn 3y ago
Yes, it works with files produced by the usual gzip! That's why it is novel. If you make requirements on the compressor, then a multitude of competing gzip implementations exist already, the most popular of which, I think, is bgzip [0]
65.
by mxmlnkn 3y ago
That is true and I mentioned oathtool in order to spread my knowledge. At least this trick can be used for almost any 2FA method. And, more and more keep bothering me with this, e.g., when trying to read Google mails via Thunderbird, you ha
66.
by mxmlnkn 3y ago
Currently, none. ISA-L might use some, I'm not sure. I have also tried to accelerate some crucial algorithms with SIMD and my trial and errors are still inside the src/benchmarks subfolder. But, in all of those cases, a simple std
67.
by mxmlnkn 3y ago
The mention of phone numbers also confused me. You can even store the secret, which shows besides the QR code, somewhere else and use oathtool --totp --base32 to generate tokens without a smartphone.
68.
Show HN: Rapidgzip – Truly Parallel Gzip Decompression with 10 GB/s(github.com)
35 points by mxmlnkn 3y ago|12 comments
69.
by mxmlnkn 3y ago
> After that, it’s an interesting question if they will be critiquing our code or just trash canning it all. I feel like with many replacement technologies, the human condition leads to nothing being fully replaced, even assuming your ru
70.
by mxmlnkn 3y ago
That is not the case. I have compressed the file with gzip and then tested decompression with multiple gzip decompressors. I.e., they all are fed the same compressed binary. Furthermore, I have used rapidgzip --analyze, to print out all def
71.
by mxmlnkn 3y ago
I assume you wanted to link to TurboBench and not that particular issue, which for some reason also contains a link for some car listing? Secondly, did you not see my answer to ebiggers under this comment you replied to? Yes, for Silesia, l
72.
by mxmlnkn 3y ago
I also did benchmarks with zlib and libarchivemount via their library interface here [0]. It has been a while that I have run them, so I forgot. Unfortunately, I did not add libdeflate. I did not even add ISA-l. At that point, I was already
73.
by mxmlnkn 3y ago
My extended benchmarks [0] use the rapidgzip, igzip, gzip, and pigz command line utilities and simply redirect the output to /dev/null to minimize I/O write interference. That's where I got the comparison of igzip to &qu
74.
by mxmlnkn 3y ago
Based on my benchmarks, ISA-l/igzip is more than twice(!) as fast as libdeflate and zlib for decompression. I'm almost enamored with ISA-l because of its speed. And yes, it works on AMD and also has Assembler code for ARM, so prob
75.
by mxmlnkn 3y ago
Hi, author here. You are right in the index being the easy-mode. Over the years there have been lots of implementations trying to add an index like that to the gzip metadata itself or as a sidecar file, with bgzip probably being the most kn
76.
by mxmlnkn 3y ago
Ah thanks, I was not aware that these compile-time checks are only done when it is evaluated in a compile-time evaluating context. To add to your list, using C++20 consteval instead of constexpr also triggers the error.
77.
by mxmlnkn 3y ago
I wanted to reply that your point 3 should already be possible with C++ constexpr functions because it doesn't allow undefined behavior. But I it seems I was wrong about that or maybe I'm doing it wrong: [[nodiscard]] conste
78.
by mxmlnkn 3y ago
The undefined behavior I struggle with keeps me from better performance though. I have something like [(uint32_t value) >> (32 - nbits)] & (lowest nbits set). For the case of nbits=0, I would expect it to always return 0, even if
79.
by mxmlnkn 3y ago
If you have large gzip files to decompress, I can recommend igzip[1] for fastest single-core decompression or my pet project rapidgzip[2] for effective multi-core decompression, both have simple CLI tools. For using accelerators (QAT, I ass
80.
by mxmlnkn 3y ago
That might be true for the individual RFCs but the index.html for that folder sure takes it time to load, which isn't surprising taken the roughly 10k RFC, which each are present in 5 different file formats. I remember having worse sca
81.
by mxmlnkn 3y ago
Note that HTML, Shell, and SQL appear in both the most loved and most dreaded lists.
82.
by mxmlnkn 3y ago
Or simply require an AVX2-capable CPU. According to Steam survey, 90% have AVX2 support. That's not as nice as the 99.18% SSE4.2 support or the 100% SSE3 support, but for some use cases such as scientific libraries, requiring a CPU tha
83.
by mxmlnkn 3y ago
Telegram? How so? I'm still only using it for chatting. How would you even find anything on Telegram? The simple global search? I wouldn't like to join random shady groups found through there.
84.
by mxmlnkn 3y ago
I also had a lot of problems finding usable software for drawing UML diagrams. In the end, I used UMLet, it worked sufficiently for my tiny class diagram and was easy to use and I could rearrange everything minutely as opposed to the under
85.
by mxmlnkn 3y ago
This kind of information has become even more easily accessible for everyone with the introduction of the Activity tab. I really wish that force-pushed commits and deleted branches would be permanently lost instead of being permanently stor
86.
by mxmlnkn 3y ago
Your link seems to compare GNU gzip with zstd. When comparing file formats, I would compare the best software for that file format. igzip: https://github.com/intel/isa-l can decompress consistently faster than GNU gzip
87.
by mxmlnkn 3y ago
It seems, you have to contact support and ask them to remove the commit in question. This FAQ was written before the introduction of Activity View. Pull requests always had the problem that force-pushes could not be deleted from the history
88.
Everyone can now easily view old commits on GitHub even when force-pushed(github.blog)
5 points by mxmlnkn 3y ago|2 comments
89.
Pragzip: Tool and Library for Parallel Gzip Decompression and Random Access(github.com)
3 points by mxmlnkn 3y ago|0 comments
90.
by mxmlnkn 3y ago
I have not only implemented parallel decompression but also random access to offsets in the stream with https://github.com/mxmlnkn/pragzip I did some benchmarks on some really beefy machines with 128 cores and was able

More ›