5 ms·
> Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off This would ha
by tzs 9y ago
> Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off
This would have been very handy about 17 years ago when I was doing a virtual CD-ROM driver, and needed to store compressed images, and the client wanted a wide range of speed/compression options for the user--much wider than zlib offered.
I ended up doing a hack that was either the most disgusting thing I have ever done, or was brilliant. I have not been able to decide which.
My hack: I gave them a compression/speed slider that went from 0 to 100. If the slider was set to N, I would apply zlib at maximum compression to N consecutive blocks, and then apply no compression to 100-N consecutive blocks. Repeat until the whole image is stored.
The client loved it.
- bgirard 9y agoPerformance wise it's far from ideal. In your N block you're going to spend a lot of time, per-byte, to compress while leaving trivially compressible data in the remaining blocks. For N=50, you'd get much better compression for the same amount of cycles applying a faster compression to all the bytes.
- deleted 9y ago[deleted]
- gopalv 9y ago> I would apply zlib at maximum compression to N consecutive blocks, and then apply no compression to 100-N consecutive blocks Zlib is actually highly configurable - it has a huge number of tweaks beyond just that one flag of size. DEFAULT_STRATEGY is pretty much only one possible strategy. You can tweak the LZ77, RLE and Huffman separately to get different compression performance (the levels correspond to different LZ77 lookups and deflate_fast vs deflate_slow). In my work, I use different zlib strategies for different data (i.e already bit-packed integers vs raw strings), which works out pretty well. My current complaint is that the Zlib algorithm doesn't quite detect record boundaries when building a dictionary, which is all only relevant to the compression algorithm, because the output is eventually just a DEFLATE stream (Zopfli is an example of an encoder only Zlib library). Zstandard has a much wider set of options to use already built-in (fast, greedy, lazy, btlazy, btopt, btultra) etc.
- bane 9y agoThat's pretty crazy and definitely brilliant in that it solved a client problem entirely to their satisfaction.
- keyle 9y agoThe client loved it and it solved their problem. Therefore I'd call it brilliant.
- dspillett 9y agoOr the more generic: if it is you think it is stupid but it works, you might be wrong about it being stupid.
- jlgaddis 9y agoI used to like the saying, "If it's stupid but it works, then it's not stupid". Having seen some of the things my co-workers have implemented, however, I now disagree completely.
- derefr 9y agoReminds me of how LEDs are dimmed using Pulse Width Modulation: "40% brightness" actually means "on at 100% intensity for 40% of the length of each refresh cycle, and then completely off for the rest."
- szemet 9y agoOr playing PCM on the old PC speakers having only on/off beep function... It seemed like a miracle at the time :)
- vardump 9y ago> Or playing PCM on the old PC speakers having only on/off beep function That is PWM, pulse width modulation. https://en.wikipedia.org/wiki/PC_speaker#Pulse-width_modulation https://en.wikipedia.org/wiki/PC_speaker#Pulse-width_modulat...
- szemet 9y agoOk then: playing PCM with PWM...
- Cthulhu_ 9y agoPC speakers ended up being pretty amazing for some games though - there was this golf game with good environmental sounds, some platformer that had spoken text, all through the old PC speaker. Sound blaster? Pffff.
- nine_k 9y agoWell, you could play 4-channel mods via PC speaker! Many games used it, but I most fondly remember Star Control II that was bathing in great music and sound effects, all through PWM-driven PC speaker, on a 33 MHz 80386.
- Tade0 9y agoOne notable thing is that perceived brightness is not a linear function of the duty cycle. I don't know the numbers, but there exists a frequency and duty cycle at which humans perceive the light to be brighter even though there's less energy used.
- tzahola 9y agoSounds like your client was a moron.
- dspillett 9y agoNot exactly an unusual circumstance. And that is a good thing. If our clients understood the technicalities of what we do as well as we do for them, they probably wouldn't need to pay us for project work...
- tzahola 9y agoAnd it's our moral imperative to tell them why the thing they're asking for doesn't make any sense. If I were to ask a plastic surgeon to put wheels on my butt, the reasonable thing to do for him would be to reject my request, and explain me why it doesn't make any sense. I wouldn't expect him to do it and then brag on an internet forum about the clever hack he did to fit those Michelin tires onto my buttocks.
- dspillett 9y ago> And it's our moral imperative to tell them why the thing they're asking for doesn't make any sense. It is, and I do. But if they still want it anyway, unless it poses a security problem or other ethical issue or would reflect badly on us (as doing something silly very publicly could do), then it comes down to "we get paid for it or someone else does".
- tzs 9y agoHow so? The client, a major Japanese software distributor who was going to be selling copies of the software to end users, wanted a way to let the user decide how they wanted to trade off ripping speed with disk usage. That was a perfectly reasonable thing for the client to want, given his extensive knowledge of the hardware characteristics of the computers that were in widespread use among his potential customers, and his knowledge of Japanese usage patterns. What I gave him did that. So how exactly is he a moron for (1) asking for a reasonable feature, and (2) being happy that he received that feature?
- vog 9y agoNice hack, I like it! Small proposal for improvement: Instead of N * "gzip -9" + (100-N) * "cat" I would have used N * "gzip -9" + (100-N) * "gzip -1" because "gzip -1" is really, really fast. Sometimes I use it just to speed up copying data (over fast network or onto large disks). (Yes, I'm aware that zlib's output is slightly different from gzip's output, as gzip adds some more metadata on top of zlib, but this is irrelevant in terms of size and speed.)
- stardomSerf 9y agoYou would be surprised how much it is relevant in term of speed ....
- adrianmonk 9y agoMy small proposal for further improvement: change the -9 to a -7. In my experience, -9 is considerably slower than -7 but the gains in compression ratio are negligible. It's almost never a good trade-off.
- MaxBarraclough 9y agoSo your code presented a 'virtual' optical drive to the OS, and the compression was to squeeze the on-disk virtual CD image? I imagine compression could be a real performance win when reading from optical media, too. Anyone here have any experience developing games for optical-media-based games consoles? I imagine high-performance on-the-fly decompression could have been useful on the PS2, for instance. Or perhaps I'm wrong and the weak CPU and slow seek times dominate things. Vaguely related game-data madness: How Naughty Dog Fit Crash Bandicoot into 2MB of RAM on the PS1, https://news.ycombinator.com/item?id=9737156 https://news.ycombinator.com/item?id=9737156 Edit: I see compression is indeed used in those contexts - https://news.ycombinator.com/item?id=16228840 https://news.ycombinator.com/item?id=16228840
- bluedino 9y agoCompression has long been used in console games. Not only can you fit a bigger game on a ROM, but you could also use smaller ROMs (which were cheaper) to save money in the cartridge days. Some CD-ROM games would actually store more than one copy of assets on the disc. It's much faster to read all your level data in one swoop than to use performance-killing seeks.