2 ms·
It's horses for courses. If the data is random bytes compression is unlikely. Text is easy. The different algorithms have their strengths and weaknesses.
by beardyw 4y ago
It's horses for courses. If the data is random bytes compression is unlikely. Text is easy. The different algorithms have their strengths and weaknesses.
- PaulHoule 4y agoMore than that if you want the best results you need to preprocess. PNG uses a standard compression algorithm but it can subtract rgb pixel values from the pixel above, to the left or some combination, which makes the numbers smaller and more predictable most of the time. If you had a column of numbers that were hourly temperature measurements you should do the same. If it is like a csv file you are better off compressing a whole column together than compressing a row. if you keep the rows together it will usually compress faster if you sort it. Almost any simple program which compresses the data a bit based on knowledge of the file will improve the results of a gzip/zstd/bzip2 kind of compressor run after it.
- DemocracyFTW2 4y agoI'd add that when you look into the DB and process by columns I do not doubt that frequently you will be able to come up with a compression scheme that beats generic compression of an SQL dump. But at what cost? Data is only successfully compressed when it can be reliably restored. This is not the case when I invent a new compression format that is super efficient but also hard to reverse engineer and only exists on a single or a few machines on the planet. things like gzip are bound to be around for some time, my homebrew format not so much. A compression scheme that is not widely reconstructable is a fancy way to hit the delete button for that piece of data, only with more costs involved.