4 ms·

Since our move to multicore over faster processors, I'm sure we'll see a lot of this sort of thing, that is, people suddenly realizing that their code will be s

by th0ma5 14y ago

Since our move to multicore over faster processors, I'm sure we'll see a lot of this sort of thing, that is, people suddenly realizing that their code will be some multiple faster if they can find a way to do operations in parallel. I imagine that the compression itself might be slightly less optimal however since similar blocks that could be compressed are on different threads? I didn't dig into how this might or might not be a concern with this project, however. Long of the short of it, however, parallel is the reality. In theory one could arbitrarily split the file, and then compress each of the splits and get a speed up that is roughly comparable?

malkia 14y ago
"In theory one could arbitrarily split the file, and then compress each of the splits and get a speed up that is roughly comparable?" - it could. that's what is done, but for LZ type compressor bigger dictionary (e.g. avoiding splits is for the better) - one more reason why there is .tar -> .tgz (or .tbz), while .zip compresses individually. It would be even better if files are arranged in such way that contentwise they are similar one after each other. I'm not compression expert, just average user since the early days of what we used to call it back in the day "solid" compression.
wtetzner 14y ago
>In theory one could arbitrarily split the file, and then compress each of the splits and get a speed up that is roughly comparable? It doesn't seem unlikely that that's what they're doing, considering you can't pipe data to it on stdin.
CJefferson 14y ago
bz2, out of all current compression methods, is particularly parallisable, as it has already split the files up into 900k (or smaller) blocks, and compressed each individually (well, run BWT on each seperately at least).
stcredzero 14y ago
Since our move to multicore over faster processors, I'm sure we'll see a lot of this sort of thing, that is, people suddenly realizing that their code will be some multiple faster if they can find a way to do operations in parallel. Reimplementation of things like compression algorithms seems very math/algorithm-heavy and thus amenable to functional programming. How about the Haskell/OCaml guys re-implement a bunch of Un^x style utilities for us?
- th0ma5 14y ago
  in theory gnu parallel can be used to automate an idea in place like xargs

by th0ma5 14y ago

malkia 14y ago

"In theory one could arbitrarily split the file, and then compress each of the splits and get a speed up that is roughly comparable?" - it could. that's what is done, but for LZ type compressor bigger dictionary (e.g. avoiding splits is for the better) - one more reason why there is .tar -> .tgz (or .tbz), while .zip compresses individually. It would be even better if files are arranged in such way that contentwise they are similar one after each other. I'm not compression expert, just average user since the early days of what we used to call it back in the day "solid" compression.

wtetzner 14y ago

>In theory one could arbitrarily split the file, and then compress each of the splits and get a speed up that is roughly comparable? It doesn't seem unlikely that that's what they're doing, considering you can't pipe data to it on stdin.

CJefferson 14y ago

bz2, out of all current compression methods, is particularly parallisable, as it has already split the files up into 900k (or smaller) blocks, and compressed each individually (well, run BWT on each seperately at least).

stcredzero 14y ago

Since our move to multicore over faster processors, I'm sure we'll see a lot of this sort of thing, that is, people suddenly realizing that their code will be some multiple faster if they can find a way to do operations in parallel. Reimplementation of things like compression algorithms seems very math/algorithm-heavy and thus amenable to functional programming. How about the Haskell/OCaml guys re-implement a bunch of Un^x style utilities for us?

th0ma5 14y ago
in theory gnu parallel can be used to automate an idea in place like xargs