p7zip's LZMA can be parallelized, and it has plenty of tuning knobs. There are other LZMA implementations, too. You want Markoff chains for really repetitive da

p7zip's LZMA can be parallelized, and it has plenty of tuning knobs. There are other LZMA implementations, too. You want Markoff chains for really repetitive data. The big disadvantage is that p7zip doesn't really do unix pipes, courtesy of its dosbox heritage. Shameless self-promotion, though weblogs are different than database dumps: http://news.ycombinator.com/item?id=1060821 http://news.ycombinator.com/item?id=1060821