Great article Todd, One key thing to highlight is the importance of compression and using a streaming compression algorithm. Compression means there's less dat

Great article Todd, One key thing to highlight is the importance of compression and using a streaming compression algorithm. Compression means there's less data to transfer (across the network and -- even more importantly -- from disk), which means the transfers will complete faster. Not only does it allow your compressed files to be splittable (not possible with a conventional compression algorithm which requires all compressed data to have its own Huffman tree), it runs very quickly and easily adopts to a _stream_ (rather than a monolithic chunk) of data. We've just added support for LZF (a similar arithmetic/streaming compression codec) into Voldemort and performance results have been great: http://groups.google.com/group/project-voldemort/browse_thread/thread/cb366257d3714da3 http://groups.google.com/group/project-voldemort/browse_thre... Here's some background: http://en.wikipedia.org/wiki/Arithmetic_coding http://en.wikipedia.org/wiki/Arithmetic_coding http://en.wikipedia.org/wiki/Lempel_Ziv http://en.wikipedia.org/wiki/Lempel_Ziv (I had the good fortune to take an information theory class during undergrad)