4 ms·

In my experience the difference is ~10% (.3 ratio vs ~.23 ratio) but for the increase in processing speed it's well worth it. If the tool supports Hadoop's Fil

In my experience the difference is ~10% (.3 ratio vs ~.23 ratio) but for the increase in processing speed it's well worth it. If the tool supports Hadoop's FileInputFormat or a stream of whatever underlying data (you can use the 4mc or lz4 command line tool) it will work.

mattbillenstein 11y ago
Processing speed due to the decompression algorithm or the I/O? You can get another 10% with gzip -7 vs the default I bet. I don't find the compression/decompression the bottleneck in most of this work and even then an extra 10% isn't worth sacrificing the interoperability between all the systems that support .json.gz Higher gzip levels I think only really affect the compression step -- decompression is the same cost. These are mostly write-once read many times sorts of datasets.
- faizshah 11y ago
  It's due to the decompression algorithm, gzip gets better file sizes and is a better general solution. lz4 is usually 10% worse but will get vastly better decompression speeds. The I/O of an SSD is usually 300+MBps, lz4 will usually be around 270MBps, gzip will usually be <80MBps. lz4 will put less load on your cpu allowing you to get more throughput from your processing tasks. But I agree most people don't reach these bottlenecks because most people use tools which are not optimized for performance like using jackson or jq instead of boon or rapidjson. Look here for a reproducible test I ran on real data: https://news.ycombinator.com/item?id=10721645 https://news.ycombinator.com/item?id=10721645
  - mattbillenstein 11y ago
    Cool -- good stuff. Those compressors should arguably be more widely supported.
    - faizshah 11y ago
      I agree, disk is cheap it's compute that's expensive. One thing to note though is that lz4 will use a lot more RAM relative to gzip but compared to processing tasks it's not much.

4 ms·

In my experience the difference is ~10% (.3 ratio vs ~.23 ratio) but for the increase in processing speed it's well worth it. If the tool supports Hadoop's Fil

by faizshah 11y ago

mattbillenstein 11y ago
Processing speed due to the decompression algorithm or the I/O? You can get another 10% with gzip -7 vs the default I bet. I don't find the compression/decompression the bottleneck in most of this work and even then an extra 10% isn't worth sacrificing the interoperability between all the systems that support .json.gz Higher gzip levels I think only really affect the compression step -- decompression is the same cost. These are mostly write-once read many times sorts of datasets.
- faizshah 11y ago
  It's due to the decompression algorithm, gzip gets better file sizes and is a better general solution. lz4 is usually 10% worse but will get vastly better decompression speeds. The I/O of an SSD is usually 300+MBps, lz4 will usually be around 270MBps, gzip will usually be <80MBps. lz4 will put less load on your cpu allowing you to get more throughput from your processing tasks. But I agree most people don't reach these bottlenecks because most people use tools which are not optimized for performance like using jackson or jq instead of boon or rapidjson. Look here for a reproducible test I ran on real data: https://news.ycombinator.com/item?id=10721645 https://news.ycombinator.com/item?id=10721645
  - mattbillenstein 11y ago
    Cool -- good stuff. Those compressors should arguably be more widely supported.
    - faizshah 11y ago
      I agree, disk is cheap it's compute that's expensive. One thing to note though is that lz4 will use a lot more RAM relative to gzip but compared to processing tasks it's not much.