3 ms·

does the zlib compression in python make it faster or slower?

• dividuum 8y ago
  Good question. I actually didn't test that at all. I mostly added that as I saw a few mostly empty blocks written and compression lowered the total space usage a bit.
  • stock_toaster 8y ago
    You might give lz4 a try as well. ZFS uses it as a compression option, and the overhead is quite small (especially compared to zlib).
    • voidmain 8y ago
      A sophisticated approach to block compression would be to build a dictionary using something like zstandard's "training mode" and a random sample of data blocks, store it (versioned) in FDB as well and reference the dictionary version from compressed blocks. That would get better compression with the relatively small blocks.