3 ms·

Sounds like a use case for zstd with custom dictionary.

• rklaehn 8y ago
  Yes, zstd is definitely on the todo list. The increased performance should be quite welcome on low power edge devices. I tried to get it to work, but had some trouble with the existing javascript bindings. I have only recently started developing in the javascript ecosystem. PRs welcome...
  • seppel 8y ago
    It's not only increased performance. Since you can precompute a dictionary with zstd, you'll (most likely) get much better compression. Or you can at least stick to smaller block sizes.
• namibj 8y ago
  Don't forget the paq family, especially because of their tunability. You could probably replace the column representation with a context model that just classifies based on the json value or so, and context mixing, at least in the pre-zpaq days, does the rest there. Also check delta encoding. If you are able to spare the JSON format the way it is, or at least don't have to provide bit-exact representation, just data-exact ones, you might want to check out if you can make a proper json transform into a binary format suitable for this encoding, possibly checking if you can code a float represented inside a string with a printf-style encoder for the decoding, and then just store this as a datatype "single prevision, standard posix format", or "single prevision, this format, followed by a length-prefixed format string". You should make sure that some context models know where you are right now, e.g., if it's a format string, a content string, a length-prefix, etc. Context Mixing really is amazing.