4 ms·
> You can potentially get better results this way than a generic solution like the article because you might have some application knowledge about what rows are
by phiresky 4y ago
> You can potentially get better results this way than a generic solution like the article because you might have some application knowledge about what rows are likely to be similar to others
The linked project actually does exactly this - you can basically specify an SQL expression (such as strftime(date, '%Y-%m')) that is then used to group rows together in order to decide what to train dictionaries on. What you're describing as picking every 10th row would be the expression (id/10). So you can use application knowledge, you just don't have to.
The difference is that in my project the dictionary is stored separately so not affected by row deletions, and the trained dictionary is based on all samples not just one of them. Since the dictionary is stored separately it's only worth it if you combine at least maybe 100-1000 rows under one dictionary.
> Python's zlib module, you'd use the zdict parameter
Yes, the training feature of zstd is basically a better version of keeping another sample of data around and using it as a prefix. In fact you can use a data sample instead of a trained dictionary as the `-D` parameter of the zstd command line, zstd just has the additional feature of reducing a set of samples down to the most essential (data-saving) parts.
- quietbritishjim 4y agoThanks for making this project, it looks fantastic! I might end up using it if I can figure out how to from Python. I did mention that this is what you're doing, I just thought it wasn't clear from the blog post. But I did miss that you can use an SQL expression to choose the grouping. This is ideal for the application I'm thinking of, which a "message type" field that corresponds pretty closely with which rows are similar.
- phiresky 4y agoYou can use it from python as follows: db = sqlite3.connect(...) db.enable_load_extension() db.load_extension("libsqlite_zstd.so") You probably still have to figure out how to make that work cross-platform / how to ship that binary though.
- quietbritishjim 4y agoOh interesting, I was assuming that recompiling the whole sqlite module woudl be in order. Thanks again.