3 ms·
Will this mode require a separate file along the data.db Sqlite file, to store the compression dictionary? From your last link: > The result of this training
by josephernest 4y ago
Will this mode require a separate file along the data.db Sqlite file, to store the compression dictionary?
From your last link:
> The result of this training is stored in a file called "dictionary", which must be loaded before compression and decompression.
Also, what happens if a column has been compressed based on 100 typical values, and then later we insert thousands of new rows with 500 new frequent values.
Does the compression dictionary get automatically updated? Then do old rows need to be rewritten?
PS what is compact mode?
- Scaevolus 4y agoIt stores these compression dictionaries in the database: > If dictionary is an int i, it is functionally equivalent to zstd_decompress(data, (select dict from _zstd_dict where id = i)). Compact saves you 8 bytes per compressed column by removing some metadata zstd uses : > if compact is true, the output will be without magic header, without checksums, and without dictids. This will save 4 bytes when not using dictionaries and 8 bytes when using dictionaries. this also means the data will not be decodeable as a normal zstd archive with the standard tools. The same compact argument must also be passed to the decompress function. You will need to update the dicts occasionally, but there's a helper function to do this for you, read the docs about zstd_incremental_maintenance.