4 ms·
There should be a way to pool standard dictionaries somewhere, such as a "standard english text corpus data" dictionary, that you can then download on demand fo
by pmarreck 6y ago
There should be a way to pool standard dictionaries somewhere, such as a "standard english text corpus data" dictionary, that you can then download on demand for encoding, say, BLOB text fields in a database with little to no overhead.
The way this would probably work without this facility though, say, in a database, is that the dictionary is maintained internally and constructed on the fly from the field data and not exposed to users. Although, I don't know if you'd have to keep every version of the dictionary in order to successfully decompress old data? If so then perhaps this is a niche feature
- felixhandte 6y agoW.r.t. standard dictionaries, it's something we're interested in, but the fundamental reality of dictionaries is that their effectiveness is strongly tied to their specificity. Put another way, a universal dictionary is a self-contradiction. And yes, totally, I know at least RocksDB supports exactly that behavior [0]. [0] https://github.com/facebook/rocksdb/blob/12f11373554af219c519ff60a612e355508518f7/table/block_based/block_based_table_builder.cc#L1654 https://github.com/facebook/rocksdb/blob/12f11373554af219c51...