3 ms·
You don't have to recompress the whole corpus to add a single document to it. All the compression algorithms mentioned here work in a streaming fashion. You cou
by lovasoa 5y ago
You don't have to recompress the whole corpus to add a single document to it. All the compression algorithms mentioned here work in a streaming fashion. You could "just" save the internal state of the algorithm after compressing the training data, and then reuse that state for each classification task.
- Lemaxoxo 5y agoI suspected this. However, I wasn't able to grok the documentation well enough but I didn't able to find a convincing example. It seems to me that these Python compressors get "frozen" and can't be used to compress further data.
- spullara 5y agoWas going to come here to say that. Played around with this a bit for compressing small fields using a learned dictionary: https://github.com/spullara/corpuscompression https://github.com/spullara/corpuscompression