3 ms·
What compression does IA use to store websites? Using a 2x better compression will allow them to store 2x more websites/content. I am doing some compression re
by AlexanderTheGr8 5y ago
What compression does IA use to store websites? Using a 2x better compression will allow them to store 2x more websites/content.
I am doing some compression research, and would love to help IA in any way I can. There are some amazing SOTA compression algorithms available now.
And if IA ignores images/video, and focuses only on text, they can store an insane amount of websites at a very low cost.
- jrwr 5y agoright now, most of it is coming in as ZSTD with a central index. older stuff was gzip
- AlexanderTheGr8 5y agoI am working on a ML-based compression which will be slow to compress/decompress but will give much better compression than zstd/gzip (even 2-3x better). Do you think that it's a useful algorithm for archival reasons or for storing huge amounts of data in cloud?