4 ms·
It's just HTML/text and it's 11TB gzipped as of 2008. I bet the archive has grown since then, but the valuable data is the oldest. Uncompressed it's probably in
by AngryParsley 16y ago
It's just HTML/text and it's 11TB gzipped as of 2008. I bet the archive has grown since then, but the valuable data is the oldest. Uncompressed it's probably in the 20TB range.
I'd also bet everything I own that a significant fraction of the data is unavailable anywhere else. Google Reader started in 2005. Ditto for Google Blog Search. Even if they crawled the same feeds as Bloglines and kept the data indefinitely, they'll still be missing those first two years.
- gwern 16y ago> Uncompressed it's probably in the 20TB range. That seems a little low to me if you're starting with an 11TB archive. Doesn't gzip usually shrink HTML/text down to more like 1/5 or 1/10 the size of the original, rather than less than 1/2?
- mikemol 16y agoThat greatly depends on the nature of the original data. Some forms of data are already compressed, such as images. Most websites apply gzip or DEFLATE (gzip is built on DEFLATE) compression serverside if your browser indicates it understands it.