3 ms·
One community that does use this technique (although the index is in a separate file) is the web archiving community's WARC file format. The Internet Archive Wa
by greglindahl 10y ago
One community that does use this technique (although the index is in a separate file) is the web archiving community's WARC file format. The Internet Archive Wayback Machine has ~ 13 petabytes of web archive stored that way, and you use it every time you look at a webpage in the Wayback. The file format is what you'd expect: each webpage is gzipped separately, the output is catted together, and the index stores the offset and length of each individual compressed webpage.
https://en.wikipedia.org/wiki/Web_ARChive https://en.wikipedia.org/wiki/Web_ARChive