3 ms·

> 5.6 Tb uncompressed, 280 Gb in bz2 compression format, 32 Gb in 7z compression format Wow, I didn't know 7z was this much better than bz2. Is this the expect

> 5.6 Tb uncompressed, 280 Gb in bz2 compression format, 32 Gb in 7z compression format Wow, I didn't know 7z was this much better than bz2. Is this the expected result, or is there something special with Wikipedia that plays to the strengths of 7z?

_delirium 17y ago
I'd guess it has to do mainly with 7z being able to use a larger block size, while bzip2's is 900kb; and possibly being able to do something better with large runs of repeated text. There are large articles with hundreds of revisions in a row that leave most of the content unchanged; [[George W. Bush]], for example, is around 180kb per revision, and is edited a lot, mostly with minor changes. So bzip2's block size means it can only squeeze 5 revisions per block: so in the degenerate case where 100 edits in a row changed only one character each, bz2 would be storing 20 or so basically identical copies of the article. IIRC from some tests a year or so ago, Wikipedia hasn't found any significant improvement from 7z over bz2 on the current-revisions-only dump, which looks more like just normal English text; that's why it doesn't bother to provide a separate 7z version of that. It seems to only be this pattern of [200kb article][almost the same 200kb article][almost the same 200kb article again] that 7z kills bz2 on.
- JoachimSchipper 17y ago
  Does Wikipedia really store every single revision of every single file? As in, not deltas? Why is it done that way?
  - _delirium 17y ago
    In the dump, I think for robustness and ease of extracting subsets. Robustness: Having to essentially play back a log to recover any particular revision increases a chance of something eventually getting corrupted, and so it's somewhat safer to avoid it in something intended to be archival. Ease of extracting subsets: For researchers, having the revisions be independent allows you to filter the XML dump through a SAX parser (or similar) to grab only revisions meeting particular criteria. If deltas were stored, you'd have to reconstruct those revisions from the deltas, which would make it really expensive to do things like, "I want to look at every article as it appeared at noon on April 1, 2007". In the live DB, I think just because it's cheaper to get a ton of storage, esp. for rarely-retrieved old revisions, than to add the overhead of computing deltas and applying them to reconstruct revisions. In particular, you'd have to compute a diff for every edit in that situation, whereas currently MediaWiki only computes diffs when a user requests to view one from the "history" tab, which is a tiny proportion of all edits.
    - JoachimSchipper 17y ago
      I understand it's simpler to store everything, and simplicity _is_ a virtue; but one could store the current revision plus deltas (and perhaps a few intermediate revisions for oft-edited articles), and obtain performance at least as good as in the current case. It would also save lots of space.
- res0nat0r 17y ago
  It would be cool to see how the 7z archive compares to rzip, which (I think) might have the largest input size window of them all (up to 900MB). This software was written by Andrew Tridgell of rsync fame. It's major drawback is that it cant work on stdio and uses tons of RAM.

_delirium 17y ago

I'd guess it has to do mainly with 7z being able to use a larger block size, while bzip2's is 900kb; and possibly being able to do something better with large runs of repeated text. There are large articles with hundreds of revisions in a row that leave most of the content unchanged; [[George W. Bush]], for example, is around 180kb per revision, and is edited a lot, mostly with minor changes. So bzip2's block size means it can only squeeze 5 revisions per block: so in the degenerate case where 100 edits in a row changed only one character each, bz2 would be storing 20 or so basically identical copies of the article. IIRC from some tests a year or so ago, Wikipedia hasn't found any significant improvement from 7z over bz2 on the current-revisions-only dump, which looks more like just normal English text; that's why it doesn't bother to provide a separate 7z version of that. It seems to only be this pattern of [200kb article][almost the same 200kb article][almost the same 200kb article again] that 7z kills bz2 on.

JoachimSchipper 17y ago
Does Wikipedia really store every single revision of every single file? As in, not deltas? Why is it done that way?
- _delirium 17y ago
  In the dump, I think for robustness and ease of extracting subsets. Robustness: Having to essentially play back a log to recover any particular revision increases a chance of something eventually getting corrupted, and so it's somewhat safer to avoid it in something intended to be archival. Ease of extracting subsets: For researchers, having the revisions be independent allows you to filter the XML dump through a SAX parser (or similar) to grab only revisions meeting particular criteria. If deltas were stored, you'd have to reconstruct those revisions from the deltas, which would make it really expensive to do things like, "I want to look at every article as it appeared at noon on April 1, 2007". In the live DB, I think just because it's cheaper to get a ton of storage, esp. for rarely-retrieved old revisions, than to add the overhead of computing deltas and applying them to reconstruct revisions. In particular, you'd have to compute a diff for every edit in that situation, whereas currently MediaWiki only computes diffs when a user requests to view one from the "history" tab, which is a tiny proportion of all edits.
  - JoachimSchipper 17y ago
    I understand it's simpler to store everything, and simplicity _is_ a virtue; but one could store the current revision plus deltas (and perhaps a few intermediate revisions for oft-edited articles), and obtain performance at least as good as in the current case. It would also save lots of space.
res0nat0r 17y ago
It would be cool to see how the 7z archive compares to rzip, which (I think) might have the largest input size window of them all (up to 900MB). This software was written by Andrew Tridgell of rsync fame. It's major drawback is that it cant work on stdio and uses tons of RAM.