2 ms·
I'm not sure this is true. Lossless compression algorithms exist, and HTML compresses almost absurdly well. A 20x compression rate would be quite impressive, b
by marginalia_nu 4y ago
I'm not sure this is true.
Lossless compression algorithms exist, and HTML compresses almost absurdly well. A 20x compression rate would be quite impressive, but I don't think it's impossible, since 10x compression is fairly common in this domain, and there are reasons to expect wikipedia to compress even better given the significant amount of markup on each page.
I've worked with cleaning up rendered wikipedia texts, and if you strip the mediawiki class names and cruft, the uncompressed HTML shrinks a lot, often from hundreds of kilobytes to tens of kilobytes. You can then probably compress that quite a lot as well since plain text compresses very well on its own.
Basically, mediawiki uses this much markup to say <li><a href="#Assessment">Assessment</a></li>:
<li id="toc-Assessment"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#Assessment">
<div class="vector-toc-text">
<span class="vector-toc-numb">6</span>Assessment</div>
</a>
<ul id="toc-Assessment-sublist" class="vector-toc-list">
</ul>
</li>
So yeah. 20x isn't too far fetched.
- CrampusDestrus 4y agoThat is not the point. There is a physical limit of how much you can compress stuff. If your LLM is significantly smaller than even the most optimized compression we can achieve, then it simply doesn't actually preserve all the information
- marginalia_nu 4y agoBe that as it may, 5Gb just isn't realistically significantly smaller than you could squeeze the text of wikipedia.
- bmacho 4y agoIndeed, if the theoretical limit to compress wikipedia is n GB, then you will need n GB at least to create the exact same copy of wikipedia. However you may create a jpg of it. (And you can then remove the artifacts, and even enhance it with AI, and you will get something better than the original wikipedia, to keep the image analogy.)