4 ms·

LLMs are not magic. If wikipedia takes up 100GB of space and your model trained on it is only 5GB big, there is NO WAY it will hold all the original informatio

LLMs are not magic. If wikipedia takes up 100GB of space and your model trained on it is only 5GB big, there is NO WAY it will hold all the original information.

HlessClaudesman 3y ago
Furthermore, LLMs are good at condensing information, but terrible at knowing which information will actually be important or true. This is an obvious problem in a life or death medical situations.
- Robotbeat 3y ago
  …which is also unsolved by any other system or by humans. It’s a fundamental constraint of reality.
marginalia_nu 3y ago
I'm not sure this is true. Lossless compression algorithms exist, and HTML compresses almost absurdly well. A 20x compression rate would be quite impressive, but I don't think it's impossible, since 10x compression is fairly common in this domain, and there are reasons to expect wikipedia to compress even better given the significant amount of markup on each page. I've worked with cleaning up rendered wikipedia texts, and if you strip the mediawiki class names and cruft, the uncompressed HTML shrinks a lot, often from hundreds of kilobytes to tens of kilobytes. You can then probably compress that quite a lot as well since plain text compresses very well on its own. Basically, mediawiki uses this much markup to say <li><a href="#Assessment">Assessment</a></li>: <li id="toc-Assessment" class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded"> <a class="vector-toc-link" href="#Assessment"> <div class="vector-toc-text"> <span class="vector-toc-numb">6</span>Assessment</div> </a> <ul id="toc-Assessment-sublist" class="vector-toc-list"> </ul> </li> So yeah. 20x isn't too far fetched.
- CrampusDestrus 3y ago
  That is not the point. There is a physical limit of how much you can compress stuff. If your LLM is significantly smaller than even the most optimized compression we can achieve, then it simply doesn't actually preserve all the information
  - marginalia_nu 3y ago
    Be that as it may, 5Gb just isn't realistically significantly smaller than you could squeeze the text of wikipedia.
  - bmacho 3y ago
    Indeed, if the theoretical limit to compress wikipedia is n GB, then you will need n GB at least to create the exact same copy of wikipedia. However you may create a jpg of it. (And you can then remove the artifacts, and even enhance it with AI, and you will get something better than the original wikipedia, to keep the image analogy.)
unnah 3y ago
NNCP, a state-of-the-art lossless text compressor based on a transformer neural network (i.e. containing similar structures to a LLM), can reach a compression ratio of 10.8% (0.867 bits per byte) on Wikipedia text: https://bellard.org/nncp/ https://bellard.org/nncp/ Compared to NNCP, a hypothetical LLM that "knows" every fact in the English Wikipedia, would not need to keep a lossless record of the entire text of Wikipedia, since many Wikipedia pages repeat facts that also occur on other pages, and instead of reproducing every word it would be enough to paraphrase the ideas. Thus the size of such a LLM could be much smaller than 10% of the size of Wikipedia. How much, it's hard to say...
- moffkalast 3y ago
  Yeah neural nets are amazing compression algorithms since the best way to get the gist of something is to actually understand it, then algorithmically generating the data. Like a calculator having a near infinite compression ratio because it can do math instead of keeping a table of all possible inputs and outputs. Same goes for human brains I suppose, we can only remember as much as we do because of the effective compression our understanding allows for.
T-A 3y ago
As of 21 September 2022, the size of the current version of all articles compressed is about 21.23 GB without media. https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia
- doodlesdev 3y ago
  By the way, note that is the Wikipedia in English which AFAIK is the largest one. For instance, Wikipedia in Portuguese weighs about 13gb with a few pictures in my Kiwix install.
  - fbdab103 3y ago
    Has Kiwix gotten any better? I tried to use it a few years ago, but the user experience was super rough. Fantastic idea, but there were lots of UI stalls and glitches that soured my trial.
    - doodlesdev 3y ago
      Honestly? The UX is dogshit lol, at least on the Android app, when you go and download something to find out if it has downloaded or not is a chore and there is no progress notification or indication it's doing anything at all. I'm considering sending some contributions whenever I have time again for OSS contributions. However, I find it usable enough that I've been using it quite a lot recently when I don't have internet access to read Wikipedia, StackOverflow and a few other things I downloaded.
grumbel 3y ago
It doesn't need to be lossless to be useful. A lossy LLM can just throw all the redundant stuff away and focus on the actual information in the text, instead of the text word by word. It wouldn't need to keep dozens of languages around for if it could translate on the fly. And it wouldn't need to keep a full text when it can just have the raw facts and generate the text on the fly. Not sure what the state of the art is with LLM text compression, but with StableDiffusion compression ratios of up to 155:1 have been achieved[1] compared to 10:1 for JPEG. [1] https://pub.towardsai.net/stable-diffusion-based-image-compresssion-6f1f0a399202 https://pub.towardsai.net/stable-diffusion-based-image-compr...
- SXX 3y ago
  Let's be honest disk space is cheap enough for people use actual data instead of LLM-generated. What people in poor countries need the least is learn from source that have tons of random facts of knowledge censored out since some western corporation decided they're unetical or disturbing.

LLMs are not magic. If wikipedia takes up 100GB of space and your model trained on it is only 5GB big, there is NO WAY it will hold all the original information.

HlessClaudesman 3y ago

Furthermore, LLMs are good at condensing information, but terrible at knowing which information will actually be important or true. This is an obvious problem in a life or death medical situations.

Robotbeat 3y ago
…which is also unsolved by any other system or by humans. It’s a fundamental constraint of reality.

marginalia_nu 3y ago

I'm not sure this is true. Lossless compression algorithms exist, and HTML compresses almost absurdly well. A 20x compression rate would be quite impressive, but I don't think it's impossible, since 10x compression is fairly common in this domain, and there are reasons to expect wikipedia to compress even better given the significant amount of markup on each page. I've worked with cleaning up rendered wikipedia texts, and if you strip the mediawiki class names and cruft, the uncompressed HTML shrinks a lot, often from hundreds of kilobytes to tens of kilobytes. You can then probably compress that quite a lot as well since plain text compresses very well on its own. Basically, mediawiki uses this much markup to say <li><a href="#Assessment">Assessment</a></li>: <li id="toc-Assessment" class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded"> <a class="vector-toc-link" href="#Assessment"> <div class="vector-toc-text"> <span class="vector-toc-numb">6</span>Assessment</div> </a> <ul id="toc-Assessment-sublist" class="vector-toc-list"> </ul> </li> So yeah. 20x isn't too far fetched.

CrampusDestrus 3y ago
That is not the point. There is a physical limit of how much you can compress stuff. If your LLM is significantly smaller than even the most optimized compression we can achieve, then it simply doesn't actually preserve all the information
- marginalia_nu 3y ago
  Be that as it may, 5Gb just isn't realistically significantly smaller than you could squeeze the text of wikipedia.
- bmacho 3y ago
  Indeed, if the theoretical limit to compress wikipedia is n GB, then you will need n GB at least to create the exact same copy of wikipedia. However you may create a jpg of it. (And you can then remove the artifacts, and even enhance it with AI, and you will get something better than the original wikipedia, to keep the image analogy.)

unnah 3y ago

NNCP, a state-of-the-art lossless text compressor based on a transformer neural network (i.e. containing similar structures to a LLM), can reach a compression ratio of 10.8% (0.867 bits per byte) on Wikipedia text: https://bellard.org/nncp/ https://bellard.org/nncp/ Compared to NNCP, a hypothetical LLM that "knows" every fact in the English Wikipedia, would not need to keep a lossless record of the entire text of Wikipedia, since many Wikipedia pages repeat facts that also occur on other pages, and instead of reproducing every word it would be enough to paraphrase the ideas. Thus the size of such a LLM could be much smaller than 10% of the size of Wikipedia. How much, it's hard to say...

moffkalast 3y ago
Yeah neural nets are amazing compression algorithms since the best way to get the gist of something is to actually understand it, then algorithmically generating the data. Like a calculator having a near infinite compression ratio because it can do math instead of keeping a table of all possible inputs and outputs. Same goes for human brains I suppose, we can only remember as much as we do because of the effective compression our understanding allows for.

T-A 3y ago

As of 21 September 2022, the size of the current version of all articles compressed is about 21.23 GB without media. https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia

doodlesdev 3y ago
By the way, note that is the Wikipedia in English which AFAIK is the largest one. For instance, Wikipedia in Portuguese weighs about 13gb with a few pictures in my Kiwix install.
- fbdab103 3y ago
  Has Kiwix gotten any better? I tried to use it a few years ago, but the user experience was super rough. Fantastic idea, but there were lots of UI stalls and glitches that soured my trial.
  - doodlesdev 3y ago
    Honestly? The UX is dogshit lol, at least on the Android app, when you go and download something to find out if it has downloaded or not is a chore and there is no progress notification or indication it's doing anything at all. I'm considering sending some contributions whenever I have time again for OSS contributions. However, I find it usable enough that I've been using it quite a lot recently when I don't have internet access to read Wikipedia, StackOverflow and a few other things I downloaded.

grumbel 3y ago

It doesn't need to be lossless to be useful. A lossy LLM can just throw all the redundant stuff away and focus on the actual information in the text, instead of the text word by word. It wouldn't need to keep dozens of languages around for if it could translate on the fly. And it wouldn't need to keep a full text when it can just have the raw facts and generate the text on the fly. Not sure what the state of the art is with LLM text compression, but with StableDiffusion compression ratios of up to 155:1 have been achieved[1] compared to 10:1 for JPEG. [1] https://pub.towardsai.net/stable-diffusion-based-image-compresssion-6f1f0a399202 https://pub.towardsai.net/stable-diffusion-based-image-compr...

SXX 3y ago
Let's be honest disk space is cheap enough for people use actual data instead of LLM-generated. What people in poor countries need the least is learn from source that have tons of random facts of knowledge censored out since some western corporation decided they're unetical or disturbing.