6 ms·

Imagine an LLM that is trained on public data like the internet archive and it is made available on these boxes without the internet, now we can have almost the

Imagine an LLM that is trained on public data like the internet archive and it is made available on these boxes without the internet, now we can have almost the entirety of human knowledge accessible on a single box. I know that LLMs sometime hallucinate and practical limits of computation power availability etc. but imagine how quickly we can spread the knowledge. It is almost like the printing press during the renaissance.

nibbleshifter 3y ago
In my experience LLM's hallucinate a lot. Often giving "plausible but entirely wrong" output that is hard to discern from factual, real output.
- eru 3y ago
  Yes. At the moment they are great for entertainment. But for factual information, you are better off just grabbing a copy of Wikipedia.
  - zamnos 3y ago
    Wikipedia's information often isn't collated how you want. Eg List the states where the biggest city is larger in population than the state capital? There isn't specifically an article for that. Sure you could take the List of United States cities by population article and assemble the list yourself, or you could just have ChatGPT do it for you.
    - eru 3y ago
      Yes, language models are already great for collating information, but they are still too unreliable to rely on.
CrampusDestrus 3y ago
LLMs are not magic. If wikipedia takes up 100GB of space and your model trained on it is only 5GB big, there is NO WAY it will hold all the original information.
- HlessClaudesman 3y ago
  Furthermore, LLMs are good at condensing information, but terrible at knowing which information will actually be important or true. This is an obvious problem in a life or death medical situations.
  - Robotbeat 3y ago
    …which is also unsolved by any other system or by humans. It’s a fundamental constraint of reality.
- marginalia_nu 3y ago
  I'm not sure this is true. Lossless compression algorithms exist, and HTML compresses almost absurdly well. A 20x compression rate would be quite impressive, but I don't think it's impossible, since 10x compression is fairly common in this domain, and there are reasons to expect wikipedia to compress even better given the significant amount of markup on each page. I've worked with cleaning up rendered wikipedia texts, and if you strip the mediawiki class names and cruft, the uncompressed HTML shrinks a lot, often from hundreds of kilobytes to tens of kilobytes. You can then probably compress that quite a lot as well since plain text compresses very well on its own. Basically, mediawiki uses this much markup to say <li><a href="#Assessment">Assessment</a></li>: <li id="toc-Assessment" class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded"> <a class="vector-toc-link" href="#Assessment"> <div class="vector-toc-text"> <span class="vector-toc-numb">6</span>Assessment</div> </a> <ul id="toc-Assessment-sublist" class="vector-toc-list"> </ul> </li> So yeah. 20x isn't too far fetched.
  - CrampusDestrus 3y ago
    That is not the point. There is a physical limit of how much you can compress stuff. If your LLM is significantly smaller than even the most optimized compression we can achieve, then it simply doesn't actually preserve all the information
    - 3y ago

nibbleshifter 3y ago

In my experience LLM's hallucinate a lot. Often giving "plausible but entirely wrong" output that is hard to discern from factual, real output.

eru 3y ago
Yes. At the moment they are great for entertainment. But for factual information, you are better off just grabbing a copy of Wikipedia.
- zamnos 3y ago
  Wikipedia's information often isn't collated how you want. Eg List the states where the biggest city is larger in population than the state capital? There isn't specifically an article for that. Sure you could take the List of United States cities by population article and assemble the list yourself, or you could just have ChatGPT do it for you.
  - eru 3y ago
    Yes, language models are already great for collating information, but they are still too unreliable to rely on.

CrampusDestrus 3y ago

LLMs are not magic. If wikipedia takes up 100GB of space and your model trained on it is only 5GB big, there is NO WAY it will hold all the original information.

HlessClaudesman 3y ago
Furthermore, LLMs are good at condensing information, but terrible at knowing which information will actually be important or true. This is an obvious problem in a life or death medical situations.
- Robotbeat 3y ago
  …which is also unsolved by any other system or by humans. It’s a fundamental constraint of reality.
marginalia_nu 3y ago
I'm not sure this is true. Lossless compression algorithms exist, and HTML compresses almost absurdly well. A 20x compression rate would be quite impressive, but I don't think it's impossible, since 10x compression is fairly common in this domain, and there are reasons to expect wikipedia to compress even better given the significant amount of markup on each page. I've worked with cleaning up rendered wikipedia texts, and if you strip the mediawiki class names and cruft, the uncompressed HTML shrinks a lot, often from hundreds of kilobytes to tens of kilobytes. You can then probably compress that quite a lot as well since plain text compresses very well on its own. Basically, mediawiki uses this much markup to say <li><a href="#Assessment">Assessment</a></li>: <li id="toc-Assessment" class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded"> <a class="vector-toc-link" href="#Assessment"> <div class="vector-toc-text"> <span class="vector-toc-numb">6</span>Assessment</div> </a> <ul id="toc-Assessment-sublist" class="vector-toc-list"> </ul> </li> So yeah. 20x isn't too far fetched.
- CrampusDestrus 3y ago
  That is not the point. There is a physical limit of how much you can compress stuff. If your LLM is significantly smaller than even the most optimized compression we can achieve, then it simply doesn't actually preserve all the information
  - 3y ago