4 ms·
Wikipedia itself describes its size as ~25GB without media [0]. And it's probably more accurate and with broader coverage in multiple languages compared to the
by pcrh 1y ago
Wikipedia itself describes its size as ~25GB without media [0]. And it's probably more accurate and with broader coverage in multiple languages compared to the LLM downloaded by the GP.
https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia
- pessimizer 1y agoReally? I'd assume that an LLM would deduplicate Wikipedia into something much smaller than 25GB. That's its only job.
- crazygringo 1y ago> That's its only job. The vast, vast majority of LLM knowledge is not found in Wikipedia. It is definitely not its only job.
- Tostino 1y agoWhen trained on next word prediction with the standard loss function, by definition it is it's only job.