5 ms·
For reference (according to Google): > The English Wikipedia, as of June 26, 2025, contains over 7 million articles and 63 million pages. The text content alon
by nico 1y ago
For reference (according to Google):
> The English Wikipedia, as of June 26, 2025, contains over 7 million articles and 63 million pages. The text content alone is approximately 156 GB, according to Wikipedia's statistics page. When including all revisions, the total size of the database is roughly 26 terabytes (26,455 GB)
- sharkjacobs 1y agobetter point of reference might be pages-articles-multistream.xml.bz2 (current pages without edit/revision history, no talk pages, no user pages) which is 20GB https://en.wikipedia.org/wiki/Wikipedia:Database_download#Where_do_I_get_it https://en.wikipedia.org/wiki/Wikipedia:Database_download#Wh...?
- inopinatus 1y agothis is a much more deserving and reliable candidate for any labels regarding the breadth of human knowledge.
- wahnfrieden 1y agoit barely touches the surface
- inopinatus 1y agoregarding depth, not breadth, certainly
- mapt 1y agoWhat happens if you ask this 8gb model "Compose a realistic Wikipedia-style page on the Pokemon named Charizard"? How close does it come?
- pcrh 1y agoWikipedia itself describes its size as ~25GB without media [0]. And it's probably more accurate and with broader coverage in multiple languages compared to the LLM downloaded by the GP. https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia
- pessimizer 1y agoReally? I'd assume that an LLM would deduplicate Wikipedia into something much smaller than 25GB. That's its only job.
- crazygringo 1y ago> That's its only job. The vast, vast majority of LLM knowledge is not found in Wikipedia. It is definitely not its only job.
- Tostino 1y agoWhen trained on next word prediction with the standard loss function, by definition it is it's only job.