3 ms·
Easier than that would just be downloading the torrent of all of Reddit through Sept 2023. https://academictorrents.com/details/89d24ff9d5fbc1efcdaf9d7689d72b7
by monocasa 3y ago
Easier than that would just be downloading the torrent of all of Reddit through Sept 2023.
https://academictorrents.com/details/89d24ff9d5fbc1efcdaf9d7689d72b7548f699fc https://academictorrents.com/details/89d24ff9d5fbc1efcdaf9d7...
- q7xvh97o2pDhNrh 3y agoThat's fascinating that the total size is so tiny — only 2.4 TB‽ I assume this must be only the text portion, and heavily compressed?
- lxgr 3y agoText really doesn't take up that much space, and in addition it compresses pretty well. The entire English language Wikipedia is only around 60GB in a format that can be readily searched and randomly accessed (ZIM), for example: https://kiwix.org/ https://kiwix.org/
- lmm 3y agoDoes Kiwix actually work? I see people hyping it here but I could never get it to actually, y'know, download the file and display the wikipedia on my phone.
- lxgr 3y agoIt works perfectly for me, both on iOS and macOS.
- pc2slow4webpack 3y agoJust downloaded. Doesn't seem to want to download Wikipedia on my phone, it says "detecting if filesystem supports 4gb files"
- vatueil 3y agoKiwix worked for me. IIRC there may be difficulties opening an archive that was downloaded outside of the mobile app, but archives downloaded in-app were fine. For the mobile app I used one of the smaller Wikipedia subsets, since I didn't want to take up too much space on my phone. The full offline Wikipedia download is saved to my laptop.
- PaulDavisThe1st 3y agoThe question is: if you then added all of Usenet before, say, 1992, would the effective intelligence of the trained LLM go up or down?
- yjftsjthsd-h 3y agoI can't speak to intelligence, but the result would be ignorant in a meaningful way.