3 ms·
Just about 2 years ago, a long dormant project surged back to life becoming one of the best crawlers out there. Zimit was originally made to scrape mediawiki si
by kelsey98765431 2y ago
Just about 2 years ago, a long dormant project surged back to life becoming one of the best crawlers out there. Zimit was originally made to scrape mediawiki sites but it now is able to crawl literally anything into an offline archive file. I have been trying to grab the data i am guessing will shortly be put under much stricter anti scraping controls soon, and I am not the only one hoarding data. The winds are blowing towards a closed internet faster than I have ever seen in my life. Get it while you can.
- sigmoid10 2y agoThe winds may be changing, but those who don't fear resorting to piracy will always be sailing smoothly. We will just have more walled gardens and more illegal offerings, leaving normal people stranded and thirsty.
- kelsey98765431 2y agoUnfortunately things like a 500gb dump of sacredtexts website is not often found on torrent trackers or other 'warez' sites. Anna's is pretty great for written material that has been published offline, but even the wayback machine and archive.org have limited full scrapes aside from the published ones by the kiwix team.
- sigmoid10 2y agoSuch megadumps are only really interesting for people who train LLMs anyways. No way you can ever consume that for personal development. Individual sources are much more prevalent and useful to everyday life.
- mewpmewp2 2y agoIt would be interesting for a personal project to do some search and summarisation project, e.g. with RAG. Get the data, make embeddings, categorize and tag the data, then do search and have LLM summarize findings.
- kelsey98765431 2y ago:)
- mewpmewp2 2y agoWhat is the size of your storage?
- kelsey98765431 2y agoAbout 50tb on my lan.