3 ms·
The winds may be changing, but those who don't fear resorting to piracy will always be sailing smoothly. We will just have more walled gardens and more illegal
by sigmoid10 2y ago
The winds may be changing, but those who don't fear resorting to piracy will always be sailing smoothly. We will just have more walled gardens and more illegal offerings, leaving normal people stranded and thirsty.
- kelsey98765431 2y agoUnfortunately things like a 500gb dump of sacredtexts website is not often found on torrent trackers or other 'warez' sites. Anna's is pretty great for written material that has been published offline, but even the wayback machine and archive.org have limited full scrapes aside from the published ones by the kiwix team.
- sigmoid10 2y agoSuch megadumps are only really interesting for people who train LLMs anyways. No way you can ever consume that for personal development. Individual sources are much more prevalent and useful to everyday life.
- mewpmewp2 2y agoIt would be interesting for a personal project to do some search and summarisation project, e.g. with RAG. Get the data, make embeddings, categorize and tag the data, then do search and have LLM summarize findings.
- kelsey98765431 2y ago:)