3 ms·
The parent comment wants to efficiently mirror the content to prevent it from disappearing in the event of a successful take down operation. This can be done by
by edraferi 11y ago
The parent comment wants to efficiently mirror the content to prevent it from disappearing in the event of a successful take down operation. This can be done by scraping an API, but bulk methods are faster and less likely to degrade service for other users.
- xvilka 11y agoMay be even using something like DHT on top of WebRTC could allow to upload/download those papers easily.
- femto 11y agoThere's a thumping amount of data there. 485 torrents, each torrent containing 100 zips of around 500MB each, each zip containing 1000 papers. Total = 500MB * 100 * 485 ~ 25TB. If your have a 10Mbit/s link into your house, it's 280 days of continuous downloading. New torrents seem to be appearing at the rate of 14 per month = 700GB/month = 2.7Mbit/s. For the majority of people, I'd say it is beyond any one person being able to mirror the lot. It seems like an application for a Freenet type system, but one where the files are guaranteed to never be deleted, even if they are rarely accessed.
- harperlee 11y agoI would assume that most of that is due to PDFs weigthing a lot more than their content. If someone wants to donate IT skills and coding time (this is a thing that I've read several times in HN), perhaps it would be great that someone puts together a system that takes a PDF, OCRs it when necessary, detects images and graphs, and uses all of that to create a light version of the document (html/latex/markdown/whatever); and then puts all of those TB through that tool. The reduction in storage and network costs might be a lot more than a standard donation!
- kwhitefoot 11y agoIf enough people keep the torrents alive no one person would need to mirror all of it, certainly not all at once.