This is awesome. If only we could share this huge dump of PDF files as a more structured format, perhaps using SQLite [1, 2], we could search through the torren

This is awesome. If only we could share this huge dump of PDF files as a more structured format, perhaps using SQLite [1, 2], we could search through the torrents without having to wait for all of them to download beforehand. Although I guess the fact of having to "download them all beforehand" forces the data to be spread across various computers, hence increases availability of the data. One idea I had regarding this is perhaps structuring the contents of torrents as "append-only binary trees". So as new dumps are released every month, one can simply start downloading the torrent and has "search capabilities" for new data as well. 1. https://github.com/lmatteis/torrent-net https://github.com/lmatteis/torrent-net 2. https://www.youtube.com/watch?v=EKttt8PYu5M&feature=youtu.be https://www.youtube.com/watch?v=EKttt8PYu5M&feature=youtu.be