6 ms·
Millions of individual torrents is not a great solution. Keeping them all seeded is basically impossible unless they run a seed for each one, at which point th
by jandrese 5y ago
Millions of individual torrents is not a great solution. Keeping them all seeded is basically impossible unless they run a seed for each one, at which point they might as well just host the files. Plus you'll never get the economy of scale that makes BitTorrent really shine.
When you have a whole lot of tiny files that people will generally only want one or two of there isn't much better than a plain old website.
A torrent that hosts all of the papers could be useful for people who want to make sure the data can't be lost by a single police raid.
- contravariant 5y agoThere was that project some guy posted a while back that used a combination of sqlite and partial downloads to enable searches on a database before it was downloaded all the way. If you can fit PDFs somewhere into that you'd be golden. Or just use IPFS I suppose.
- hkt 5y agoIsn't that essentially mapreduce? Either way, interesting and I'd love to see the link.
- vorticalbox 5y agoI believe this is the project mentioned https://github.com/lmatteis/torrent-net https://github.com/lmatteis/torrent-net
- tmkadamcz 5y agoThis looks like it could be a good approach.
- contravariant 5y agoThat one looks familiar. Though apparently the same thing has been tried in several different ways going by the replies I got.
- ric2b 5y agoIt's this one: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database...
- o8r3oFTZPE 5y ago"There was that project some guy posted a while back that used a combination of sqlite and partial downloads to enable searches on a database before it was downloaded all the way." https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent
- jagged-chisel 5y agothis is the one: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database... HN submission: https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630
- o8r3oFTZPE 5y agoThis is the original. Then came https://github.com/lmatteis/net-torrent https://github.com/lmatteis/net-torrent and later one written in Javascript, inspired by net-torrent.
- divbzero 5y agoIPFS would face a similar challenge as the “keep torrents seeded” problem mentioned by GP. Wouldn’t there be risk to peers who host the PDFs?
- Natsu 5y agoI sort of feel like there should be some way to use some kind of construct to get people to seed things so that others seed things for them, but I haven't seen that invented yet.
- zolland 5y agoI think seed ratios and seed time (mostly used by private trackers) attempt to solve this problem.
- miloignis 5y agoBeen a while since I've looked at them, but IPFS with FileCoin and Ethereum Swarm had that kind of goal. It might be beneficial to create something like what you describe without any cryptocurrency association though, and I've been mulling over possibilities for distributed systems that are inherently currency-less to avoid all of the scams that cryptocurrency attracts.
- Taek 5y agoThe leader in that space is Skynet, which basically is like IPFS + Filecoin but also has dynamic elements to it, and a lot better performance + reliability. Cryptocurrency is helpful because it allows you to incentivize people to hold the data. If you don't have cryptocurrency, you're basically dependent on altruism to keep data alive (like bittorrent, or ipfs without filecoin). Popular files do okay for a while, but even popular files struggle to provide good uptime and reliability after a few years. On an incentivized network like Sia or Filecoin, you can have high performance and reliability without ever needing any users to keep their machines online and seeding.
- 5y ago
- posterboy 5y agoa plain old website or a publishing house with distribution services and syndication attached, but for a sane price. "a whole lot of tiny files" severely underestimates the scale at work. Libgen's coverage is relatively shallow, and pdf books tend to be huge, at least for older material. Scihub piggy backs on the publishers, so that's your reference. syndication, syndicate, quite apt don't you think? Libraries that coluded with the publishers and accepted the pricing must have been a huge part of the problem, at least historically. Now you know there's only one way out of a mafia.
- einpoklum 5y agoIf the sane price is an optional "Donate to keep this site going" link, then ok. But only free access, without authentication or payment, to scientific papers, is sane. IMHO.
- munk-a 5y agoMight this be a case where the best resolution would be to have the government (which is at least partially funding nearly all of these papers) step in and add a ledger of papers as a proof of investment? The cost of maintaining a free and open DB of scientific advances and publications would be so incredibly insignificant compared to both the value and the continued investment in those advancements.
- einpoklum 5y agoWell, some research venues (and publication venues) are not government-funded, and even if they are indirectly government funded, it's more of a sophistry than something which would make publishers hand over copies of the papers. Also, a per-government ledger would not be super-practicable. But if, say, the US, the EU and China would agree on something like this, and implement it, and have a common ledger, then it would not be some a big leap to make it properly international. Maybe even UN-based. That's a pretty big "if" though.
- jpeloquin 5y ago> Might this be a case where the best resolution would be to have the government (which is at least partially funding nearly all of these papers) step in and add a ledger of papers as a proof of investment? I feel that we're halfway there already and are gaining ground. Does Pubmed Central [0] (a government-hosted open access repository for NIH-funded work) count as a "ledger" like you're referring to? The NSF's site does a good job of explaining current US open access policy [1]. There are occasional attempts to expand the open access mandate by legislation, such as FASTR [2]. A hypothetical expansion of the open access mandate to apply to all works from /institutions/ that receive indirect costs, not just individual projects that receive direct costs, would open things up even more. [0] https://www.ncbi.nlm.nih.gov/pmc/ https://www.ncbi.nlm.nih.gov/pmc/ [1] https://www.nsf.gov/pubs/2016/nsf16009/nsf16009.jsp#q1 https://www.nsf.gov/pubs/2016/nsf16009/nsf16009.jsp#q1 [2] https://sparcopen.org/our-work/fastr/ https://sparcopen.org/our-work/fastr/
- tmkadamcz 5y agoThere are already torrents of the archives. But supposing scihub was taken down it's pretty non trivial to get from the archive back to a working site with search functionality. For one thing, none of Sci-Hub's code is available.
- derefr 5y agoSeems like what should be in each torrent is a virtual appliance preloaded with one shard of the data, where that virtual appliance has a pre-baked index for searching just that shard's data. Then one more torrent for a small search-coordinator appliance that fans your search query out to all N shard appliances.
- dredmorbius 5y agoWhat documents (books, scientific articles) benefit from specifically is a number of highly consistent, highly accurate identifiers: DOI (scientific articles), ISBN (published books), and others (OCLC identifier, Library of Congress Catalogue Number, etc.) With the addition of hashsums (even MD5 ad SHA1, though longer and more robust hashsums are preferred), a pretty reliable archive of content can be made. It's a curious case where increased legibility seems to be breaking rather than creating a gatekeeper monopoly. I've been interested in the notion of more reliable content-based identifiers or fingerprints themselves, though I've found little reliable reference on this. Ngram tuples of 4-5 words are often sufficient to identify a work, particularly if a selection of several are made. Aggreeing on which tuples to use, how many, and how to account for potential noise / variations (special characters, whitespace variance, OCR inaccuracy) is also a stumbling point.
- spicybright 5y agoWhy map anything to words for strict identification? Words and language are very error prone vs an id number or hash.
- mandelken 5y agoProbably because for written text the words identify the content while the hash relates more to the digital carrier format (pdf vs epub) and id number can change between publications, countries, etc.
- dredmorbius 5y agoBingo. And to drag in metadata, it may: - Not be present. - Be inaccurately applied to the correct work (metadata say the work is different, work is in fact related/same). - Be inaccurately applied to the wrong work (metadata say the works are the same/related, they are not).
- dredmorbius 5y agoIt's a bit of an itch I've been scratching for a few years. Most especially, given two or more instances of what you suspect to be the same or a substantively similar work, how can you assess this in a robust and format-independent manner, programmatically? For works with well-formed metadata, this isn't an issue. For identical duplicate copies of the same file, a hash is effective. But for the circumstance most often encountered in reality --- different forms and formats derived from different sources but containing substantially the same work --- there is no simple solution of which I'm aware. As examples, say you have a reference document The Reference Document. How do I determine that: - An ACSCII-only textfile - Markdown, HTML, DocBook, and LaTeX sources - PDF, MS Word (which version), PS, DJVU, ePub, or .mobi files (sling any other formats you care to mention). - Hardbound and paperback physical copies - Scans made from the same or different physical books or instances, versions, and/or translations. - Audiobooks based on a work. By the same or different readers. - Dramatic performances, films, video series, comic-book adaptations, etc., of a work. (Say: Hamlet or Romeo and Juliet. What is the relationship to "West Side Story" (and which version), or Pyramus and Thisbe?) - Re-typed or OCRed text ... all refer to the same work? How do you define "work"? How do you define "differences between works"? How do you distinguish intentional, accidental, and incidental differences between instances? (Say: translations, errata, corrections, additions for the one, transcription errors for the second, and scanning or rendering artefacts for the third.) If you're working in an environment in which instances of works come from different sources with different provenances, these questions arise. At least some of these questions are prominent in library science itself. It's the technical mapping of digitised formats I'm focusing on most closely, so the physical instantiations aren't as critical here, though the presumption is that these could be converted to some machine-readable form. In bibliographic / library science, the term is "work, expression, manifestation" https://www.loc.gov/marc/marbi/2011/2011-dp03.html https://www.loc.gov/marc/marbi/2011/2011-dp03.html
- sildur 5y agoMaybe we can create a freesite, on Freenet.
- mark-wagner 5y agoMaybe Usenet? It already support massive copyright infringement yet it is still around.
- fabioyy 5y agoonly if it was possible to use chia to store content.. it would be a game changer
- dpacmittal 5y agoBitTorrent does allow you to download a single file from a torrent though. You could have torrent for each month, and a client which allows you to search inside these torrents and download only the files you need.