4 ms·
Fund SciHub! What I'd like to see is an IPFS feature that showed the "least shared" files in a set, so you could say "I want to help host the rarest 10 GB", fo
by StavrosK 9y ago
Fund SciHub!
What I'd like to see is an IPFS feature that showed the "least shared" files in a set, so you could say "I want to help host the rarest 10 GB", for example.
- toomuchtodo 9y agoYou can do this, as all of SciHub is available as torrents. Edit: you should be able to find the URLs with some googling, not going to post them here
- m-p-3 9y agoNow I'm curious, how much data would that be with the latest dump?
- snowpanda 9y agoLast time I checked it was between 40TB - 60TB for just the articles, not including the libgen books. At that time I calculated that it would cost about $1000 in refurbished 4TB external hard drives. Which is probably not the best way to store it. Or about 2000 Empty Blu-Rays if you want to make searching for an article a nightmare.
- deleted 9y ago[deleted]
- dredmorbius 9y agoSearch indices != primary storage. Ask Google about that.
- snowpanda 9y agoDo you mind expanding on that? What are you suggesting I search for on Google? I genuinely am trying to understand.
- Skuzzzy 9y agoForgive my brevity, typos, and poor writing as I'm writing this on my phone. He is stating that you would never actually search on your primary storage mediums. You would build your search index off the primary storage medium such that when looking up an article you do not search over the primary medium and only search the index and then once the article is found you read from said primary medium.
- dredmorbius 9y agoThis, yes.
- Redoubts 9y agoWell sure, but then you have to crawl though thousands of pieces of physical medium to load the file you want...
- sli 9y agoDo search indexes typically not include an identifier for a given resource? Seems to me that an index would be almost useless without them, as the match itself is more often than not far less useful than the document that contains it. Unless the information you need happens to be in the surrounding blurb. So yeah, I'm not sure why you'd have to "crawl" through physical media when a search index ought to tell you right where your match is located. Is that not the entire point of indexing and searching?
- dredmorbius 9y agoAn index maps a query to a specific item. Say, a file reference within a storage hierarchy or record within a database. (The concepts are fundamentally identical.)
- snowpanda 9y agoThank you for explaining :)
- dredmorbius 9y ago
- userbinator 9y ago...or about 10 LTO-7 tapes (6TB each --- "15TB" compressed, but I assume the article collection is already compressed), at slightly less cost for the media, and lot more for the drive; but with better reliability and even worse access times...
- m-p-3 9y agoOh damn, this is way bigger than I imagined.
- StavrosK 9y agoThe problem with that is that you can't serve content directly over HTTP from a torrent, whereas with IPFS you can.
- shpx 9y agohttps://webtorrent.io https://webtorrent.io
- deleted 9y ago[deleted]
- StavrosK 9y agohttps://www.eternum.io/ipfs/QmNmrNkyLygYDt9t5ptXkukPdpGSabLByG9tYKZ5kyTTkq https://www.eternum.io/ipfs/QmNmrNkyLygYDt9t5ptXkukPdpGSabLB... Which UX is better?
- toomuchtodo 9y agoMore importantly, webtorrent is a client side hack when clients should be using webseeding [1], which is baked into torrent clients that use libtorrent. [1] https://en.wikipedia.org/wiki/BitTorrent#Web_seeding https://en.wikipedia.org/wiki/BitTorrent#Web_seeding
- deleted 9y ago[deleted]
- kanzure 9y agoDo you have a complete copy of scihub? The seeders seem to be very slow. Please let me know -- email is in my HN profile -- I'll even pay for complete copies.
- amelius 9y agoIf you get a hold of it, please seed! :)
- ric2b 9y agoIPFS could be such an amazing thing for the academic world, imagine every new paper including an IPFS hash alongside the citations, so you could instantly jump to them while reading the pdf. It would guarantee that you'd be seeing the same version that was cited and the link would never break as long as at least one person kept sharing it. It would achieve the original goal of the world wide web as envisioned by Tim Berners-Lee, except even better.
- StavrosK 9y agoYes! I had exactly the same idea. I also love how anyone can reshare the file transparently, whereas with HTTP, if the original server doesn't have it any more, you have to manually hunt around for alternate links.