11 ms·

By the way, Sci-Hub has stopped adding new articles to the database for a few months now (background: https://www.reddit.com/r/scihub/comments/mk46x4/scihub_v_e

By the way, Sci-Hub has stopped adding new articles to the database for a few months now (background: https://www.reddit.com/r/scihub/comments/mk46x4/scihub_v_elsevier_wiley_acs_can_scihub_be_shut/ https://www.reddit.com/r/scihub/comments/mk46x4/scihub_v_els...). It would be great to develop a truly decentralised solution. Having a database of individual torrent links for each paper might be a start.

HideousKojima 5y ago
Torrents are great and all but it's dependent on people seeding them, sci-hub/libgen is great because you don't have to worry about a download suddenly breaking because no one is seeding
- kortilla 5y ago
  But they could just always be a seeder. Doesn’t that have the upsides of the existing solution plus resiliency?
  - whimsicalism 5y ago
    Isn't scihub/libgen already backed by torrents? I'm confused.
    - ta988 5y ago
      Nope it was mostly direct download for the endusers with some ipfs in some places.
deleted 5y ago
[deleted]
jandrese 5y ago
Millions of individual torrents is not a great solution. Keeping them all seeded is basically impossible unless they run a seed for each one, at which point they might as well just host the files. Plus you'll never get the economy of scale that makes BitTorrent really shine. When you have a whole lot of tiny files that people will generally only want one or two of there isn't much better than a plain old website. A torrent that hosts all of the papers could be useful for people who want to make sure the data can't be lost by a single police raid.
- contravariant 5y ago
  There was that project some guy posted a while back that used a combination of sqlite and partial downloads to enable searches on a database before it was downloaded all the way. If you can fit PDFs somewhere into that you'd be golden. Or just use IPFS I suppose.
  - hkt 5y ago
    Isn't that essentially mapreduce? Either way, interesting and I'd love to see the link.
    - vorticalbox 5y ago
      I believe this is the project mentioned https://github.com/lmatteis/torrent-net https://github.com/lmatteis/torrent-net
      - tmkadamcz 5y ago
        This looks like it could be a good approach.
      - contravariant 5y ago
        That one looks familiar. Though apparently the same thing has been tried in several different ways going by the replies I got.
    - ric2b 5y ago
      It's this one: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database...
  - o8r3oFTZPE 5y ago
    "There was that project some guy posted a while back that used a combination of sqlite and partial downloads to enable searches on a database before it was downloaded all the way." https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent
andyxor 5y ago
IPFS seems like a perfect fit for this and some of the scihub torrents are already in IPFS, but it's not an anonymous network. IPFS via the DHT tells the network of your whole network topology, including internal address you may have, and VPN endpoints too. It's all public by design as they don't want to associate IPFS with piracy per one of their developers. this thread has some discussions on the alternatives https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_mission_for_scihub_and_open_science_we_are/ https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_...
- whimsicalism 5y ago
  Isn't scihub already on IPFS?
  - andyxor 5y ago
    some torrent files are archived there, but i don't think scihub is serving the pdfs from IPFS, they likely use private storage network. I believe libgen.fun which is a new (official) libgen mirror is running fully on ipfs, and it serves some scientific papers, but I wasn't able to search by DOI or title there, looks like it redirects to scihub, also there is no index of the papers on IPFS. Edit: this doc talks about scihub+ipfs (it was created by the leader of Scihub rescue effort on Reddit, /u/shrine): https://freeread.org/ipfs/ https://freeread.org/ipfs/
- vvll 5y ago
  Can files be taken down off ipfs? There was a fairy widely circulated link that had all the IEC and ANSI standards on there that has since been taken down.
- kodablah 5y ago
  You could use libp2p's DHT over Tor (I did a poc of this long ago, and the situation's only improved). Combined with other libp2p/IPFS components, you can essentially have a private IPFS over onion services (not to be confused with accessing the existing IPFS network via Tor exit nodes).
- brunoqc 5y ago
Azrael3000 5y ago
Thanks for the background link. I did not know about that and it's a good incentive to donate them some money for the legal battle. TL;DR of the link: No more uploads to support a court case in India which SciHub might win and thus establish a legal basis for operation in the biggest democracy.
- andyxor 5y ago
  when donating bitcoin make sure to get the address from the official scihub mirrors, which are currently sci-hub.do, sci-hub.st or sci-hub.se. there are some unaffiliated "mirrors" that only redirect to scihub but list their own bitcoin address for donations, so beware. /r/scihub on reddit keeps track of this https://www.reddit.com/r/scihub/wiki/index https://www.reddit.com/r/scihub/wiki/index
- lowkey 5y ago
  Sci-hub is such a great example of a clear and compelling use-case for Bitcoin. Bitcoin is censorship-resistant money that doesn't rely on countries, laws, central bankers or politicians. The US dollar cannot be used for purposes not aligned with the US government. Sometimes ideas that the US Government doesn't agree with can be useful (e.g. Wikileaks, Sci-hub.) When I hear complaints that Bitcoin has no use except for speculation, I think of Sci-hub, Wikileaks and other organizations that may be bad for the interests of the US government but may be good for mankind.
  - hyperbovine 5y ago
    It's a censorship resistant technology that also indelibly records, publicly, every transaction you ever participated in. Talk about a double-edged sword...
    - lowkey 5y ago
      Agree, it's an interesting trade-off: Completely private if you can use an anonymous address but completely traceable if the address is identified. I wonder if Satoshi intended it that way or if he would have preferred the greater anonymity of Monero.
keithnz 5y ago
from what I understand, the Authors are still free to send you their papers. So perhaps the simplest decentralized system is for each author to run an automatic email request system and have feeds / aggregators of papers titles/abstracts with author emails to make it easy to get the papers you want?
- ta988 5y ago
  That would only work for a tiny tiny intersection of : 1) technical enough people to install and maintain that 2) with an email address still working (if there is one on the paper!) 3) still alive that's probably a really small set of scientists.
TZubiri 5y ago
I'm growning skeptical of the idea that every science text should be free. As a community that values privacy, it's hypocritical that these widespread leaking is so well accepted. The fact that scihub needs to skip domain names every couple of months and that ISPs start blocking the website, looks to confirm my theory, rather than imply some worldwide conspiracy.
- ta988 5y ago
  How is that a leak? Only the publishing companies are making money. Authors, reviewers and often editors all work for free.
  - TZubiri 5y ago
    I don't think they work for free, they have a salary, usually paid by universities I think.
- agucova 5y ago
  What does this have to do with privacy?
  - TZubiri 5y ago
    Publishers want to keep the papers relatively private.
    - ric2b 5y ago
      Except from literally anyone that pays them a monthly fee? That's not privacy.
Taek 5y ago
You can build something like this on Skynet. Links on Skynet (called skylinks) can point to hashes of data (similar to IPFS)or can point to a pubkey. If the data points to a pubkey, it can be tweaked repeatedly and people with the old links will see the new data. You can also point to full webapps. So for example you could have a pubkey point to a webapp that then loads the database from a list of moderators who update the data under their pubkeys to point to the latest version of scihub. Then you can even have different moderators curate articles of different subjects, and the webapp can combine everything together. All the data is stored on Sia, which is a decentralized cloud storage platform. Skynet allows anyone to upload or download data from Sia directly from the web, no need for extensions or custom software. The Sia network handles what IPFS calls "seeding", so contributors don't need to worry about leaving their machines on.
deleted 5y ago
[deleted]
i5heu 5y ago
This is where IPFS shines. In fact, there is already a ipfs mirror of scihub.
josephagoss 5y ago
This might be a good use case for Arweave.
nandhinianand 5y ago
Wouldn't IPFS(https://ipfs.io/ https://ipfs.io/) work well?..

HideousKojima 5y ago

Torrents are great and all but it's dependent on people seeding them, sci-hub/libgen is great because you don't have to worry about a download suddenly breaking because no one is seeding

kortilla 5y ago
But they could just always be a seeder. Doesn’t that have the upsides of the existing solution plus resiliency?
- whimsicalism 5y ago
  Isn't scihub/libgen already backed by torrents? I'm confused.
  - ta988 5y ago
    Nope it was mostly direct download for the endusers with some ipfs in some places.

deleted 5y ago

[deleted]

jandrese 5y ago

Millions of individual torrents is not a great solution. Keeping them all seeded is basically impossible unless they run a seed for each one, at which point they might as well just host the files. Plus you'll never get the economy of scale that makes BitTorrent really shine. When you have a whole lot of tiny files that people will generally only want one or two of there isn't much better than a plain old website. A torrent that hosts all of the papers could be useful for people who want to make sure the data can't be lost by a single police raid.

contravariant 5y ago
There was that project some guy posted a while back that used a combination of sqlite and partial downloads to enable searches on a database before it was downloaded all the way. If you can fit PDFs somewhere into that you'd be golden. Or just use IPFS I suppose.
- hkt 5y ago
  Isn't that essentially mapreduce? Either way, interesting and I'd love to see the link.
  - vorticalbox 5y ago
    I believe this is the project mentioned https://github.com/lmatteis/torrent-net https://github.com/lmatteis/torrent-net
    - tmkadamcz 5y ago
      This looks like it could be a good approach.
    - contravariant 5y ago
      That one looks familiar. Though apparently the same thing has been tried in several different ways going by the replies I got.
  - ric2b 5y ago
    It's this one: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database...
- o8r3oFTZPE 5y ago
  "There was that project some guy posted a while back that used a combination of sqlite and partial downloads to enable searches on a database before it was downloaded all the way." https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent

andyxor 5y ago

IPFS seems like a perfect fit for this and some of the scihub torrents are already in IPFS, but it's not an anonymous network. IPFS via the DHT tells the network of your whole network topology, including internal address you may have, and VPN endpoints too. It's all public by design as they don't want to associate IPFS with piracy per one of their developers. this thread has some discussions on the alternatives https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_mission_for_scihub_and_open_science_we_are/ https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_...

whimsicalism 5y ago
Isn't scihub already on IPFS?
- andyxor 5y ago
  some torrent files are archived there, but i don't think scihub is serving the pdfs from IPFS, they likely use private storage network. I believe libgen.fun which is a new (official) libgen mirror is running fully on ipfs, and it serves some scientific papers, but I wasn't able to search by DOI or title there, looks like it redirects to scihub, also there is no index of the papers on IPFS. Edit: this doc talks about scihub+ipfs (it was created by the leader of Scihub rescue effort on Reddit, /u/shrine): https://freeread.org/ipfs/ https://freeread.org/ipfs/
vvll 5y ago
Can files be taken down off ipfs? There was a fairy widely circulated link that had all the IEC and ANSI standards on there that has since been taken down.
kodablah 5y ago
You could use libp2p's DHT over Tor (I did a poc of this long ago, and the situation's only improved). Combined with other libp2p/IPFS components, you can essentially have a private IPFS over onion services (not to be confused with accessing the existing IPFS network via Tor exit nodes).
brunoqc 5y ago

Azrael3000 5y ago

Thanks for the background link. I did not know about that and it's a good incentive to donate them some money for the legal battle. TL;DR of the link: No more uploads to support a court case in India which SciHub might win and thus establish a legal basis for operation in the biggest democracy.

andyxor 5y ago
when donating bitcoin make sure to get the address from the official scihub mirrors, which are currently sci-hub.do, sci-hub.st or sci-hub.se. there are some unaffiliated "mirrors" that only redirect to scihub but list their own bitcoin address for donations, so beware. /r/scihub on reddit keeps track of this https://www.reddit.com/r/scihub/wiki/index https://www.reddit.com/r/scihub/wiki/index
lowkey 5y ago
Sci-hub is such a great example of a clear and compelling use-case for Bitcoin. Bitcoin is censorship-resistant money that doesn't rely on countries, laws, central bankers or politicians. The US dollar cannot be used for purposes not aligned with the US government. Sometimes ideas that the US Government doesn't agree with can be useful (e.g. Wikileaks, Sci-hub.) When I hear complaints that Bitcoin has no use except for speculation, I think of Sci-hub, Wikileaks and other organizations that may be bad for the interests of the US government but may be good for mankind.
- hyperbovine 5y ago
  It's a censorship resistant technology that also indelibly records, publicly, every transaction you ever participated in. Talk about a double-edged sword...
  - lowkey 5y ago
    Agree, it's an interesting trade-off: Completely private if you can use an anonymous address but completely traceable if the address is identified. I wonder if Satoshi intended it that way or if he would have preferred the greater anonymity of Monero.

keithnz 5y ago

from what I understand, the Authors are still free to send you their papers. So perhaps the simplest decentralized system is for each author to run an automatic email request system and have feeds / aggregators of papers titles/abstracts with author emails to make it easy to get the papers you want?

ta988 5y ago
That would only work for a tiny tiny intersection of : 1) technical enough people to install and maintain that 2) with an email address still working (if there is one on the paper!) 3) still alive that's probably a really small set of scientists.

TZubiri 5y ago

I'm growning skeptical of the idea that every science text should be free. As a community that values privacy, it's hypocritical that these widespread leaking is so well accepted. The fact that scihub needs to skip domain names every couple of months and that ISPs start blocking the website, looks to confirm my theory, rather than imply some worldwide conspiracy.

ta988 5y ago
How is that a leak? Only the publishing companies are making money. Authors, reviewers and often editors all work for free.
- TZubiri 5y ago
  I don't think they work for free, they have a salary, usually paid by universities I think.
agucova 5y ago
What does this have to do with privacy?
- TZubiri 5y ago
  Publishers want to keep the papers relatively private.
  - ric2b 5y ago
    Except from literally anyone that pays them a monthly fee? That's not privacy.

Taek 5y ago

You can build something like this on Skynet. Links on Skynet (called skylinks) can point to hashes of data (similar to IPFS)or can point to a pubkey. If the data points to a pubkey, it can be tweaked repeatedly and people with the old links will see the new data. You can also point to full webapps. So for example you could have a pubkey point to a webapp that then loads the database from a list of moderators who update the data under their pubkeys to point to the latest version of scihub. Then you can even have different moderators curate articles of different subjects, and the webapp can combine everything together. All the data is stored on Sia, which is a decentralized cloud storage platform. Skynet allows anyone to upload or download data from Sia directly from the web, no need for extensions or custom software. The Sia network handles what IPFS calls "seeding", so contributors don't need to worry about leaving their machines on.

deleted 5y ago

[deleted]

i5heu 5y ago

This is where IPFS shines. In fact, there is already a ipfs mirror of scihub.

josephagoss 5y ago

This might be a good use case for Arweave.

nandhinianand 5y ago

Wouldn't IPFS(https://ipfs.io/ https://ipfs.io/) work well?..